Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yuunlimm/parquet fa migrate #509

Merged
merged 1 commit into from
Sep 16, 2024

Conversation

yuunlimm
Copy link
Contributor

@yuunlimm yuunlimm commented Sep 16, 2024

Validated locally using parser.yaml like below:

# This is a template yaml for the indexer processor. It assumes you have a data service
# running locally, for example as part of local testnet you ran with this command:
# aptos node run-local-testnet
health_check_port: 8084
server_config:
  processor_config:
    type: parquet_fungible_asset_activities_processor
    bucket_name: "etl-yuunnet"
    bucket_root: "local-airflow-teerst"
    parquet_handler_response_channel_size: 100
    max_buffer_size: 30000000
    parquet_upload_interval: 30
  parquet_gap_detection_batch_size: 10000000
  postgres_connection_string: postgresql://postgres:@localhost:5432/postgres
  indexer_grpc_data_service_address: "https://grpc.testnet.aptoslabs.com:443"
  auth_token: auth_token
  number_concurrent_processing_tasks: 1
  starting_version: 0

Screenshot 2024-09-16 at 11 56 51 AM
Screenshot 2024-09-16 at 11 56 49 AM
Screenshot 2024-09-16 at 12 04 53 PM

@yuunlimm yuunlimm requested review from bowenyang007 and a team September 16, 2024 19:06
@bowenyang007 bowenyang007 changed the base branch from main to parquet_migrate September 16, 2024 21:31
@bowenyang007 bowenyang007 merged commit 84c01c1 into parquet_migrate Sep 16, 2024
6 of 7 checks passed
@bowenyang007 bowenyang007 deleted the yuunlimm/parquet-fa-migrate branch September 16, 2024 21:31
yuunlimm added a commit that referenced this pull request Sep 23, 2024
yuunlimm added a commit that referenced this pull request Sep 23, 2024
* parse fa activities with parquet

* add new processor for fa activities

* Yuunlimm/parquet fa migrate (#509)

* fix lint

---------

Co-authored-by: Yuun Lim <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants