[SDK-parquet] add parquet version tracker

[yuunlimm]

Description

1. Added Parquet Version Tracker Functionality

• Updated steps to include set_backfill_table_flag logic for selective table processing.
• added a processor status saver for parquet

2. Schema and Table Handling Updates

• Updated the logic for handling backfill tables:
• Renamed the tables field with backfill_table in - ParquetDefaultProcessorConfig.
• Adjusted validations and logic to ensure only valid tables are processed.
  ParquetTypeEnum improvements:
  • Added mappings and validations for table names.
  • Enhanced schema initialization and writer creation.

3. Tests updated

• Modified tests to reflect changes in backfill_table handling and validation.
• Updated table name checks to ensure compatibility with the new logic.
• Added test coverage for: Invalid backfill tables.

4. General Code Improvements

• Removed redundant logic in ParquetDefaultProcessor.
• Moved shared functionality (e.g., writer creation) to reusable helper functions.
• initialize_database_pool centralizes database pool setup for Postgres.
• Handles error cases cleanly.
• initialize_gcs_client abstracts GCS client setup using provided credentials.
• Consolidated initialization of schemas, writers, and GCS uploaders into modular functions.

Enhanced comments for better readability and maintainability.

Test Plan

[yuunlimm]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [SDK-parquet] add parquet version tracker #609 👈 (View in Graphite)
• [SDK-parquet] parquet sized buffer and gcs handler #602
• [SDK-parquet] parquet default processor extractor step #601
• [parquet-sdk-migration] add a logic to determine the starting version for parquet processor #587 : 1 other dependent PR (#591 )
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[dermanyang]

what's the difference between this and the non-parquet processor_status_saver?

[yuunlimm]

only difference is it takes in table_name: &str, arg b/c it should be able to to update row per table. wasn't sure how we could reduce the duplicated code here :(

[dermanyang]

i'm wondering if the parquet version tracker could also just use the postgres processor_status_saver?

[yuunlimm]

hmm if we could add another function(save_parquet_processor_status) to processor_status_saver with arg and let that call the existing function in the processor_status_saver, that would also work

[dermanyang]

mmmm right. would it work if we added parquet_table_name as an optional param?

[dermanyang]

discussed offline: consider using a more descriptive enum instead of this bool flag. Ideally we are able to determine what type of processor_status_saver to return just from the config. It's weird/inconsistent that we have this function determine which enum to return while relying on a user input to help with that decision