Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V1.3 #454

Merged
merged 15 commits into from
Dec 1, 2024
Merged

V1.3 #454

merged 15 commits into from
Dec 1, 2024

Conversation

flarco
Copy link
Collaborator

@flarco flarco commented Dec 1, 2024

Added Features

  1. Partitioned Parquet Support

    • Added DuckDB-based partitioned parquet file writing
    • New partition field extraction functionality
    • Support for file size limits and compression options
  2. Certificate Handling

    • Improved certificate handling in database connections
    • Added support for both file-based and direct PEM certificate values
  3. Hook System Changes

    • Split Hooks into PreHooks and PostHooks
    • Improved hook parsing and execution

Removed Features

  1. Removed TrimSpace Option
    • Removed the trim_space configuration option
    • Simplified string handling logic

Behavioral Changes

  1. Constraint Handling

    • Added support for skipping rows on constraint failures
    • New environment variable SLING_ON_CONSTRAINT_FAILURE controls behavior
  2. File Writing

    • Improved file writing logic with DuckDB integration
    • Better handling of file size limits and compression
  3. Database Connections

    • Added validation to prevent writing to unsupported databases
    • Improved connection property handling

Other Changes

  1. Configuration

    • Added support for custom paths for sqlldr and bcp
    • Improved environment file handling
    • Better default values for file operations
  2. Code Organization

    • Improved error handling
    • Better separation of concerns in file operations
    • Enhanced logging and debugging capabilities

- added .python-version, pyproject.toml, and uv.lock to ignore files
- Improved error handling for invalid certificate values and file paths in the `makeTlsConfig` function.
- Added support for PEM-encoded certificates passed directly as values, not just file paths.
- Updated the `processValue` function to handle both files and PEM-encoded certificates.
- Enhanced error messages to provide more specific information about invalid input.
- Added checks to ensure that certificates and keys are properly loaded before creating the TLS config.
- Updated unit tests to cover these changes.
- Modified existing tests to accommodate changes in the `makeTlsConfig` function.
- Ensured that the new error handling and value processing logic functions correctly in both normal and edge cases.
- Refactored the code to improve readability and maintainability.
- Removed the `trim_space` option from the `StreamConfig` struct and related functions.
- This option was redundant as the trimming of spaces is now handled implicitly based on the column type.
- This change simplifies the code and improves maintainability.
- Updated related test cases to reflect the removal of the `trim_space` option.
- Removed the `trim_space` option from the sling configuration.
- Updated the stream processor to handle empty values more robustly, using `empty_as_null` instead of relying on `trim_space`.
- Improved the handling of transforms in the stream processor, adding a dedicated `empty_as_null` transform.
- Refactored the `transforms` field in `StreamConfig` to use a `TransformList` instead of a simple array, allowing for more efficient transform lookup.
- Updated the test cases to reflect the changes.
- Prevent unnecessary processing when SampleSize is 0
- Prevent unnecessary processing when all columns are coerced to strings
@flarco flarco merged commit c5fa293 into main Dec 1, 2024
4 of 6 checks passed
@flarco flarco deleted the v1.3 branch December 4, 2024 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant