Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redshift Destination: update spec #12100

Merged
merged 6 commits into from
Apr 27, 2022
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"host": {
"description": "Host Endpoint of the Redshift Cluster (must include the cluster-id, region and end with .redshift.amazonaws.com)",
"type": "string",
"title": "Host"
"title": "Host *"
},
"port": {
"description": "Port of the database.",
Expand All @@ -23,48 +23,48 @@
"maximum": 65536,
"default": 5439,
"examples": ["5439"],
"title": "Port"
"title": "Port *"
},
"username": {
"description": "Username to use to access the database.",
"type": "string",
"title": "Username"
"title": "Username *"
},
"password": {
"description": "Password associated with the username.",
"type": "string",
"airbyte_secret": true,
"title": "Password"
"title": "Password *"
},
"database": {
"description": "Name of the database.",
"type": "string",
"title": "Database"
"title": "Database *"
},
"schema": {
"description": "The default schema tables are written to if the source does not specify a namespace. Unless specifically configured, the usual value for this field is \"public\".",
"type": "string",
"examples": ["public"],
"default": "public",
"title": "Default Schema"
"title": "Default Schema *"
},
"s3_bucket_name": {
"title": "S3 Bucket Name",
"title": "S3 Bucket Name (Optional)",
"type": "string",
"description": "The name of the staging S3 bucket to use if utilising a COPY strategy. COPY is recommended for production workloads for better speed and scalability. See <a href=\"https://docs.aws.amazon.com/redshift/latest/dg/c_loading-data-best-practices.html\">AWS docs</a> for more details.",
"examples": ["airbyte.staging"]
},
"s3_bucket_path": {
"title": "S3 Bucket Path",
"title": "S3 Bucket Path (Optional)",
"type": "string",
"description": "The directory under the S3 bucket where data will be written. If not provided, then defaults to the root directory.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there no relevant docs to link here?

"examples": ["data_sync/test"]
},
"s3_bucket_region": {
"title": "S3 Bucket Region",
"title": "S3 Bucket Region (Optional)",
"type": "string",
"default": "",
"description": "The region of the S3 staging bucket to use if utilising a copy strategy.",
"description": "The region of the S3 staging bucket to use if utilising a COPY strategy. See the docs for details. See the docs for details. https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html#:~:text=In-,Region,-%2C%20choose%20the%20AWS",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the URL not hyperlinked? And I think you've put the text in twice for 'see the docs..'?

"enum": [
"",
"us-east-1",
Expand Down Expand Up @@ -94,28 +94,28 @@
},
"access_key_id": {
"type": "string",
"description": "The Access Key Id granting allow one to access the above S3 staging bucket. Airbyte requires Read and Write permissions to the given bucket.",
"title": "S3 Key Id",
"description": "This ID grants access to the above S3 staging bucket. Airbyte requires Read and Write permissions to the given bucket. See [this](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys) on how to generate an access key.",
"title": "S3 Key Id (Optional)",
"airbyte_secret": true
},
"secret_access_key": {
"type": "string",
"description": "The corresponding secret to the above access key id.",
"title": "S3 Access Key",
"title": "S3 Access Key (Optional)",
"airbyte_secret": true
},
"part_size": {
"type": "integer",
"minimum": 10,
"maximum": 100,
"examples": ["10"],
"description": "Optional. Increase this if syncing tables larger than 100GB. Only relevant for COPY. Files are streamed to S3 in parts. This determines the size of each part, in MBs. As S3 has a limit of 10,000 parts per file, part size affects the table size. This is 10MB by default, resulting in a default limit of 100GB tables. Note, a larger part size will result in larger memory requirements. A rule of thumb is to multiply the part size by 10 to get the memory requirement. Modify this with care.",
"title": "Stream Part Size"
"description": "Increase this if syncing tables larger than 100GB. Only relevant for COPY. Files are streamed to S3 in parts. This determines the size of each part, in MBs. As S3 has a limit of 10,000 parts per file, part size affects the table size. This is 10MB by default, resulting in a default limit of 100GB tables. Note: a larger part size will result in larger memory requirements. A rule of thumb is to multiply the part size by 10 to get the memory requirement. Modify this with care.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are they no docs to link here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"title": "Stream Part Size (Optional)"
},
"purge_staging_data": {
"title": "Purge Staging Files and Tables",
"title": "Purge Staging Files and Tables (Optional)",
"type": "boolean",
"description": "Whether to delete the staging files from S3 after completing the sync. See the docs for details. Only relevant for COPY. Defaults to true.",
"description": "Whether to delete the staging files from S3 after completing the sync. See the docs for details. https://docs.airbyte.com/integrations/destinations/redshift#2a-fill-up-s3-info-for-copy-strategy. Only relevant for COPY. Defaults to true.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are we including the URL directly and not hyperlinking the text?

"default": true
}
}
Expand Down