Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redshift Destination: update spec #12100

Merged
merged 6 commits into from
Apr 27, 2022
Merged

Redshift Destination: update spec #12100

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d701e5b
Redshift Destination: update spec
VitaliiMaltsev Apr 18, 2022
229f387
update spec.json
VitaliiMaltsev Apr 19, 2022
2989f68
update links in spec.json
VitaliiMaltsev Apr 19, 2022
32bb749
added more links to spec.json | refactoring
VitaliiMaltsev Apr 21, 2022
90dc957
updated docs with stadard connector template
VitaliiMaltsev Apr 26, 2022
2179693
added hyperlink to documentation for part_size field
VitaliiMaltsev Apr 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions airbyte-integrations/connectors/destination-redshift/src/main/resources/spec.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"host": {
"description": "Host Endpoint of the Redshift Cluster (must include the cluster-id, region and end with .redshift.amazonaws.com)",
"type": "string",
"title": "Host"
"title": "Host *"
},
"port": {
"description": "Port of the database.",
Expand All @@ -23,45 +23,45 @@
"maximum": 65536,
"default": 5439,
"examples": ["5439"],
"title": "Port"
"title": "Port *"
},
"username": {
"description": "Username to use to access the database.",
"type": "string",
"title": "Username"
"title": "Username *"
},
"password": {
"description": "Password associated with the username.",
"type": "string",
"airbyte_secret": true,
"title": "Password"
"title": "Password *"
},
"database": {
"description": "Name of the database.",
"type": "string",
"title": "Database"
"title": "Database *"
},
"schema": {
"description": "The default schema tables are written to if the source does not specify a namespace. Unless specifically configured, the usual value for this field is \"public\".",
"type": "string",
"examples": ["public"],
"default": "public",
"title": "Default Schema"
"title": "Default Schema *"
},
"s3_bucket_name": {
"title": "S3 Bucket Name",
"title": "S3 Bucket Name (Optional)",
"type": "string",
"description": "The name of the staging S3 bucket to use if utilising a COPY strategy. COPY is recommended for production workloads for better speed and scalability. See <a href=\"https://docs.aws.amazon.com/redshift/latest/dg/c_loading-data-best-practices.html\">AWS docs</a> for more details.",
"examples": ["airbyte.staging"]
},
"s3_bucket_path": {
"title": "S3 Bucket Path",
"title": "S3 Bucket Path (Optional)",
"type": "string",
"description": "The directory under the S3 bucket where data will be written. If not provided, then defaults to the root directory.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there no relevant docs to link here?

"examples": ["data_sync/test"]
},
"s3_bucket_region": {
"title": "S3 Bucket Region",
"title": "S3 Bucket Region (Optional)",
"type": "string",
"default": "",
"description": "The region of the S3 staging bucket to use if utilising a copy strategy.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"description": "The region of the S3 staging bucket to use if utilising a copy strategy.",
"description": "The region of the S3 staging bucket to use if utilising a COPY strategy.",

Expand Down Expand Up @@ -95,13 +95,13 @@
"access_key_id": {
"type": "string",
"description": "The Access Key Id granting allow one to access the above S3 staging bucket. Airbyte requires Read and Write permissions to the given bucket.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"description": "The Access Key Id granting allow one to access the above S3 staging bucket. Airbyte requires Read and Write permissions to the given bucket.",
"description": "This ID grants access to the above S3 staging bucket. Airbyte requires Read and Write permissions to the given bucket.",

"title": "S3 Key Id",
"title": "S3 Key Id (Optional)",
"airbyte_secret": true
},
"secret_access_key": {
"type": "string",
"description": "The corresponding secret to the above access key id.",
"title": "S3 Access Key",
"title": "S3 Access Key (Optional)",
"airbyte_secret": true
},
"part_size": {
Expand All @@ -110,10 +110,10 @@
"maximum": 100,
"examples": ["10"],
"description": "Optional. Increase this if syncing tables larger than 100GB. Only relevant for COPY. Files are streamed to S3 in parts. This determines the size of each part, in MBs. As S3 has a limit of 10,000 parts per file, part size affects the table size. This is 10MB by default, resulting in a default limit of 100GB tables. Note, a larger part size will result in larger memory requirements. A rule of thumb is to multiply the part size by 10 to get the memory requirement. Modify this with care.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"description": "Optional. Increase this if syncing tables larger than 100GB. Only relevant for COPY. Files are streamed to S3 in parts. This determines the size of each part, in MBs. As S3 has a limit of 10,000 parts per file, part size affects the table size. This is 10MB by default, resulting in a default limit of 100GB tables. Note, a larger part size will result in larger memory requirements. A rule of thumb is to multiply the part size by 10 to get the memory requirement. Modify this with care.",
"description": "Increase this if syncing tables larger than 100GB. Only relevant for COPY. Files are streamed to S3 in parts. This determines the size of each part, in MBs. As S3 has a limit of 10,000 parts per file, part size affects the table size. This is 10MB by default, resulting in a default limit of 100GB tables. Note: a larger part size will result in larger memory requirements. A rule of thumb is to multiply the part size by 10 to get the memory requirement. Modify this with care.",

"title": "Stream Part Size"
"title": "Stream Part Size (Optional)"
},
"purge_staging_data": {
"title": "Purge Staging Files and Tables",
"title": "Purge Staging Files and Tables (Optional)",
"type": "boolean",
"description": "Whether to delete the staging files from S3 after completing the sync. See the docs for details. Only relevant for COPY. Defaults to true.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are the docs?

"See the docs for details."

"default": true
Expand Down