Skip to content

Commit

Permalink
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
Browse files Browse the repository at this point in the history
…5912_bigquery_denorm_anyof_and_allof
  • Loading branch information
andriikorotkov committed Apr 5, 2022
2 parents 7d73451 + 6a60314 commit 72fc35b
Show file tree
Hide file tree
Showing 654 changed files with 17,774 additions and 8,825 deletions.
8 changes: 7 additions & 1 deletion .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.35.59-alpha
current_version = 0.35.65-alpha
commit = False
tag = False
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-[a-z]+)?
Expand Down Expand Up @@ -41,3 +41,9 @@ serialize =
[bumpversion:file:kube/overlays/stable-with-resource-limits/.env]

[bumpversion:file:kube/overlays/stable-with-resource-limits/kustomization.yaml]

[bumpversion:file:octavia-cli/install.sh]

[bumpversion:file:octavia-cli/README.md]

[bumpversion:file:octavia-cli/Dockerfile]
2 changes: 1 addition & 1 deletion .env
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@


### SHARED ###
VERSION=0.35.59-alpha
VERSION=0.35.65-alpha

# When using the airbyte-db via default docker image
CONFIG_ROOT=/data
Expand Down
2 changes: 1 addition & 1 deletion .github/actions/start-aws-runner/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ runs:
aws-region: us-east-2
- name: Start EC2 runner
id: start-ec2-runner
uses: machulav/ec2-github-runner@v2.3.2
uses: supertopher/ec2-github-runner@base64v1.0.10
with:
mode: start
github-token: ${{ inputs.github-token }}
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/connector_integration_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ jobs:
steps:
- name: Checkout Airbyte
uses: actions/checkout@v2
- name: Install Java
uses: actions/setup-java@v1
with:
java-version: '17'
- name: Launch Integration Tests
run: ./tools/bin/ci_integration_workflow_launcher.sh
env:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/github-community-label-bot.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@ jobs:
steps:
- uses: harshithmullapudi/label-actions@75686c2b3de17244526f10a22424f319d0bc134f
with:
github-token: ${{ secrets.LABEL_BOT_TOKEN }}
github-username: ${{ secrets.LABEL_BOT_USERNAME }}
github-token: ${{ secrets.OCTAVIA_PAT }}
github-username: octavia-squidington-iii
34 changes: 25 additions & 9 deletions .github/workflows/gke-kube-test-command.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
name: GKE Kube Acceptance Test
on:
schedule:
- cron: '0 */6 * * *'
workflow_dispatch:
inputs:
repo:
Expand All @@ -13,14 +11,32 @@ on:
required: false
default: master
comment-id:
description: 'The comment-id of the slash command. Used to update the comment with the status.'
description: "The comment-id of the slash command. Used to update the comment with the status."
required: false

jobs:
find_valid_pat:
name: "Find a PAT with room for actions"
timeout-minutes: 10
runs-on: ubuntu-latest
outputs:
pat: ${{ steps.variables.outputs.pat }}
steps:
- name: Checkout Airbyte
uses: actions/checkout@v2
- name: Check PAT rate limits
id: variables
run: |
./tools/bin/find_non_rate_limited_PAT \
${{ secrets.AIRBYTEIO_PAT }} \
${{ secrets.OSS_BUILD_RUNNER_GITHUB_PAT }} \
${{ secrets.SUPERTOPHER_PAT }} \
${{ secrets.DAVINCHIA_PAT }}
start-gke-kube-acceptance-test-runner:
timeout-minutes: 10
name: Start GKE Kube Acceptance Test EC2 Runner
runs-on: ubuntu-latest
needs: find_valid_pat
outputs:
label: ${{ steps.start-ec2-runner.outputs.label }}
ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
Expand All @@ -36,7 +52,7 @@ jobs:
with:
aws-access-key-id: ${{ secrets.SELF_RUNNER_AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.SELF_RUNNER_AWS_SECRET_ACCESS_KEY }}
github-token: ${{ secrets.SELF_RUNNER_GITHUB_ACCESS_TOKEN }}
github-token: ${{ needs.find_valid_pat.outputs.pat }}
gke-kube-acceptance-test:
# In case of self-hosted EC2 errors, removed the `needs` line and switch back to running on ubuntu-latest.
needs: start-gke-kube-acceptance-test-runner # required to start the main job when the runner is ready
Expand All @@ -60,11 +76,11 @@ jobs:

- uses: actions/setup-java@v1
with:
java-version: '17'
java-version: "17"

- uses: actions/setup-node@v1
with:
node-version: '16.13.0'
node-version: "16.13.0"

- name: Fix EC-2 Runner
run: |
Expand Down Expand Up @@ -126,6 +142,7 @@ jobs:
needs:
- start-gke-kube-acceptance-test-runner # required to get output from the start-runner job
- gke-kube-acceptance-test # required to wait when the main job is done
- find_valid_pat
runs-on: ubuntu-latest
if: ${{ always() }}
steps:
Expand All @@ -136,10 +153,9 @@ jobs:
aws-secret-access-key: ${{ secrets.SELF_RUNNER_AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-2
- name: Stop EC2 runner
uses: machulav/ec2-github-runner@v2.3.0
uses: supertopher/ec2-github-runner@base64v1.0.10
with:
mode: stop
github-token: ${{ secrets.SELF_RUNNER_GITHUB_ACCESS_TOKEN }}
github-token: ${{ needs.find_valid_pat.outputs.pat }}
label: ${{ needs.start-gke-kube-acceptance-test-runner.outputs.label }}
ec2-instance-id: ${{ needs.start-gke-kube-acceptance-test-runner.outputs.ec2-instance-id }}

Loading

2 comments on commit 72fc35b

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SonarQube Report

SonarQube report for Airbyte Connectors Destination Bigquery(#11166)

Measures

Name Value Name Value Name Value
Lines of Code 1268 Bugs 2 Vulnerabilities 0
Code Smells 36 Duplicated Blocks 0 Security Rating A
Reliability Rating C Lines to Cover 612 Duplicated Lines (%) 0.0
Quality Gate Status OK Coverage 0.0 Blocker Issues 0
Critical Issues 3 Major Issues 30 Minor Issues 3

Detected Issues

Rule File Description Message
java:S1135 (INFO) bigquery/BigQueryDestination.java:81 Track uses of "TODO" tags Complete the task associated to this TODO comment.
java:S112 (MAJOR) bigquery/BigQueryRecordConsumer.java:69 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) writer/BigQueryTableWriter.java:34 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1452 (CRITICAL) uploader/BigQueryUploaderFactory.java:34 Generic wildcard types should not be used in return types Remove usage of generic wildcard type.
java:S5993 (MAJOR) formatter/BigQueryRecordFormatter.java:37 Constructors of an "abstract" class should not be declared "public" Change the visibility of this constructor to "protected".
java:S1116 (MINOR) formatter/BigQueryRecordFormatter.java:45 Empty statements should be removed Remove this empty statement.
java:S112 (MAJOR) uploader/AbstractBigQueryUploader.java:56 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) uploader/AbstractBigQueryUploader.java:70 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S3457 (MAJOR) uploader/AbstractBigQueryUploader.java:88 Printf-style format strings should be used correctly %n should be used in place of \n to produce the platform-specific line separator.
java:S112 (MAJOR) uploader/AbstractBigQueryUploader.java:90 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S125 (MAJOR) uploader/AbstractBigQueryUploader.java:111 Sections of code should not be commented out This block of commented-out lines of code should be removed.
java:S112 (MAJOR) uploader/AbstractBigQueryUploader.java:120 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S2142 (MAJOR) uploader/AbstractBigQueryUploader.java:166 "InterruptedException" should not be ignored Either re-interrupt this method or rethrow the "InterruptedException" that can be caught here.
java:S112 (MAJOR) uploader/AbstractBigQueryUploader.java:185 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S2142 (MAJOR) uploader/AbstractGscBigQueryUploader.java:83 "InterruptedException" should not be ignored Either re-interrupt this method or rethrow the "InterruptedException" that can be caught here.
java:S112 (MAJOR) uploader/AbstractGscBigQueryUploader.java:85 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1124 (MINOR) uploader/AbstractGscBigQueryUploader.java:89 Modifiers should be declared in the correct order Reorder the modifiers to comply with the Java Language Specification.
java:S1117 (MAJOR) uploader/AbstractGscBigQueryUploader.java:97 Local variables should not shadow class fields Rename "gcsDestinationConfig" which hides the field declared at line 32.
java:S1068 (MAJOR) uploader/BigQueryDirectUploader.java:18 Unused "private" fields should be removed Remove this unused "LOGGER" private field.
java:S1118 (MAJOR) uploader/BigQueryUploaderFactory.java:32 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S107 (MAJOR) uploader/BigQueryUploaderFactory.java:84 Methods should not have too many parameters Method has 8 parameters, which is greater than 7 authorized.
java:S1068 (MAJOR) config/UploaderConfig.java:20 Unused "private" fields should be removed Remove this unused "config" private field.
java:S1068 (MAJOR) config/UploaderConfig.java:21 Unused "private" fields should be removed Remove this unused "configStream" private field.
java:S1068 (MAJOR) config/UploaderConfig.java:22 Unused "private" fields should be removed Remove this unused "targetTableName" private field.
java:S1068 (MAJOR) config/UploaderConfig.java:23 Unused "private" fields should be removed Remove this unused "tmpTableName" private field.
java:S1068 (MAJOR) config/UploaderConfig.java:24 Unused "private" fields should be removed Remove this unused "bigQuery" private field.
java:S1068 (MAJOR) config/UploaderConfig.java:25 Unused "private" fields should be removed Remove this unused "formatterMap" private field.
java:S1068 (MAJOR) config/UploaderConfig.java:26 Unused "private" fields should be removed Remove this unused "isDefaultAirbyteTmpSchema" private field.
java:S1068 (MAJOR) writer/BigQueryTableWriter.java:21 Unused "private" fields should be removed Remove this unused "LOGGER" private field.
java:S1186 (CRITICAL) writer/BigQueryTableWriter.java:30 Methods should not be empty Add a nested comment explaining why this method is empty, throw an UnsupportedOperationException or complete the implementation.
java:S1118 (MAJOR) bigquery/BigQueryConsts.java:7 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S115 (CRITICAL) bigquery/BigQueryConsts.java:9 Constant names should comply with a naming convention Rename this constant name to match the regular expression '^[A-Z][A-Z0-9](_[A-Z0-9]+)$'.
java:S1450 (MINOR) bigquery/BigQueryRecordConsumer.java:27 Private fields only used as local variables in methods should become local variables Remove the "lastStateMessage" field and declare it as a local variable in the relevant methods.
java:S1135 (INFO) bigquery/BigQueryRecordConsumer.java:37 Track uses of "TODO" tags Complete the task associated to this TODO comment.
java:S1118 (MAJOR) bigquery/BigQueryUtils.java:45 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S112 (MAJOR) bigquery/BigQueryUtils.java:61 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) bigquery/BigQueryUtils.java:76 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) bigquery/BigQueryDestination.java:163 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.

Coverage (0.0%)

File Coverage File Coverage
src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryConsts.java 0.0 src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryDestination.java 0.0
src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryRecordConsumer.java 0.0 src/main/java/io/airbyte/integrations/destination/bigquery/BigQuerySQLNameTransformer.java 0.0
src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryUtils.java 0.0 src/main/java/io/airbyte/integrations/destination/bigquery/formatter/BigQueryRecordFormatter.java 0.0
src/main/java/io/airbyte/integrations/destination/bigquery/formatter/DefaultBigQueryRecordFormatter.java 0.0 src/main/java/io/airbyte/integrations/destination/bigquery/formatter/GcsAvroBigQueryRecordFormatter.java 0.0
src/main/java/io/airbyte/integrations/destination/bigquery/formatter/GcsCsvBigQueryRecordFormatter.java 0.0 src/main/java/io/airbyte/integrations/destination/bigquery/helpers/LoggerHelper.java 0.0
src/main/java/io/airbyte/integrations/destination/bigquery/uploader/AbstractBigQueryUploader.java 0.0 src/main/java/io/airbyte/integrations/destination/bigquery/uploader/AbstractGscBigQueryUploader.java 0.0
src/main/java/io/airbyte/integrations/destination/bigquery/uploader/BigQueryDirectUploader.java 0.0 src/main/java/io/airbyte/integrations/destination/bigquery/uploader/BigQueryUploaderFactory.java 0.0
src/main/java/io/airbyte/integrations/destination/bigquery/uploader/GcsAvroBigQueryUploader.java 0.0 src/main/java/io/airbyte/integrations/destination/bigquery/uploader/GcsCsvBigQueryUploader.java 0.0
src/main/java/io/airbyte/integrations/destination/bigquery/writer/BigQueryTableWriter.java 0.0

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SonarQube Report

SonarQube report for Airbyte Connectors Destination Bigquery Denormalized(#11166)

Measures

Name Value Name Value Name Value
Security Rating A Quality Gate Status OK Duplicated Lines (%) 0.0
Reliability Rating A Coverage 0.0 Bugs 0
Lines of Code 462 Lines to Cover 236 Duplicated Blocks 0
Vulnerabilities 0 Code Smells 7 Blocker Issues 0
Critical Issues 3 Major Issues 3 Minor Issues 1

Detected Issues

Rule File Description Message
java:S1602 (MINOR) formatter/DefaultBigQueryDenormalizedRecordFormatter.java:275 Lambdas containing only one statement should not nest this statement in a block Remove useless curly braces around statement (sonar.java.source not set. Assuming 8 or greater.)
java:S3776 (CRITICAL) formatter/DefaultBigQueryDenormalizedRecordFormatter.java:286 Cognitive Complexity of methods should not be too high Refactor this method to reduce its Cognitive Complexity from 26 to the 15 allowed.
java:S1192 (CRITICAL) formatter/DefaultBigQueryDenormalizedRecordFormatter.java:309 String literals should not be duplicated Use already-defined constant 'ARRAY_ITEMS_FIELD' instead of duplicating its value here.
java:S3776 (CRITICAL) formatter/DefaultBigQueryDenormalizedRecordFormatter.java:77 Cognitive Complexity of methods should not be too high Refactor this method to reduce its Cognitive Complexity from 16 to the 15 allowed.
java:S1700 (MAJOR) bigquery/JsonSchemaFormat.java:19 A field should not duplicate the name of its containing class Rename field "jsonSchemaFormat"
java:S1068 (MAJOR) bigquery/BigQueryDenormalizedDestination.java:20 Unused "private" fields should be removed Remove this unused "LOGGER" private field.
java:S1700 (MAJOR) bigquery/JsonSchemaType.java:27 A field should not duplicate the name of its containing class Rename field "jsonSchemaType"

Coverage (0.0%)

File Coverage File Coverage
src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryDenormalizedDestination.java 0.0 src/main/java/io/airbyte/integrations/destination/bigquery/formatter/DefaultBigQueryDenormalizedRecordFormatter.java 0.0
src/main/java/io/airbyte/integrations/destination/bigquery/formatter/GcsBigQueryDenormalizedRecordFormatter.java 0.0 src/main/java/io/airbyte/integrations/destination/bigquery/JsonSchemaFormat.java 0.0
src/main/java/io/airbyte/integrations/destination/bigquery/JsonSchemaType.java 0.0

Please sign in to comment.