gRPC fails to load data using storage_write_api

[rubenvereecken]

Specifically, using TARGET_BIGQUERY_DEBUG=true, I get the error:

google.api_core.exceptions.NotFound: 404 Requested entity was not found. Entity: projects/summitsift/datasets/test_data/tables/users/streams/_default

I don't know how gRPC works well enough to investigate this error. Looking at BigQuery, the dataset summitsift.test_data is created correctly, and it contains a table users with the correct schema.

What I can say is that

• it's a custom tap, though loadertarget-jsonl works just fine
• tap-carbon-intensity works fine most of the time with the same config (I can't reproduce it now)

meltano.yml:

  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+https://github.com/z3z1ma/target-bigquery.git
    config:
      credentials_path: ${credentials}
      dataset: test_data
      denormalized: true
      fail_fast: true
      flattening_enabled: false
      location: europe-west1
      project: ${project}
      column_name_transforms:
        snake_case: true
        add_underscore_when_invalid: true

[rubenvereecken]

Interesting... I had this weird idea that it might be a race condition because I was only loading 40 records. So I ran it for 400 records and got this error instead:

google.api_core.exceptions.InvalidArgument: 400 Invalid proto schema: BqMessage.proto: AnonymousProto_bd328dbff4be7bd443dbe33af00f5f063ab87b11.attributes: ".net.proto2.python.public.target_bigquery.AnonymousProto_d1305d34ce666248739e2394ea7836d25b5d29a5" is not defined. Entity: projects/summitsift/datasets/test_data/tables/users/streams/_default