Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interactive CLI for upsert option saves as a string, but needs to be a boolean #93

Open
JamesStratford opened this issue Jul 16, 2024 · 1 comment

Comments

@JamesStratford
Copy link

When using meltano config tap-bigquery set --interactive, if you set upsert to true, it will save as a string and the following error occurs

2024-07-15T03:31:26.531723Z [info     ]   File "/workspaces/meltano/meltano-xxx/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/referencing/jsonschema.py", line 56, in _legacy_dollar_id cmd_type=elb consumer=True job_name=dev:tap-tiktok-to-target-bigquery name=target-bigquery producer=False run_id=b0731a91-d41a-4027-9540-c1ac834ea862 stdio=stderr string_id=target-bigquery
2024-07-15T03:31:26.531817Z [info     ]     id = contents.get("$id")   cmd_type=elb consumer=True job_name=dev:tap-tiktok-to-target-bigquery name=target-bigquery producer=False run_id=b0731a91-d41a-4027-9540-c1ac834ea862 stdio=stderr string_id=target-bigquery
2024-07-15T03:31:26.531901Z [info     ]          ^^^^^^^^^^^^          cmd_type=elb consumer=True job_name=dev:tap-tiktok-to-target-bigquery name=target-bigquery producer=False run_id=b0731a91-d41a-4027-9540-c1ac834ea862 stdio=stderr string_id=target-bigquery
2024-07-15T03:31:26.532010Z [info     ] AttributeError: 'str' object has no attribute 'get' cmd_type=elb consumer=True job_name=dev:tap-tiktok-to-target-bigquery name=target-bigquery producer=False run_id=b0731a91-d41a-4027-9540-c1ac834ea862 stdio=stderr string_id=target-bigquery

This is what the YAML produces

  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+https://github.com/z3z1ma/target-bigquery.git
    config:
      dataset: '${BQ_DATASET}'
      project: REDACTED
      location: REDACTED
      denormalized: true
      options:
        storage_write_batch_mode: true
      method: batch_job
      upsert: 'True'

Manually changing it to just the boolean true fixes the error

@AlejandroUPC
Copy link
Contributor

Mmm interesting, I have not taken a look or debugged my self but it looks like it just gets the value from a dictionary, if not found is False otherwise uses the value, which should be evaluated as true in Python?

    def _is_upsert_candidate(self) -> bool:
        """Determine if this stream is an upsert candidate based on user configuration."""
        upsert_selection = self.config.get("upsert", False)
        upsert_candidate = False
        if isinstance(upsert_selection, list):
            selection: str
            for selection in upsert_selection:
                invert = selection.startswith("!")
                if invert:
                    selection = selection[1:]
                if fnmatch(self.stream_name, selection):
                    upsert_candidate = True ^ invert
        elif upsert_selection:
            upsert_candidate = True
        return upsert_candidate

https://github.com/z3z1ma/target-bigquery/blob/main/target_bigquery/core.py#L383C9-L383C25

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants