We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hello, Thanks for the work on the connector, it rocks !
Since PR #96 was merged, I have the following issue: (using meltano and python 3.11)
2024-11-18T11:00:54.236730Z [info ] 2024-11-18 12:00:54,236 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 3.69802, "tags": {"stream": "product", "endpoint": "/api/analysis/pagedProductList", "http_status_code": 200, "status": "succeeded"}} cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:00:54.250227Z [info ] 2024-11-18 12:00:54,249 | INFO | singer_sdk.helpers.jsonpath | JSONPath $[*] match count: 200 cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:00:57.767347Z [info ] 2024-11-18 12:00:57,766 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 3.186238, "tags": {"stream": "product", "endpoint": "/api/analysis/pagedProductList", "http_status_code": 200, "status": "succeeded"}} cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:00:57.779962Z [info ] 2024-11-18 12:00:57,779 | INFO | singer_sdk.helpers.jsonpath | JSONPath $[*] match count: 200 cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:01:01.232434Z [info ] 2024-11-18 12:01:01,232 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 3.095553, "tags": {"stream": "product", "endpoint": "/api/analysis/pagedProductList", "http_status_code": 200, "status": "succeeded"}} cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:01:01.250335Z [info ] 2024-11-18 12:01:01,250 | INFO | singer_sdk.helpers.jsonpath | JSONPath $[*] match count: 200 cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:01:06.405633Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406176Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/bin/target-bigquery", line 8, in <module> cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406610Z [info ] sys.exit(TargetBigQuery.cli()) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406826Z [info ] ^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406939Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407064Z [info ] return self.main(*args, **kwargs) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407175Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407294Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407436Z [info ] rv = self.invoke(ctx) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407558Z [info ] ^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407669Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407783Z [info ] return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407893Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408017Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408159Z [info ] return __callback(*args, **kwargs) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408271Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408380Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 578, in cli cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408491Z [info ] target.listen(file_input) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408599Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/io_base.py", line 34, in listen cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408709Z [info ] self._process_lines(file_input) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408821Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 278, in _process_lines cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408966Z [info ] counter = super()._process_lines(file_input) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409086Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409195Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/io_base.py", line 87, in _process_lines cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409330Z [info ] self._process_state_message(line_dict) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409449Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 409, in _process_state_message cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409568Z [info ] self.drain_all() cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409677Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/target.py", line 578, in drain_all cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409796Z [info ] sink.pre_state_hook() cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409905Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/core.py", line 507, in pre_state_hook cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410067Z [info ] self._create_overwrite_table() cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410184Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/core.py", line 348, in _create_overwrite_table cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410293Z [info ] self.table.create_table( cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410404Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/core.py", line 187, in create_table cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410513Z [info ] self._dataset = client.get_dataset(self.as_dataset_ref()) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410623Z [info ] ^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410740Z [info ] AttributeError: 'Client' object has no attribute 'get_dataset' cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.605239Z [error ] ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/logging/output_lo │ │ gger.py:217 in redirect_logging │ │ │ │ 214 │ │ │ *ignore_errors, │ │ 215 │ │ ) │ │ 216 │ │ try: │ │ ❱ 217 │ │ │ yield │ │ 218 │ │ except ignored_errors: │ │ 219 │ │ │ raise │ │ 220 │ │ except RunnerError as err: │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:506 in run │ │ │ │ 503 │ │ │ # TODO: legacy meltano elt style logging should be deprecated │ │ 504 │ │ │ legacy_log_handler = self.output_logger.out("meltano", logger) │ │ 505 │ │ │ with legacy_log_handler.redirect_logging(): │ │ ❱ 506 │ │ │ │ await self.run_with_job() │ │ 507 │ │ │ │ return │ │ 508 │ │ else: │ │ 509 │ │ │ logger.warning( │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:538 in run_with_job │ │ │ │ 535 │ │ │ │ 536 │ │ with closing(self.context.session) as session: │ │ 537 │ │ │ async with job.run(session): │ │ ❱ 538 │ │ │ │ await self.execute() │ │ 539 │ │ │ 540 │ async def terminate(self, *, graceful: bool = False) -> None: │ │ 541 │ │ """Terminate an in flight ExtractLoad execution, potentially disruptive. │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:498 in execute │ │ │ │ 495 │ │ async with self._start_blocks(): │ │ 496 │ │ │ await self._link_io() │ │ 497 │ │ │ manager = ELBExecutionManager(self) │ │ ❱ 498 │ │ │ await manager.run() │ │ 499 │ │ │ 500 │ async def run(self) -> None: │ │ 501 │ │ """Run the ELT task.""" │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:705 in run │ │ │ │ 702 │ │ for IO to complete as appropriate. Expect a RunnerError to be raised if │ │ 703 │ │ any of the blocks exit with a non 0 exit code. │ │ 704 │ │ """ │ │ ❱ 705 │ │ await self._wait_for_process_completion(self.elb.head) │ │ 706 │ │ _check_exit_codes( │ │ 707 │ │ │ self._producer_code, │ │ 708 │ │ │ self._consumer_code, │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:778 in _wait_for_process_completion │ │ │ │ 775 │ │ │ │ │ line_length_limit=self.line_length_limit, │ │ 776 │ │ │ │ │ stream_buffer_size=self.stream_buffer_size, │ │ 777 │ │ │ │ ) │ │ ❱ 778 │ │ │ raise output_futures_failed.exception() │ │ 779 │ │ │ │ 780 │ │ # If all the output handlers completed without raising an │ │ 781 │ │ # exception, we still need to wait for all the underlying block │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/logging/utils.py: │ │ 229 in capture_subprocess_output │ │ │ │ 226 │ │ │ continue │ │ 227 │ │ │ │ 228 │ │ for writer in line_writers: │ │ ❱ 229 │ │ │ if not await _write_line_writer(writer, line): │ │ 230 │ │ │ │ # If the destination stream is closed, we can stop capturing output. │ │ 231 │ │ │ │ return │ │ 232 │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/logging/utils.py: │ │ 197 in _write_line_writer │ │ │ │ 194 │ │ │ writer.write(line) │ │ 195 │ │ │ await writer.drain() │ │ 196 │ │ except (BrokenPipeError, ConnectionResetError): │ │ ❱ 197 │ │ │ await writer.wait_closed() │ │ 198 │ │ │ return False │ │ 199 │ else: │ │ 200 │ │ writer.writeline(line.decode(errors="replace")) │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/asyncio/streams.py:364 in wait_closed │ │ │ │ 361 │ │ return self._transport.is_closing() │ │ 362 │ │ │ 363 │ async def wait_closed(self): │ │ ❱ 364 │ │ await self._protocol._get_close_waiter(self) │ │ 365 │ │ │ 366 │ def get_extra_info(self, name, default=None): │ │ 367 │ │ return self._transport.get_extra_info(name, default) │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ BrokenPipeError
2024-11-18T11:00:54.236730Z [info ] 2024-11-18 12:00:54,236 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 3.69802, "tags": {"stream": "product", "endpoint": "/api/analysis/pagedProductList", "http_status_code": 200, "status": "succeeded"}} cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:00:54.250227Z [info ] 2024-11-18 12:00:54,249 | INFO | singer_sdk.helpers.jsonpath | JSONPath $[*] match count: 200 cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:00:57.767347Z [info ] 2024-11-18 12:00:57,766 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 3.186238, "tags": {"stream": "product", "endpoint": "/api/analysis/pagedProductList", "http_status_code": 200, "status": "succeeded"}} cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:00:57.779962Z [info ] 2024-11-18 12:00:57,779 | INFO | singer_sdk.helpers.jsonpath | JSONPath $[*] match count: 200 cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:01:01.232434Z [info ] 2024-11-18 12:01:01,232 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 3.095553, "tags": {"stream": "product", "endpoint": "/api/analysis/pagedProductList", "http_status_code": 200, "status": "succeeded"}} cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:01:01.250335Z [info ] 2024-11-18 12:01:01,250 | INFO | singer_sdk.helpers.jsonpath | JSONPath $[*] match count: 200 cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:01:06.405633Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406176Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/bin/target-bigquery", line 8, in <module> cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406610Z [info ] sys.exit(TargetBigQuery.cli()) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406826Z [info ] ^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406939Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407064Z [info ] return self.main(*args, **kwargs) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407175Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407294Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407436Z [info ] rv = self.invoke(ctx) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407558Z [info ] ^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407669Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407783Z [info ] return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407893Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408017Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408159Z [info ] return __callback(*args, **kwargs) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408271Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408380Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 578, in cli cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408491Z [info ] target.listen(file_input) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408599Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/io_base.py", line 34, in listen cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408709Z [info ] self._process_lines(file_input) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408821Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 278, in _process_lines cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408966Z [info ] counter = super()._process_lines(file_input) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409086Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409195Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/io_base.py", line 87, in _process_lines cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409330Z [info ] self._process_state_message(line_dict) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409449Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 409, in _process_state_message cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409568Z [info ] self.drain_all() cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409677Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/target.py", line 578, in drain_all cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409796Z [info ] sink.pre_state_hook() cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409905Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/core.py", line 507, in pre_state_hook cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410067Z [info ] self._create_overwrite_table() cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410184Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/core.py", line 348, in _create_overwrite_table cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410293Z [info ] self.table.create_table( cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410404Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/core.py", line 187, in create_table cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410513Z [info ] self._dataset = client.get_dataset(self.as_dataset_ref()) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410623Z [info ] ^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410740Z [info ] AttributeError: 'Client' object has no attribute 'get_dataset' cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.605239Z [error ] ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/logging/output_lo │ │ gger.py:217 in redirect_logging │ │ │ │ 214 │ │ │ *ignore_errors, │ │ 215 │ │ ) │ │ 216 │ │ try: │ │ ❱ 217 │ │ │ yield │ │ 218 │ │ except ignored_errors: │ │ 219 │ │ │ raise │ │ 220 │ │ except RunnerError as err: │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:506 in run │ │ │ │ 503 │ │ │ # TODO: legacy
style logging should be deprecated │ │ 504 │ │ │ legacy_log_handler = self.output_logger.out("meltano", logger) │ │ 505 │ │ │ with legacy_log_handler.redirect_logging(): │ │ ❱ 506 │ │ │ │ await self.run_with_job() │ │ 507 │ │ │ │ return │ │ 508 │ │ else: │ │ 509 │ │ │ logger.warning( │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:538 in run_with_job │ │ │ │ 535 │ │ │ │ 536 │ │ with closing(self.context.session) as session: │ │ 537 │ │ │ async with job.run(session): │ │ ❱ 538 │ │ │ │ await self.execute() │ │ 539 │ │ │ 540 │ async def terminate(self, *, graceful: bool = False) -> None: │ │ 541 │ │ """Terminate an in flight ExtractLoad execution, potentially disruptive. │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:498 in execute │ │ │ │ 495 │ │ async with self._start_blocks(): │ │ 496 │ │ │ await self._link_io() │ │ 497 │ │ │ manager = ELBExecutionManager(self) │ │ ❱ 498 │ │ │ await manager.run() │ │ 499 │ │ │ 500 │ async def run(self) -> None: │ │ 501 │ │ """Run the ELT task.""" │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:705 in run │ │ │ │ 702 │ │ for IO to complete as appropriate. Expect a RunnerError to be raised if │ │ 703 │ │ any of the blocks exit with a non 0 exit code. │ │ 704 │ │ """ │ │ ❱ 705 │ │ await self._wait_for_process_completion(self.elb.head) │ │ 706 │ │ _check_exit_codes( │ │ 707 │ │ │ self._producer_code, │ │ 708 │ │ │ self._consumer_code, │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:778 in _wait_for_process_completion │ │ │ │ 775 │ │ │ │ │ line_length_limit=self.line_length_limit, │ │ 776 │ │ │ │ │ stream_buffer_size=self.stream_buffer_size, │ │ 777 │ │ │ │ ) │ │ ❱ 778 │ │ │ raise output_futures_failed.exception() │ │ 779 │ │ │ │ 780 │ │ # If all the output handlers completed without raising an │ │ 781 │ │ # exception, we still need to wait for all the underlying block │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/logging/utils.py: │ │ 229 in capture_subprocess_output │ │ │ │ 226 │ │ │ continue │ │ 227 │ │ │ │ 228 │ │ for writer in line_writers: │ │ ❱ 229 │ │ │ if not await _write_line_writer(writer, line): │ │ 230 │ │ │ │ # If the destination stream is closed, we can stop capturing output. │ │ 231 │ │ │ │ return │ │ 232 │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/logging/utils.py: │ │ 197 in _write_line_writer │ │ │ │ 194 │ │ │ writer.write(line) │ │ 195 │ │ │ await writer.drain() │ │ 196 │ │ except (BrokenPipeError, ConnectionResetError): │ │ ❱ 197 │ │ │ await writer.wait_closed() │ │ 198 │ │ │ return False │ │ 199 │ else: │ │ 200 │ │ writer.writeline(line.decode(errors="replace")) │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/asyncio/streams.py:364 in wait_closed │ │ │ │ 361 │ │ return self._transport.is_closing() │ │ 362 │ │ │ 363 │ async def wait_closed(self): │ │ ❱ 364 │ │ await self._protocol._get_close_waiter(self) │ │ 365 │ │ │ 366 │ def get_extra_info(self, name, default=None): │ │ 367 │ │ return self._transport.get_extra_info(name, default) │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ BrokenPipeError
Here is my conf: config: method: gcs_stage denormalized: true batch_size: 500 column_name_transforms: snake_case: false upsert: true generate_view: false schema_resolver_version: 2 project: $PROJECT dataset: $DATASET location: EU bucket: $BUCKET
config: method: gcs_stage denormalized: true batch_size: 500 column_name_transforms: snake_case: false upsert: true generate_view: false schema_resolver_version: 2 project: $PROJECT dataset: $DATASET location: EU bucket: $BUCKET
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Hello,
Thanks for the work on the connector, it rocks !
Since PR #96 was merged, I have the following issue:
(using meltano and python 3.11)
2024-11-18T11:00:54.236730Z [info ] 2024-11-18 12:00:54,236 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 3.69802, "tags": {"stream": "product", "endpoint": "/api/analysis/pagedProductList", "http_status_code": 200, "status": "succeeded"}} cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:00:54.250227Z [info ] 2024-11-18 12:00:54,249 | INFO | singer_sdk.helpers.jsonpath | JSONPath $[*] match count: 200 cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:00:57.767347Z [info ] 2024-11-18 12:00:57,766 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 3.186238, "tags": {"stream": "product", "endpoint": "/api/analysis/pagedProductList", "http_status_code": 200, "status": "succeeded"}} cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:00:57.779962Z [info ] 2024-11-18 12:00:57,779 | INFO | singer_sdk.helpers.jsonpath | JSONPath $[*] match count: 200 cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:01:01.232434Z [info ] 2024-11-18 12:01:01,232 | INFO | singer_sdk.metrics | METRIC: {"type": "timer", "metric": "http_request_duration", "value": 3.095553, "tags": {"stream": "product", "endpoint": "/api/analysis/pagedProductList", "http_status_code": 200, "status": "succeeded"}} cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:01:01.250335Z [info ] 2024-11-18 12:01:01,250 | INFO | singer_sdk.helpers.jsonpath | JSONPath $[*] match count: 200 cmd_type=elb consumer=False job_name=default:tap-customapi-product-to-target-bigquery name=tap-customapi-product producer=True run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=tap-customapi-product 2024-11-18T11:01:06.405633Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406176Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/bin/target-bigquery", line 8, in <module> cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406610Z [info ] sys.exit(TargetBigQuery.cli()) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406826Z [info ] ^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.406939Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407064Z [info ] return self.main(*args, **kwargs) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407175Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407294Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407436Z [info ] rv = self.invoke(ctx) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407558Z [info ] ^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407669Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407783Z [info ] return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.407893Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408017Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408159Z [info ] return __callback(*args, **kwargs) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408271Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408380Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 578, in cli cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408491Z [info ] target.listen(file_input) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408599Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/io_base.py", line 34, in listen cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408709Z [info ] self._process_lines(file_input) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408821Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 278, in _process_lines cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.408966Z [info ] counter = super()._process_lines(file_input) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409086Z [info ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409195Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/io_base.py", line 87, in _process_lines cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409330Z [info ] self._process_state_message(line_dict) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409449Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/singer_sdk/target_base.py", line 409, in _process_state_message cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409568Z [info ] self.drain_all() cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409677Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/target.py", line 578, in drain_all cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409796Z [info ] sink.pre_state_hook() cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.409905Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/core.py", line 507, in pre_state_hook cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410067Z [info ] self._create_overwrite_table() cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410184Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/core.py", line 348, in _create_overwrite_table cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410293Z [info ] self.table.create_table( cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410404Z [info ] File "/home/etienne/dev/custom/modules/customapi/.meltano/loaders/target-bigquery/venv/lib/python3.11/site-packages/target_bigquery/core.py", line 187, in create_table cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410513Z [info ] self._dataset = client.get_dataset(self.as_dataset_ref()) cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410623Z [info ] ^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.410740Z [info ] AttributeError: 'Client' object has no attribute 'get_dataset' cmd_type=elb consumer=True job_name=default:tap-customapi-product-to-target-bigquery name=target-bigquery producer=False run_id=85737c8c-4c60-4295-a022-f031ce5f8d09 stdio=stderr string_id=target-bigquery 2024-11-18T11:01:06.605239Z [error ] ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/logging/output_lo │ │ gger.py:217 in redirect_logging │ │ │ │ 214 │ │ │ *ignore_errors, │ │ 215 │ │ ) │ │ 216 │ │ try: │ │ ❱ 217 │ │ │ yield │ │ 218 │ │ except ignored_errors: │ │ 219 │ │ │ raise │ │ 220 │ │ except RunnerError as err: │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:506 in run │ │ │ │ 503 │ │ │ # TODO: legacymeltano eltstyle logging should be deprecated │ │ 504 │ │ │ legacy_log_handler = self.output_logger.out("meltano", logger) │ │ 505 │ │ │ with legacy_log_handler.redirect_logging(): │ │ ❱ 506 │ │ │ │ await self.run_with_job() │ │ 507 │ │ │ │ return │ │ 508 │ │ else: │ │ 509 │ │ │ logger.warning( │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:538 in run_with_job │ │ │ │ 535 │ │ │ │ 536 │ │ with closing(self.context.session) as session: │ │ 537 │ │ │ async with job.run(session): │ │ ❱ 538 │ │ │ │ await self.execute() │ │ 539 │ │ │ 540 │ async def terminate(self, *, graceful: bool = False) -> None: │ │ 541 │ │ """Terminate an in flight ExtractLoad execution, potentially disruptive. │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:498 in execute │ │ │ │ 495 │ │ async with self._start_blocks(): │ │ 496 │ │ │ await self._link_io() │ │ 497 │ │ │ manager = ELBExecutionManager(self) │ │ ❱ 498 │ │ │ await manager.run() │ │ 499 │ │ │ 500 │ async def run(self) -> None: │ │ 501 │ │ """Run the ELT task.""" │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:705 in run │ │ │ │ 702 │ │ for IO to complete as appropriate. Expect a RunnerError to be raised if │ │ 703 │ │ any of the blocks exit with a non 0 exit code. │ │ 704 │ │ """ │ │ ❱ 705 │ │ await self._wait_for_process_completion(self.elb.head) │ │ 706 │ │ _check_exit_codes( │ │ 707 │ │ │ self._producer_code, │ │ 708 │ │ │ self._consumer_code, │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/block/extract_loa │ │ d.py:778 in _wait_for_process_completion │ │ │ │ 775 │ │ │ │ │ line_length_limit=self.line_length_limit, │ │ 776 │ │ │ │ │ stream_buffer_size=self.stream_buffer_size, │ │ 777 │ │ │ │ ) │ │ ❱ 778 │ │ │ raise output_futures_failed.exception() │ │ 779 │ │ │ │ 780 │ │ # If all the output handlers completed without raising an │ │ 781 │ │ # exception, we still need to wait for all the underlying block │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/logging/utils.py: │ │ 229 in capture_subprocess_output │ │ │ │ 226 │ │ │ continue │ │ 227 │ │ │ │ 228 │ │ for writer in line_writers: │ │ ❱ 229 │ │ │ if not await _write_line_writer(writer, line): │ │ 230 │ │ │ │ # If the destination stream is closed, we can stop capturing output. │ │ 231 │ │ │ │ return │ │ 232 │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/site-packages/meltano/core/logging/utils.py: │ │ 197 in _write_line_writer │ │ │ │ 194 │ │ │ writer.write(line) │ │ 195 │ │ │ await writer.drain() │ │ 196 │ │ except (BrokenPipeError, ConnectionResetError): │ │ ❱ 197 │ │ │ await writer.wait_closed() │ │ 198 │ │ │ return False │ │ 199 │ else: │ │ 200 │ │ writer.writeline(line.decode(errors="replace")) │ │ │ │ /home/etienne/.pyenv/versions/3.11.9/lib/python3.11/asyncio/streams.py:364 in wait_closed │ │ │ │ 361 │ │ return self._transport.is_closing() │ │ 362 │ │ │ 363 │ async def wait_closed(self): │ │ ❱ 364 │ │ await self._protocol._get_close_waiter(self) │ │ 365 │ │ │ 366 │ def get_extra_info(self, name, default=None): │ │ 367 │ │ return self._transport.get_extra_info(name, default) │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ BrokenPipeErrorHere is my conf:
config: method: gcs_stage denormalized: true batch_size: 500 column_name_transforms: snake_case: false upsert: true generate_view: false schema_resolver_version: 2 project: $PROJECT dataset: $DATASET location: EU bucket: $BUCKETThe text was updated successfully, but these errors were encountered: