KeyError: commit_timestamp

[visch]

If we follow the steps from Meltanos getting starting documentation here https://docs.meltano.com/getting-started/part1 but instead of using the singer variant we use this variant you end up getting an error.

This error is fixed by adding - commits.commit_timestamp to the select. This should probably be automatic in this tap? Probably all replication keys should be automatic especially if Incremental replication is enabled in the catalog.

2022-12-16 11:41:44,596 INFO METRIC: {"metric_type": "counter", "metric": "record_count", "value": 0, "tags": {"stream": "repositories", "context": {"org": "meltano", "repo": "meltano", "repo_id": 378997214}}}
Traceback (most recent call last):
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/bin/tap-github", line 8, in <module>
    sys.exit(cli())
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 509, in cli
    tap.sync_all()
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 381, in sync_all
    stream.sync()
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 1088, in sync
    for _ in self._sync_records(context=context):
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 995, in _sync_records
    self._process_record(
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 943, in _process_record
    self._sync_children(child_context)
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 1094, in _sync_children
    child_stream.sync(context=child_context)
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 1088, in sync
    for _ in self._sync_records(context=context):
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 1021, in _sync_records
    self._increment_stream_state(record, context=current_context)
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/singer_sdk/streams/core.py", line 725, in _increment_stream_state
    increment_state(
  File "/home/visch/git/meltano-projects/github_target_postgres/.meltano/extractors/tap-github-meltanolabs/venv/lib/python3.8/site-packages/singer_sdk/helpers/_state.py", line 220, in increment_state
    new_rk_value = to_json_compatible(latest_record[replication_key])
KeyError: 'commit_timestamp'

meltano.yml

version: 1
default_environment: dev
project_id: 4e114124-a67e-43d4-8cde-ac192c790b2b
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-github-meltanolabs
    capabilities:
      - state
      - catalog
      - discover
    namespace: tap_github
    pip_url: git+https://github.com/MeltanoLabs/tap-github.git
    executable: tap-github
    config:
      repositories:
      - meltano/meltano
      start_date: '2022-01-01'
    select:
    - commits.url
    - commits.sha
    settings:
      - name: access_token
        kind: password
      - name: auth_token
        kind: password
  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: -e /home/visch/git/target-postgres
    config:
      host: localhost
      user: postgres
      database: postgres
      default_target_schema: $MELTANO_EXTRACT__LOAD_SCHEMA
      ```

[ericboucher]

@visch Thanks for flagging this. I agree that it should be added automatically, but this feels more like an SDK feature than a tap feature. @edgarrmondragon what do you think? Is there an easy way for us to do this in the tap?

[edgarrmondragon]

@ericboucher @visch I have an old WIP branch for that. I'll try to make it a PR by EOD :)

[edgarrmondragon]

Is there an easy way for us to do this in the tap?

Can't think of anything other than overriding Stream.metadata to change the inclusion of the replication key.

[edgarrmondragon]

UPDATE: PR's here meltano/sdk#1283. LMK what you folks think :)

[ericboucher]

Closing this since the SDK version we pull now includes the above-mentioned PR.