Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Update stream metadata for all fields #102

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

ers81239
Copy link

@ers81239 ers81239 commented Jun 30, 2021

Problem

Field Level metadata is being stripped out by the refresh_streams_schema() function. This affects at least Meltano users who use the meltano select feature to populate the selected metadatum to indicate whether fields should be copied or not. In that case, the selected field is being stripped and therefore ignored and all fields get copied regardless.

Proposed changes

Previous code discovers schema and then copies only the stream-level metadata into the discovered schema and then replaces the catalog with the result.

This change updates the provided catalog with the discovered metadata for each field.

Fixes #101

Types of changes

What types of changes does your code introduce to PipelineWise?
Put an x in the boxes that apply

  • [ x] Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)

Checklist

  • [ x] Description above provides context of the change
  • [ x] I have added tests that prove my fix is effective or that my feature works
  • [ x] Unit tests for changes (not needed for documentation changes)
  • [ x] CI checks pass with my changes
  • Bumping version in setup.py is an individual PR and not mixed with feature or bugfix PRs
  • Commit message/PR title starts with [AP-NNNN] (if applicable. AP-NNNN = JIRA ID)
  • Branch name starts with AP-NNN (if applicable. AP-NNN = JIRA ID)
  • [ x] Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions

@Samira-El
Copy link
Contributor

Hi @ers81239 , thanks for this fix!
Can you please fix the CI?

@deanmorin deanmorin mentioned this pull request Oct 7, 2021
13 tasks
@elephantum
Copy link

@ers81239 @Samira-El I fixed CI for this PR in epoch8#1

Should I create a new PR or @ers81239 can pull from mine and update his PR?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Field-Level Metadata Removed
3 participants