Capture Operational Metadata

[cristianvasquez]

Description

Update the pipeline so that it captures the set of metadata fields needed to support common update operations.

Tasks

• Analyze and update the current list of metadata fields to allow scheduling specific sets of sources for transformation and enabling common update operations.
• establish a plan on how new metadata fields will be collected, grouped, and persisted within the current pipeline codebase. The outcome should update or replace the Metadata lifecycle documentation page.
• Implement the capture of the required metadata fields
• Remove obsolete/redundant metadata fields across the system.

Deliverables

• Metadata capture implementation plan and inventory.
• A task demonstrator: either a standalone application or a script that captures metadata when transforming a Ted-Mon package or equivalent.

References

• Metadata is accessed through interfaces provided by Provide interfaces to the metadata #553
• Metadata enables operations supported by Enable Data Operations for Updates #552.