Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: extended metadata annotations for @Transform, @Action and @HopMetadataProperty #3981

Open
bamaer opened this issue May 26, 2024 · 1 comment

Comments

@bamaer
Copy link
Contributor

bamaer commented May 26, 2024

What would you like to happen?

additional metadata will help to identify if a @Transform or @Action works on files, databases (relational and NoSQL), is input/output etc.
on the property level, additional metadata information can provide information if a @HopMetadataProperty works on files, relational database tables/columns etc.

Suggested changes:

  • add a new @ActionTransformType annotation that indicates the purpose for an action or transform. Possible values in the enum could include (not exhaustive):
    • INPUT
    • OUTPUT
    • FILE
    • RDBMS
    • GRAPH
    • NOSQL
  • extend HopMetadataProperty with a new metadata property type that is an array of a new HopMetadataPropertyType enum. Possibile initial values in this enum could include (not exhaustive):
    • FILE_PATH
    • FILE_NAME
    • FILE_WILDCARD
    • FILE_ENCODING
    • FILE_FIELD
    • FILE_SEPARATOR
    • FILE_ENCLOSURE
    • FILE_EXTENSION,
    • RDBMS_SCHEMA
    • RDBMS_TABLE
    • RBMS_COLUMN
    • RDBMS_SQL
    • RDBMS_TRUNCATE
    • RDBMS_SQL_SELECT
    • RDBMS_SQL_INSERT
    • RDBMS_SQL_UPDATE
    • RDBMS_SQL_DELETE
    • RDBMS_SQL_BULK
    • GRAPH_QUERY
    • GRAPH_NODE
    • GRAPH_RELATIONSHIP

Issue Priority

Priority: 3

Issue Component

Component: Actions, Component: Metadata, Component: Transforms

@bamaer
Copy link
Contributor Author

bamaer commented May 26, 2024

.take-issue

@github-actions github-actions bot added this to the 2.10 milestone May 26, 2024
bamaer added a commit to bamaer/hop that referenced this issue Jun 23, 2024
bamaer added a commit to bamaer/hop that referenced this issue Jun 23, 2024
bamaer added a commit to bamaer/hop that referenced this issue Jun 23, 2024
bamaer added a commit to bamaer/hop that referenced this issue Jun 23, 2024
bamaer added a commit to bamaer/hop that referenced this issue Jun 23, 2024
bamaer added a commit to bamaer/hop that referenced this issue Jun 27, 2024
hansva pushed a commit that referenced this issue Jul 5, 2024
* initial draft implementation of additional metadata for #3981

* resolved merge conflicts on delete transform. #3981

* additional metadata annotation updates. #3981

* minor changes. #3981

* resolved merge conflicts on delete transform. #3981

* added missing ASF headers. #3981

* additional metadata properties. #3981
bamaer added a commit to bamaer/hop that referenced this issue Jul 6, 2024
@hansva hansva modified the milestones: 2.10, 2.11 Sep 25, 2024
bamaer added a commit to bamaer/hop that referenced this issue Oct 26, 2024
bamaer added a commit to bamaer/hop that referenced this issue Nov 4, 2024
hansva pushed a commit to bamaer/hop that referenced this issue Nov 11, 2024
hansva added a commit that referenced this issue Nov 11, 2024
additional action, transform and property metadata annotations. #3981
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants