Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema for issues.fields.customfield_* is not generalized #58

Open
akoebbe-uu opened this issue Dec 5, 2023 · 3 comments · May be fixed by #104
Open

Schema for issues.fields.customfield_* is not generalized #58

akoebbe-uu opened this issue Dec 5, 2023 · 3 comments · May be fixed by #104

Comments

@akoebbe-uu
Copy link

akoebbe-uu commented Dec 5, 2023

The schema defined in the IssueStream of this tap will not work for everyone because Jira uses something akin to EAV for custom fields. One project's issues.fields.customfield_10004 may be a string (how it's currently defined), but mine is a number. Therefore, the validation will always fail. Either the schema needs to allow for any datatype in issues.fields.customfield_*, or the tap needs to inspect Jira's field schema API to dynamically set the schema correctly for the instance it's extracting.

Field information API: /rest/api/3/field/ (Docs)

@prakharcode
Copy link
Contributor

Hi @akoebbe-uu , I think we can create some sort of schema generator for this tap but don't know if that's super handy. @edgarrmondragon I know that tap-salesforce does something like this.
But is it included in general plan of meltano, to have such schema generator when source provide a schema itself.

@akoebbe-uu
Copy link
Author

@prakharcode I think you're right. I'm pretty sure every instance of Jira will have different field types for custom fields. Meaning customfield_10006 may be the "Sprint" field for one Jira instance and "Story Points" for another. So I'm pretty sure this is mandatory.

The /rest/api/3/field endpoint will get the information needed to generate the schema. See: https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issue-fields/#api-rest-api-3-field-get

@akoebbe-uu
Copy link
Author

akoebbe-uu commented Jan 26, 2024

Here's a specific case where the extract fails because of the custom field mismatch.

In the tap code, customfield_10005 is defined as the following. This looks like the "Sprint" field schema.

tap-jira/tap_jira/streams.py

Lines 1680 to 1693 in 80638d8

Property(
"customfield_10005",
ArrayType(
ObjectType(
Property("id", IntegerType),
Property("boardId", IntegerType),
Property("name", StringType),
Property("state", StringType),
Property("goal", DateTimeType),
Property("startDate", DateTimeType),
Property("completeDate", DateTimeType),
)
),
),

But in an instance I'm trying to pull from the Jira API defines the schema as:

  {
    "id": "customfield_10005",
    "key": "customfield_10005",
    "name": "Business Value",
    "untranslatedName": "Business Value",
    "custom": true,
    "orderable": true,
    "navigable": true,
    "searchable": true,
    "clauseNames": [
      "Business Value",
      "Business Value[Number]",
      "cf[10005]"
    ],
    "schema": {
      "type": "number",
      "custom": "com.atlassian.jira.plugin.system.customfieldtypes:float",
      "customId": 10005
    }
  }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants