Expose "meta" model config dictionary on TaskInstance

[wseaton]

I'm working on integrating a PagerDuty setup with a Cosmos repo:

def on_failure_callback(context: Context):
    summary = f"Airflow: DBT DAG {context.get('task_instance').dag_id} task failure"
    severity = "error"
    source = context.get("task_instance").task_id

    dedup_key = f"{context.get('task_instance').dag_id}-{context.get('task_instance').task_id}-{context.get('execution_date')}"

    pd_hook = PagerdutyEventsHook(pagerduty_events_conn_id="default_pagerduty")
    pd_hook.send_event(
        summary=summary,
        severity=severity,
        source=source,
        group=context.get("task_instance").dag_id,
        dedup_key=dedup_key,
        links=[{"href": context.get("task_instance").log_url, "text": "Airflow Log"}],
    )

This works well, but I'd love to be able to have metadata from a model like the .meta.owner field or even the tags attached to the Context object to be able to more intelligently route alerts within my callbacks. Is this possible to implement?

[dosubot]

🤖

Hello @wseaton,

I'm Dosu, a friendly bot assisting with the Cosmos repository. I'm here to help you navigate through issues, answer your questions, and guide you on your contribution journey while we wait for a human maintainer.

I'm currently reviewing your issue regarding the exposure of the "meta" model config dictionary on TaskInstance. I'll provide a detailed response soon.

[dosubot]

🤖

Hello @wseaton,

I'm Dosu, a bot that is here to assist you with Cosmos repository. I'm designed to help with bug resolution, answer queries, and guide you on how to contribute to this project. While we wait for a human maintainer, feel free to ask me anything related to the repository.

Based on the current state of the Cosmos repository, there isn't a built-in mechanism to attach metadata to the TaskInstance object. The metadata is created for tasks and used to create Airflow tasks or task groups, but it's not being attached to the TaskInstance object. You can see this in the cosmos/airflow/graph.py file.

As for accessing the .meta.owner field or tags from the Context object, I wasn't able to find a direct way to do this within the Cosmos codebase.

However, it's possible to extend the functionality of Cosmos to include this feature. You could potentially modify the TaskInstance object to include the metadata you need, or create a new object that inherits from TaskInstance and includes additional metadata fields. This would require a good understanding of both the Cosmos and Airflow codebases, and careful testing to ensure that the changes don't introduce any unexpected behavior.

Please note that this is just a suggestion and might not be the best solution for your specific use case. It's also possible that there's a better solution that I wasn't able to find. If you need further assistance, please provide more details about your setup and requirements.

Sources

• cosmos/airflow/graph.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot]

Hi, @wseaton,

I'm helping the Cosmos team manage their backlog and am marking this issue as stale. From what I understand, the issue requests the exposure of the "meta" model config dictionary on TaskInstance to enable the use of metadata such as the .meta.owner field or tags within callbacks for more intelligent alert routing. Dosu has provided information on the current state of the Cosmos repository and suggested potential ways to extend the functionality of Cosmos to include the desired feature. They emphasized the need for a good understanding of both the Cosmos and Airflow codebases and careful testing to ensure that the changes do not introduce unexpected behavior.

Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you!