-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AIP 39: Documentation #17552
AIP 39: Documentation #17552
Conversation
ef28489
to
dd07a4b
Compare
I created an example DAG showcasing how timetable subclassing works and finished the howto guide referencing the example DAG. I’m more or less done for now, but since the howto’s content depends on changes in #17414, this will stay a draft until that PR is merged (or at least close to it). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is really clear to me (but then I'm familiar with both the code and the proposal, so I'm not the best to judge.)
Tests are failing because we haven’t actually implemented |
1c90094
to
2ffbf30
Compare
331694b
to
159b186
Compare
I think this is ready for review. In addition to the new timetable how-to page, I also rewrote many execution date and |
(Just rebased) |
Might need another rebase |
Not all usages can be removed, unfortunately, since there are still many usages requiring the value, especially in Python APIs. This also adds additional values for Sentry tags and Elastic log filenames based on data interval instead of execution date, so the user can migrate as soon as possible, even if we have not changed the default.
We need to use execution_date here :/
A DAG run CAN technically start before its data interval ends, but that's an edge case we shouldn't cover early in the documentation, so let's make wording vague to focus on the general cases to make things less complicated for users reading this for the first time.
A custom timetable requires plugin initialization, which is not done for unit tests. But we really don't want to automatically register that plugin since it makes testing an unregistered plugin impossible. Since that example DAG is very short and referenced only once anyway, let's just not bother.
This ensures the log ID is unique across the board, similar to the guarantee we had prior to AIP-39 (via execution date).
We don't want to expose TaskInstance(execution_date=...) anymore, but the extra link interface still relies on execution date, so we need an alternative.
Very much a work in progress.Ready for review now.I have very low confidence on my ability writing comprehensive prose 🙂