[AIRFLOW-3160] Load latest_dagruns asynchronously, speed up front page load time #4005

aoen · 2018-10-05T14:34:08Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-3160] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3160
- In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-3160], code changes always need a Jira issue.

Description

The front page loads very slowly when the DB has latency because one blocking query is made per DAG against the DB. Even if we were to add indexes, the queries should still be batched to avoid an overhead of RTT * # of DAGs.

The latest dagruns should be loaded asynchronously and in batch like the other UI elements that query the database.

From my tests I was able to get the front page to reduce page load time of the front page from 8s to ~0.7s after this change with a DB in the west part of the USA with the Airflow cluster in the east part of the USA.

The load on the DB should also be slightly reduced due to this change as well.

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:
UI changes only, tested on local webservers and our prod hosts.

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

Passes git diff upstream/master -u -- "*.py" | flake8 --diff

cc @ashb

airflow/www_rbac/views.py

ashb · 2018-10-05T14:44:33Z

The load on the DB should also be slightly reduced due to this change as well.

Aren't we still making all the same queries?

What's the time to the page being fully populated too please?

aoen · 2018-10-05T15:10:08Z

Good call on the RBAC templates, will make a fix.

Aren't we still making all the same queries?

Queries are slightly different, in the past we made one query per DAG, now it's just a single query, so the constant overhead of each query is removed.

What's the time to the page being fully populated too please?

Locally the time for the page to be fully populated is pretty much instant. Prod is within a second, the query is pretty much a more lightweight version of the one being used to fetch and display task instance state on the same page so performance shouldn't be a concern.

aoen · 2018-10-05T21:11:37Z

@ashb addressed your comments, retrying CI now, but the two build jobs that failed out of nine were caused by CI flakiness.

codecov-io · 2018-10-05T21:48:54Z

Codecov Report

Merging #4005 into master will decrease coverage by 0.04%.
The diff coverage is 57.69%.

@@            Coverage Diff             @@
##           master    #4005      +/-   ##
==========================================
- Coverage   75.79%   75.74%   -0.05%     
==========================================
  Files         199      199              
  Lines       15946    15972      +26     
==========================================
+ Hits        12086    12098      +12     
- Misses       3860     3874      +14

Impacted Files	Coverage Δ
airflow/www/views.py	`69.06% <100%> (+0.2%)`	⬆️
airflow/www_rbac/views.py	`71.54% <26.66%> (-0.5%)`	⬇️
airflow/jobs.py	`82.13% <0%> (-0.27%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8be322...6607e48. Read the comment docs.

aoen · 2018-10-08T14:10:02Z

CI has passed, ready to be merged.

codecov-io · 2018-10-08T19:31:10Z

Codecov Report

Merging #4005 into master will decrease coverage by 0.04%.
The diff coverage is 57.69%.

@@            Coverage Diff             @@
##           master    #4005      +/-   ##
==========================================
- Coverage   75.79%   75.74%   -0.05%     
==========================================
  Files         199      199              
  Lines       15946    15972      +26     
==========================================
+ Hits        12086    12098      +12     
- Misses       3860     3874      +14

Impacted Files	Coverage Δ
airflow/www/views.py	`69.06% <100%> (+0.2%)`	⬆️
airflow/www_rbac/views.py	`71.54% <26.66%> (-0.5%)`	⬇️
airflow/jobs.py	`82.13% <0%> (-0.27%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8be322...6607e48. Read the comment docs.

aoen · 2018-10-15T15:16:28Z

Ready to merge FYI.

aoen · 2018-10-29T15:51:16Z

I'm happy to merge myself and take responsibility in fixing it if there is a breakage FYI.

@ashb

Fokko · 2018-11-05T18:11:51Z

Merged @aoen. High risk, high reward! 💪

aoen · 2018-11-05T22:54:03Z

@Fokko thanks! I owe you a PR review :).

ashb · 2018-11-06T14:27:41Z

@aoen Speak of the devil :D this seems to be causing test failures after this PR was merged, but only on MySQL somewhat odly. Can you take a look please, (we might revert this PR temporarily)

ashb · 2018-11-06T17:07:05Z

Examples that (we think) started happening after this PR was merged:

https://travis-ci.org/apache/incubator-airflow/jobs/451428747#L4660

in ERROR: test_success (tests.www_rbac.test_views.TestAirflowBaseViews) (postgres this time)

And same build on Mysql https://travis-ci.org/apache/incubator-airflow/jobs/451428748#L4660

Going to try reverting this PR and see if it fixes things, even though the error doesn't make any sense.

This reverts commit 0287cce.

…4145) * Revert "[AIRFLOW-3190] Make flake8 compliant" This reverts commit 1691c98. * Revert "[AIRFLOW-3160] Load latest_dagruns asynchronously (#4005)" This reverts commit 0287cce.

…" (apache#4145) * Revert "[AIRFLOW-3190] Make flake8 compliant" This reverts commit 1691c98. * Revert "[AIRFLOW-3160] Load latest_dagruns asynchronously (apache#4005)" This reverts commit 0287cce.

aoen · 2019-05-29T15:22:43Z

@ashb Tests were passing for me before I merged this several times, and it looks like the old logs are gone. Did the issue disappear after the commit was reverted? Does look suspicious I admit. This has been running without issue in our prod. Let me try to run this 10 or so times in CI and see if it's still failing, if it is not I propose remerging, and at least getting a traceback next time it fails in master CI to help fix (I can monitor CI for a while to make sure).

Also sorry for the late reply, need to figure out why I'm not getting email notifications anymore...

ashb · 2019-05-30T08:56:48Z

:D I don't remember exactly what was going on anymore, sorry. It was probable this PR wasn't at fault

aoen · 2019-05-30T15:01:11Z

@ashb Got it, I think it was a reasonable course of action, mind taking a look at the new PR? It's a lot tinier since non-RBAC has been deprecated, and the testing framework has been made simpler: #5339

…" (apache#4145) * Revert "[AIRFLOW-3190] Make flake8 compliant" This reverts commit 1691c98. * Revert "[AIRFLOW-3160] Load latest_dagruns asynchronously (apache#4005)" This reverts commit 0287cce.

…ously, speed up front page load time apache#4005

…synchronously, speed up front page load time apache#4005 (#7) * fb64f2e: [TWTR][AIRFLOW-XXX] Twitter Airflow Customizations + Fixup job scheduling without explicit_defaults_for_timestamp * reformat * 6607e48(airflow:master): [AIRFLOW-3160] Load latest_dagruns asynchronously, speed up front page load time apache#4005 * flake8 fix

… Default Retries and fix a small DAG refresh bug (#8) * fb64f2e: [TWTR][AIRFLOW-XXX] Twitter Airflow Customizations + Fixup job scheduling without explicit_defaults_for_timestamp * reformat * 6607e48(airflow:master): [AIRFLOW-3160] Load latest_dagruns asynchronously, speed up front page load time apache#4005 * a93d550: * a93d550: (HEAD, twitter/1.10+twtr) [TWTR][[AIRFLOW-4939]] Add Default Retries and fix a small DAG refresh bug (#3) (2 weeks ago) * flake8 fix

ashb reviewed Oct 5, 2018

View reviewed changes

airflow/www_rbac/views.py Show resolved Hide resolved

aoen force-pushed the ddavydov--lazy_load_last_dag_run branch 7 times, most recently from dfcaf50 to f5593a2 Compare October 5, 2018 20:11

aoen force-pushed the ddavydov--lazy_load_last_dag_run branch from f5593a2 to 62aa5a7 Compare October 8, 2018 17:44

[AIRFLOW-3160] Load latest_dagruns asynchronously

6607e48

aoen force-pushed the ddavydov--lazy_load_last_dag_run branch from 62aa5a7 to 6607e48 Compare October 8, 2018 17:57

aoen changed the title ~~[AIRFLOW-3160] Load latest_dagruns asynchronously~~ [AIRFLOW-3160] Load latest_dagruns asynchronously, speed up front page load time Oct 10, 2018

Fokko merged commit 0287cce into apache:master Nov 5, 2018

ashb added a commit to ashb/airflow that referenced this pull request Nov 6, 2018

Revert "[AIRFLOW-3160] Load latest_dagruns asynchronously (apache#4005)"

a4fc042

This reverts commit 0287cce.

wyndhblb pushed a commit to asappinc/incubator-airflow that referenced this pull request Nov 9, 2018

[AIRFLOW-3160] Load latest_dagruns asynchronously (apache#4005)

dc9b413

tmiller-msft pushed a commit to cse-airflow/incubator-airflow that referenced this pull request Nov 27, 2018

[AIRFLOW-3160] Load latest_dagruns asynchronously (apache#4005)

b80a537

elizabethhalper pushed a commit to cse-airflow/incubator-airflow that referenced this pull request Dec 7, 2018

[AIRFLOW-3160] Load latest_dagruns asynchronously (apache#4005)

2b16099

aliceabe pushed a commit to aliceabe/incubator-airflow that referenced this pull request Jan 3, 2019

[AIRFLOW-3160] Load latest_dagruns asynchronously (apache#4005)

bedad57

aoen mentioned this pull request May 29, 2019

[AIRFLOW-3160] (Unrevert) Load latest_dagruns asynchronously #5339

Merged

wmorris75 pushed a commit to modmed/incubator-airflow that referenced this pull request Jul 29, 2019

[AIRFLOW-3160] Load latest_dagruns asynchronously (apache#4005)

f1a42c8

vshshjn7 pushed a commit to vshshjn7/airflow that referenced this pull request Aug 28, 2019

6607e48(airflow:master): [AIRFLOW-3160] Load latest_dagruns asynchron…

d884791

…ously, speed up front page load time apache#4005

alexbegg mentioned this pull request Aug 28, 2020

Fix Start Date tooltip on DAGs page not showing actual start_date #10637

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-3160] Load latest_dagruns asynchronously, speed up front page load time #4005

[AIRFLOW-3160] Load latest_dagruns asynchronously, speed up front page load time #4005

aoen commented Oct 5, 2018 •

edited by Fokko

Loading

ashb commented Oct 5, 2018

aoen commented Oct 5, 2018 •

edited

Loading

aoen commented Oct 5, 2018

codecov-io commented Oct 5, 2018 •

edited

Loading

aoen commented Oct 8, 2018

codecov-io commented Oct 8, 2018

aoen commented Oct 15, 2018

aoen commented Oct 29, 2018

Fokko commented Nov 5, 2018

aoen commented Nov 5, 2018

ashb commented Nov 6, 2018

ashb commented Nov 6, 2018 •

edited

Loading

aoen commented May 29, 2019 •

edited

Loading

ashb commented May 30, 2019

aoen commented May 30, 2019

[AIRFLOW-3160] Load latest_dagruns asynchronously, speed up front page load time #4005

[AIRFLOW-3160] Load latest_dagruns asynchronously, speed up front page load time #4005

Conversation

aoen commented Oct 5, 2018 • edited by Fokko Loading

Jira

Description

Tests

Commits

Documentation

Code Quality

ashb commented Oct 5, 2018

aoen commented Oct 5, 2018 • edited Loading

aoen commented Oct 5, 2018

codecov-io commented Oct 5, 2018 • edited Loading

Codecov Report

aoen commented Oct 8, 2018

codecov-io commented Oct 8, 2018

Codecov Report

aoen commented Oct 15, 2018

aoen commented Oct 29, 2018

Fokko commented Nov 5, 2018

aoen commented Nov 5, 2018

ashb commented Nov 6, 2018

ashb commented Nov 6, 2018 • edited Loading

aoen commented May 29, 2019 • edited Loading

ashb commented May 30, 2019

aoen commented May 30, 2019

aoen commented Oct 5, 2018 •

edited by Fokko

Loading

aoen commented Oct 5, 2018 •

edited

Loading

codecov-io commented Oct 5, 2018 •

edited

Loading

ashb commented Nov 6, 2018 •

edited

Loading

aoen commented May 29, 2019 •

edited

Loading