Add another way to dynamically generate DAGs to docs

[makrushin-evgenii]

This PR created as a result of discussion #21278

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: [email protected]
  Slack: https://s.apache.org/airflow-slack

[dstandish]

Hi @makrushin-evgenii

I like this example. Taking a look at this, it seems this section on "dynamic dag generation" is getting rather long and starting to look like more of a "how to" than a "best practices".

I think it might be best if we create a "how to" page called "dynamic dag generation" or perhaps "Config-based dag generation". Then in the "best practices" page we can keep a more succinct message about what really is the "best practice" we're trying to emphasize in this area, and we can link to the how-to ("for more details on dag generation, see our how-to...").

Would you be interested in takking this on as part of this PR?

[github-actions]

The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

Hi @makrushin-evgenii

I like this example. Taking a look at this, it seems this section on "dynamic dag generation" is getting rather long and starting to look like more of a "how to" than a "best practices".

I think it might be best if we create a "how to" page called "dynamic dag generation" or perhaps "Config-based dag generation". Then in the "best practices" page we can keep a more succinct message about what really is the "best practice" we're trying to emphasize in this area, and we can link to the how-to ("for more details on dag generation, see our how-to...").

Would you be interested in takking this on as part of this PR?

Might be a good idea actuallly if you want to do it @makrushin-evgenii :)

[makrushin-evgenii]

Hi @makrushin-evgenii

I like this example. Taking a look at this, it seems this section on "dynamic dag generation" is getting rather long and starting to look like more of a "how to" than a "best practices".

I think it might be best if we create a "how to" page called "dynamic dag generation" or perhaps "Config-based dag generation". Then in the "best practices" page we can keep a more succinct message about what really is the "best practice" we're trying to emphasize in this area, and we can link to the how-to ("for more details on dag generation, see our how-to...").

Would you be interested in takking this on as part of this PR?

You're right. Want to do it this way, but forgot about the "How To" page. I'll make changes in a few days

A few clarifications:

1. Should I move all "dynamic dag generation" section content from "best practies" page to "how to", or just mine? If just mine, should I copy all "dynamic dag generation" section content to "how to"?
2. Can I use a links to external resources, e.g. Astronomer guide "Dynamically Generating DAGs in Airflow"?

[dstandish]

I was thinking move all of it but just keep any salient points that really are "best practices". I think the key point there was really that if you are generating a dag from a config to be mindful of network IO because dags are parsed frequently ( though less so now than in the past). IIRC everything else is more subjective, but still helpful examples to have in a how-to. But of course apply your own judgment too about what makes sense.

I think external links are fine

[potiuk]

Could you please rebase this one @makrushin-evgenii if it fails?

[makrushin-evgenii]

@potiuk, i fixed all failed checks. What else should I do?

[boring-cyborg]

Awesome work, congrats on your first merged pull request!