allow using requirments.txt in PythonVirtualEnvOperator

[alexInhert]

Currently the operator allows to set requirement as list that needs to be hard coded.
It would be nice if airflow can support reading from file directly (something similar to how operators read sql file)

[mik-laj]

Can you see if you can pass ["-r", "requireements.txt] as. requirements?

[potiuk]

How about this (I know it's not perfect, but I think it should do the job):

requirements = (lambda: open("requirements.txt").readlines())()

[uranusjr]

The requirements file format supports some things that can’t be passed into the command line, so the open-readline approach can only partially solve the issue. Making things worse, the file format is internal to pip, so there’s no way for the user to parse it on their own, the file must be passed into pip. So some way to support this would be nice. (I think I read another issue on this before…?)

I believe requirements=["-r", "requireements.txt] will work. Pretty hacky though.

[potiuk]

Agree. Having a good, templated parameter could work. Now when I think obout it might actually be as simple as adding this to the operator and handling the case where requirements is also a string and properly using it with -r/temp file by the operator in this case:

    template_fields = ("requirements",)
    template_ext = (".txt",)

The template_ext is one of the lesser-known features of Airflow operators

[uranusjr]

Or maybe we can do something similar to bash_command? If a requirements argument looks like a file (os.path.isfile() or contains os.sep), automatically insert a -r in from of it.

[potiuk]

I thinkl it the same feature we are talking about :).

The "template_ext" works in the way that it will read the file as string when the parameter has the extension registered there and it will automatically process the file trough the template engine, so you can add jinja-variables from context /xcom variables in the file. It's built-in feature in BaseOperator

All that you need is in this case to have the right extension (".txt") in the value of the parameter.

requirements="[<path>/]reqiurements.txt"

[potiuk]

If the parameter is not string or does not have the extension it is processed as usual (i.e. passing list of strings will continue to work - but it will additionally process the strings through template engine (since "requirements" will be in "template_fields" tuple) so you could also add jinja variables in the strings in the array and they will be interpolated.

[alexInhert]

But in this case you want the functionality of template_ext without the template_fields
There is no need to go through jinja engine

[potiuk]

But in this case you want the functionality of template_ext without the template_fields
There is no need to go through jinja engine

But this might be very useful. For example you can use jinja variable for version of a dependency and get it from another task. or have conditional requirements.txt content based on dev/prod settings etc.

This is pretty useful feature IMHO.

[alexInhert]

I think in your use case the whole requirement file is in fact not a requirement.txt but a value pushed to xcom by previous task.

[potiuk]

I think in your use case the whole requirement file is in fact not a requirement.txt but a value pushed to xcom by previous task.

No. I am talking about "requirement.txt" file in which some values might be provided by XCom.

I can imagine this kind of requirement.txt file for example:

dependency1==1.0.0
dependency2==2.0.0
dependency3=={{ code to retrieve depedency version from xcom}}
dependency4==1.1.0
{{- % if retrireve deployment == 'production'  % }}
# only needed in production deployment
dependency5==5.0.0
{{- % endif % }}

[alexInhert]

ah cool!!
This is even better than what I had in mind

[vijaykiran]

I'd like to pick this (as part of the contributors-workshop)

[rounakdatta]

Hello @potiuk @alexInhert, I have tried an implementation of the above discussions, can you review once? Thanks.