Execute pex archive at runtime with -m parameter

[fhoering]

We would like to create a pex without a fixed entry point and provide different entry points at runtime.
The doc indicates that this can be achieved by tuning the environment variable PEX_MODULE

PEX_MODULE
Override the entry point into the PEX file. Can either be a module, e.g. ‘SimpleHTTPServer’, or a specific entry point in module:symbol form, e.g. “myapp.bin:main”.

https://pex.readthedocs.io/en/stable/api/index.html#module-pex.variables

This indeed works but is a bit counter intuitive.
It would be nice to provide the entry point with the -m argument similar to how the python interpreter works but this doesn't work
./mypexfile.pex -m mymodule arg0 arg1 arg2

Is this something that can be implemented or it there a technical restriction not to to this ? In theory as we already propagate the arguments in the main file we could also remove the beginning and redirect to the right module ?

[fhoering]

Providing a module as entry point that does the redirection (like runpy) actually works. Nevertheless shouldn't this be something implemented by default ?

pex -m runpy -o interpreter.pex --python=python3.6
./interpreter.pex http.server

[jsirois]

@fhoering maybe - it's slightly tricky. So clearly, if there is no entrypoint defined for the pex at build time, having pex look for -m at runtime steps on no ones toes. If an entrypoint is defined at build time though, pex should probably not stomp on -m, since the builtin entrypoint may expose a -m argument which should pass through unused by pex to the entrypoint. If you agree, then we have a perhaps strange split in pex runtime behvior - namely -m works sometimes, at other times you need to PEX_MODULE. Today you are assured PEX_MODULE always works so there is no ambiguity, with the downside the built pex doesn't act exactly like a python interpreter.

Do you have any idea for principles we could abide by to resolve this tension?

[fhoering]

I agree that having it parsed all the time and therefore users can't use the python interpreter options (-m -c, ..) anymore seems too intrusive.

Having it only set when no entry point is specified seems acceptable to me. When no entry point is specified it behaves like the python interpreter. Easy to explain as well.
Another option would be to provide some default template and activate it when invoking the pex command line tool with a special option.
Still another option would be to use the linux double dash syntax (--) to separate the arguments.

My use case is that I want to use pex to ship code to the pyspark executors. I started fixing pyspark code (fhoering/spark@23447d5 (pythonexec can be provided, so it would be mypackpackage.pex) but finally I think this should be included in the pex package. So I will provide a custom entry point using runpy that redirects to the according module.
I still think it should be possible by default or at least documented somehow. I personally prefer the double dash syntax but this won't work for my use case on pyspark.