A Hy script can't always import another file in its directory

[madhat2r]

Hy does not appear to be working when called as a cron job. Below is a simple test file I wrote when my real script would not execute via cron. I have verified that it is being called, but nothing appears to happen as the log file stays empty.

I wrote a fairly complex application for a client that needs to be scheduled via cron, so any help is much appreciated.

$ python --version
Python 3.6.2

$ hy --version
hy 0.13.0

test.hy:

#! /usr/bin/env hy

(import
  logging)
(def log-format "%(asctime)-15s %(levelname)-8s %(filename)-15s:%(lineno)-3s %(message)s")
(defn cli []
  (.basicConfig logging
                :filename "crontest.log"
                :level (. logging DEBUG)
                :format log-format)
  (setv l (.getLogger logging))
  (.info l "Running"))

(when (= --name-- "__main__")
  (cli))

cron log entries:

Nov 13 10:51:01 ArchNemesis CROND[29713]: (micah) CMD (/tmp/test.hy)
Nov 13 10:52:01 ArchNemesis CROND[31133]: (micah) CMD (/tmp/test.hy)
Nov 13 10:54:01 ArchNemesis CROND[3467]: (micah) CMD (/tmp/test.hy)
Nov 13 10:57:01 ArchNemesis CROND[9210]: (micah) CMD (/tmp/test.hy)

[Kodiologist]

What happens when you rewrite the simple test file in Python and test it the same way? If the same thing happens, then your issue can't be with Hy.

[madhat2r]

Thanks @Kodiologist for your reply.

I have done some more testing, and it appears this only happens when importing another module in the same directory. I have captured the error thrown from cron:

Traceback (most recent call last):
  File "/usr/bin/hy", line 11, in <module>
    load_entry_point('hy==0.13.0', 'console_scripts', 'hy')()
  File "/usr/lib/python3.6/site-packages/hy/cmdline.py", line 344, in hy_main
    sys.exit(cmdline_handler("hy", sys.argv))
  File "/usr/lib/python3.6/site-packages/hy/cmdline.py", line 332, in cmdline_handler
    return run_file(options.args[0])
  File "/usr/lib/python3.6/site-packages/hy/cmdline.py", line 211, in run_file
    pretty_error(import_file_to_module, "__main__", filename)
  File "/usr/lib/python3.6/site-packages/hy/cmdline.py", line 184, in pretty_error
    return func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/hy/importer.py", line 87, in import_file_to_module
    load_module(module_name))
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 248, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 608, in _exec
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/tmp/test_hy/test.hy", line 3, in <module>
    (import
ModuleNotFoundError: No module named 'logger'

keep in mind, this works when calling in the shell.

test_hy.zip

[Kodiologist]

I don't think logger.py and test.hy are equivalent—the Python version has a lot more code.

[madhat2r]

Why would they need to be equivalent.

Hy is breaking on importing logger.

[Kodiologist]

Sorry, I'd gotten confused and thought that logger.py was supposed to be a Python translation of test.hy rather than a library imported by it. Anyway, try rewriting test.hy in Python and see if you get the same error.

[madhat2r]

Remember this only breaks when called from a cron job. If you unzip to /tmp then you can use this in your crontab (crontab -e) which will run it and you can see the errors.

# */1 * * * * /tmp/test_hy/test.hy >> /tmp/test_hy/test.hy.log 2>&1

[madhat2r]

I have rewritten it in python and it works as expected.

[refi64]

Try running python -c 'import logger' from your cron job.

[madhat2r]

@kirbyfan64 what do you mean from my cron job? Do you mean in my hy file? or use it as a shell command from cronntab?

[Kodiologist]

I have rewritten it in python and it works as expected.

Okay, good, so this should indeed be a Hy bug. Can you provide the Python version?

[madhat2r]

test_hy_with_py.zip

Okay, good, so this should indeed be a Hy bug. Can you provide the Python version?

test.py: add it to the test_hy folder and change the cron job to execute this to test.

#! /usr/bin/env python

import logging
import click
from logger import Logger


@click.command()
@click.option("--debug", is_flag=True, default=False, help="Run in debug mode")
def main(debug):
    log_format = "%(asctime)-15s %(levelname)-8s %(filename)-15s:%(lineno)-3s %(message)s"
    logging.basicConfig(
        filename="crontest.log", level=logging.DEBUG, format=log_format)
    l = logging.getLogger()
    l.info("running")


if __name__ == '__main__':
    main()

[Kodiologist]

Thanks. I've replicated this with Hy master. I guess we aren't correctly emulating Python's search path. #459, our oldest outstanding bug, might be related.

[Kodiologist]

Oh wait, we can replicate this without cron:

$ cd /tmp/
$ mkdir foo
$ cd foo
$ echo 'print("in mymodule")' > mymodule.py
$ echo 'import mymodule' > testpy.py
$ echo '(import mymodule)' > testhy.hy
$ cd
$ PYTHONDONTWRITEBYTECODE=1 python3 /tmp/foo/testpy.py
in mymodule
$ PYTHONDONTWRITEBYTECODE=1 hy /tmp/foo/testhy.hy
[ … ]
ModuleNotFoundError: No module named 'mymodule'

[madhat2r]

Just posting this for someone who may need to solve similar problem.

As a work-around, I found that I could create a python file that just stubs out the CLI entry functions and that imports hy and then the hy module.

example

# test.py
import hy
from test_hy import run
def main():
    run()
if __name__ = '__main__':
    main()

;; test_hy.hy
(defn run[]
  (print "I work from cron, when called from py"))

[rkday]

$ mkdir bar
$ cd bar
$ mkdir -p /tmp/foo
$ echo "import sys; print(sys.path)" > /tmp/foo/testpy.py
$ python /tmp/foo/testpy.py
['/private/tmp/foo', '/Users/rkd/bar', '/Users/rkd/Envs/hy/lib/python36.zip', '/Users/rkd/Envs/hy/lib/python3.6', '/Users/rkd/Envs/hy/lib/python3.6/lib-dynload', '/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Users/rkd/Envs/hy/lib/python3.6/site-packages']
$ echo "(import sys) (print sys.path)" > /tmp/foo/testhy.hy
$ hy /tmp/foo/testhy.hy
['', '/Users/rkd/Envs/hy/bin', '/Users/rkd/bar', '/Users/rkd/Envs/hy/lib/python36.zip', '/Users/rkd/Envs/hy/lib/python3.6', '/Users/rkd/Envs/hy/lib/python3.6/lib-dynload', '/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Users/rkd/Envs/hy/lib/python3.6/site-packages']

So in the Python case the path includes the directory that testpy.py is in, and in the Hy case it includes the current directory and the directory that hy is in (presumably because to the Python interpreter, hy is in the same position that testpy.py is otherwise). The rest of the path is the same.

Adding sys.path = [os.path.dirname(filename)] + sys.path at https://github.com/hylang/hy/blob/master/hy/cmdline.py#L212 fixes it, but that feels a bit hacky - any better thoughts?

[Kodiologist]

It seems like a reasonable workaround, if nothing else. Fixing #459 might fix this, too; I'd focus on that first if I were you. Also, I wonder why the empty string appears in sys.path under Hy. Perhaps somewhere in Hy we're already doing sys.path manipulation to fix some related bug, but we got it wrong, so now it's adding the empty string instead.

[krosenmann]

@rkday No, it doesn't.

$ echo "import sys; print(sys.path)" > testpy.py
$ python testpy.py 
['/home/roman/sources/hyjango', '/home/roman/sources/hyjango/.venv/lib/python36.zip', '/home/roman/sources/hyjango/.venv/lib/python3.6', '/home/roman/sources/hyjango/.venv/lib/python3.6/lib-dynload', '/usr/lib/python3.6', '/home/roman/sources/hyjango/.venv/lib/python3.6/site-packages']
$ echo "(import sys) (print sys.path)" > testhy.hy
$ hy testhy.hy 
['', '', '/home/roman/sources/hyjango/.venv/bin', '/home/roman/sources/hyjango/.venv/lib/python36.zip', '/home/roman/sources/hyjango/.venv/lib/python3.6', '/home/roman/sources/hyjango/.venv/lib/python3.6/lib-dynload', '/usr/lib/python3.6', '/home/roman/sources/hyjango/.venv/lib/python3.6/site-packages']

And this is a reason:

ipdb> p os.path.dirname(filename)
''
ipdb> l
    207 
    208 
    209 def run_file(filename):
    210     from hy.importer import import_file_to_module
    211     from IPython.core.debugger import Tracer; Tracer()()
--> 212     sys.path = [os.path.dirname(filename)] + sys.path
    213     pretty_error(import_file_to_module, "__main__", filename)
    214     return 0
    215 
    216 
    217 def run_repl(hr=None, **kwargs):

ipdb>