Refactor

[tarpas]

This addresses issues: #53 #52 #51 #50 #32
partially #42 (poor mans solution)

With the limited test cases it worked and the performance was OK. If @blueyed @boxed tell me it works for them I'll merge and release.

[boxed]

Well it works fine with xdist as far as I can tell. Pretty cool!

[blueyed]

Cool!

Getting an exception, likely because of an old format in .testmondata?!

Traceback (most recent call last):
  File "…/pyenv/project/bin/py.test", line 11, in <module>
    sys.exit(main())
  File "…/pyenv/project/lib/python3.5/site-packages/_pytest/config.py", line 57, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "…/pyenv/project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "…/pyenv/project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "…/Vcs/pytest-testmon/testmon/pytest_testmon.py", line 82, in pytest_cmdline_main
    init_testmon_data(config)
  File "…/Vcs/pytest-testmon/testmon/pytest_testmon.py", line 76, in init_testmon_data
    affected = testmon_data.read_fs()
  File "…/Vcs/pytest-testmon/testmon/testmon_core.py", line 281, in read_fs
    self.read_data()
  File "…/Vcs/pytest-testmon/testmon/testmon_core.py", line 221, in read_data
    self.lastfailed = self._fetch_attribute('mtimes', default={}), \
  File "…/Vcs/pytest-testmon/testmon/testmon_core.py", line 178, in _fetch_attribute
    return json.loads(result[0])  # zlib.decompress(result[0]).decode('utf-8)'))
  File "/usr/lib64/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

[tarpas]

Sure, you have to delete .testmondata and start from scratch.

[blueyed]

Yep, that worked - should be handled though automatically, shouldn't it? (or display some error that it has to be moved/deleted).

Apart from that: it seems to behave better in my small testing.
Thanks!

[boxed]

The db went from ~10 meg to ~70 meg in my case. It's fine, but something to just keep tabs on.

Looking at just the modification times of files is the problem for me still I think. It would be better for my case if it could look at the git diff since the SHA the db is from. I know it might be a bit of a special case, but maybe it's worth having some different ways to calculate changed files? Git is pretty good at this...

This change is pretty great in my testing, even for the special case I am trying to use testmon for. You should definitely merge this!

[tarpas]

Thanks for feedback. Yeah, I disabled the compression so it's easier to poke around while debugging. I'm glad it's not a big problem for you. So I guess the mtimes of the identical file/versions checked out from git in different locations are different. It's trivial to add CRC/adler32 checksum implementation, no need for git dependency. I just didn't get around to do it.

[boxed]

Ah, yea, that seems like a better solution.

[boxed]

I'm going to continue writing my tooling here like it's already doing checksums on the files and eat the extra ~3 minutes of initial startup time I get because I don't have it for now I think.

[tarpas]

I think you would better of just implement the crc into testmon. I'm going to work on it now for 20 minutes, and share with you where I left of.

[boxed]

Great!

[tarpas]

Actually, it's a can of worms. I pushed something but it's shows the need to completely refactor the part of TestmonData into FileSystem and change a lot of things, including fixing the test suite so that it doesn't assume it's enough to change modified time of a file to force parsing...

[boxed]

Pity. Well, in any case, will you release a new version based on this branch in the mean time?

[boxed]

The answer is yes clearly :P Thanks!

[boxed]

Btw, wanted to say it works great now for our weird use case. Thanks again!