-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can rspectre be run in parallel? #23
Comments
Interesting consideration. None of the projects I work on have test suites that take longer than ~5 minutes to run so this hasn't been a priority for me so far. There's nothing fundamentally impossible about doing this, but it would require moving to a client-server kind of architecture or at least writing everything to a file. Right now I'm just mutating a global variable to track rspec invocations. Definitely something I'd consider, but it just may be a little more effort than I'm up for in the near future for a side-project. I do want to add some more features and improve some stuff soon, but I don't know if parallel runs will be a priority. |
@bquorning I know this is super stale but in your use case do you run 15 tasks on the same machine and so you something like a |
In my case, I use the parallel_tests gem to split a large spec suite into 45 separate jobs. They then run on 15 machines with 3 processes on each (saving the 4th CPU for mysql). Currently, everything is single threaded. It sounds quite complex to make rspectre work across multiple processes, let alone across multiple machines. I know that e.g. Simplecov can do a similar thing, by outputting each process’ result to a JSON file, and providing a tool for “merging” all the resulting files. But I totally understand if this is a problem is out of scope. |
Yeah, I use I would like to make this tool more accessible/usable for others but I'm not sure if I will invest in that or not. If you said single-machine parallelism would help you a lot I might have tried the narrower/simpler version but I think when your tests are big enough where this matters people probably want the produce-and-merge-artifacts workflow. The other way I see this working is just a very long cron job since I don't think it's worth running all your tests twice for this and I also don't think it'd be a good idea to use rspectre as a "real" test runner. The other angles I have thought about for approaching this:
|
I also wonder if I could do something (maybe slightly janky) where I just find all the specs which could likely include or are known to include a shared example and then only report when it seems like those have happened. That would still have false positives and one of the goals of this tool is to be very low on false positives... but it seems like there's not a great way to use it with large/slow test suites right now. |
I think it would be only worth the effort to parallelize if it could be integrated into the normal spec run in a convenient way - not as a "spec runner", but more as an rspec plugin of sorts. I agree that right now, as a standalone utility, approximately doubling CI usage for the average project seems rather wasteful to find the occasional obsoleted If it could run alongsid the regular spec run and there were ready-made github actions to build/collect/merge/validate the results, people could get its benefit on every build without much drag on the build time. It might be challenging to achieve the same ease of use as the standalone version has, though. I myself am quite happy to just run it once a quarter or so. In this case it is not affecting any normal pipeline, so IMHO it doesn't matter so much if it takes 5 minutes or 3 hours.
Another idea, or rather an additional idea: the execution of non-shared examples in a given spec file could be stopped once all |
Yeah. I have thought about this as well but I think I lean towards "I don't think I want to do this" on the grounds that this tool does some relatively invasive rewriting of
Huh, that's a really interesting idea. I would have to think about that in more detail because I could see it somehow leading to subtle false positives if I prune in the wrong cases (mainly thinking about cross-file shared example support right now) but in theory I should be able to detect if that's relevant. It might be too big of a project for an unknown gain... but I could also see it making a big difference on heavy-weight tests. Tracking in #82 😄 |
This could probably be avoided by using the TracePoint API instead of overrides, though perhaps at a greater performance cost. |
Yeah, I considered that when I created |
i did some basic benchmarking because i was interested in this for my own project. i guess the performance overhead in code that does any kind of IO will be barely noticeable. i agree about the awkwardness, though 😁 |
I am working on a large project where running the full test suite takes around 30 minutes. So we split into 15 tasks that are run in parallel.
It would be really neat to run rspectre as part of the test suite, but unless in can be parallelised, I would be adding a 30 minute job to an otherwise 2 minute CI run…
The text was updated successfully, but these errors were encountered: