Script for automatically "smoke testing" against a set of libraries, looking for regressions.

[jamestalmage]

#108 (comment)

I think this could be an awesome general purpose module. My thoughts:

• It needs a simple config file that lists the projects that depend on the one you want to "smoke test".
• Mimic .travis.yml for configuring scripts (just the script names from the travis Build Lifecyle, not all their config options).
• The config file would need to point to the repository and branch to clone.
• The module would clone each individual downstream dependent in a temp directory and run the scripts described above, npm linking the module under test.
• If there are any failures, it will open the temp folder in your Finder/File Explorer, and list the failing dependents (maybe deleting the folders of passing dependents).

Folder structure:

some/temp/dir/
   downstream-dependent-1/
     ...
   downstream-dependent-2/
     ...

// @bcoe @novemberborn

I think we have a need for something like this for AVA as well.

// @sindresorhus @vdemedes

[sindresorhus]

👍 We should definitely have this for AVA. I currently do it manually once in awhile and before a release.

Prior art: https://github.com/bahmutov/dont-break // @bahmutov

[jamestalmage]

Cool. Let's try using that and see how it goes.

[bahmutov]

I will be happy to help

[bcoe]

@jamestalmage I like this idea, and I could see it being a useful hosted service like travis 👍

I've only thought about it a bit, but some things I'd like to be able to do:

• run another test-suite with an arbitrary library linked in against it (the library you're testing).
• it would be nice to be able to assert to CLI output.
• it would be nice to be able to assert that a file exists, and that it has certain content in it.
• it would be nice to be able to assert that an HTML, JSON, or XML file has specific fields.

[jamestalmage]

@bcoe - instead of building all that in, what if the config file just added a validate field that pointed to a script that was passed the output directory of the automated run?

[coreyfarrell]

I'm cleaning up some old issues. I think this is a good idea, but I feel like this is outside the scope of nyc. Maybe this belongs in https://github.com/nodejs/tooling/issues instead? Could we leverage dont-break for this (either as a dependency or by contributing required features to dont-break)?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.