Only care about tests/ if no globs given #306

jackfranklin · 2018-10-24T19:18:43Z

We run our tests via elm-test 'frontend/**/*Tests.elm' as our app is a
big monorepo and we don't want our Elm tests in the top level tests
directory.

However, doing this gives us an error that the tests/ folder doesn't
exist. I updated the code to only check for and use tests/ if not
given a glob.

@rtfeldman let me know what you think!

We run our tests via `elm-test 'frontend/**/*Tests.elm'` as our app is a big monorepo and we don't want our Elm tests in the top level `tests` directory. However, doing this gives us an error that the `tests/` folder doesn't exist. I updated the code to only check for and use `tests/` if not given a glob.

rtfeldman · 2018-10-24T21:31:26Z

This is interesting!

One of my design goals with elm-test is: "You can open up any project and run the command elm-test and have it run all the project's tests."

That said, I've heard a few times that the hard requirement creates problems for people incrementally adopting Elm in existing applications.

Something we've talked about as a long-term goal is integrating elm-test into elm publish such that it can run all your tests, and your dependencies' tests, and tell you if what you're about to publish will break someone else's project. I definitely do not want to sacrifice that!

However, that concern only applies to packages, not applications. So here's how I'd like to proceed:

This is fine for applications. If elm.json has "type": "application" then it can use the glob as an alternative to tests/. It'll be less ergonomic in the sense that you can no longer run elm-test by itself, but it's presumably not the end of the world to invoke elm-test with the appropriate glob some other way.
I'd like packages to continue having the tests/ requirement. I don't see a use case for incrementally adopting a package, and I think having that requirement will pay off in the long run.

Thoughts?

jackfranklin · 2018-10-24T22:07:21Z

This is fine for applications. If elm.json has "type": "application" then it can use the glob as an alternative to tests/

Agree that this is good for applications and less good for packages 👍

It'll be less ergonomic in the sense that you can no longer run elm-test by itself,

The goal of this PR is to not do this - the goal of this PR is that elm-test still works and assumes the tests/ directory, but if you supply a glob, it no longer cares about it. I think that's the behaviour my changes create.

I'd like packages to continue having the tests/ requirement. I don't see a use case for incrementally adopting a package, and I think having that requirement will pay off in the long run.

Agree with this. For a package you have complete control of the repo and so this requirement makes sense.

I will update the PR - thanks for your thoughts!

jackfranklin · 2018-10-24T22:10:22Z

the goal of this PR is that elm-test still works and assumes the tests/ directory, but if you supply a glob, it no longer cares about it. I think that's the behaviour my changes create.

I am wrong! This changed the behaviour I think of running just elm-test in an application, so I need to fix that.

rtfeldman · 2018-10-24T22:22:58Z

Yeah, so I think the goal should be:

If you're a package, it should work exactly as pre-PR - if you don't have a tests/ directory, it errors regardless of whether you supplied a glob.
If you're an application, and you didn't supply a glob, and you don't have a tests/ directory, it also errors just like before.
If you're an application, and you did supply a glob, then it skips the tests/ directory check.

I think that should do the trick!

sporto · 2018-10-25T05:36:09Z

I quite like the original idea, why the need for a /tests folder in packages? Collocating the tests with the code is quite nice and it would be great if it is supported for both applications and packages.

jackfranklin · 2018-10-25T06:52:18Z

@rtfeldman I think I've now updated the code here to work as you explained it above. Let me know what you think!

lib/Generate.js

rtfeldman · 2018-10-25T12:40:49Z

lib/elm-test.js

@@ -302,15 +302,15 @@ if (args._[0] === "make") {
        ? 'No tests found for the file pattern "' +
          fileGlobs.toString() +
          '"\n\nMaybe try running elm-test with no arguments?'
-        : "No tests found in the tests/ directory.\n\nNOTE: Make sure you're running elm-test from your project's root directory, where its elm.json lives.\n\nTo generate some initial tests to get things going, run elm-test init";
+        : "No tests found in the tests/ directory.\n\nNOTE: Make sure you're running elm-test from your project's root directory, where its elm.json lives.\n\nTo generate some initial tests to get things going, run elm-test init.\n\nAlternatively, if you have tests in a different directory, try calling elm-test with a glob: elm-test 'frontend-app/**/*Tests.elm'.";


Maybe say "Alternatively, if your application has tests" instead? This advice won't work if they are working on a package!

Nice idea - updated 👍

rtfeldman · 2018-10-25T12:42:01Z

why the need for a /tests folder in packages? Collocating the tests with the code is quite nice and it would be great if it is supported for both applications and packages.

It's very easy to add flexibility and very painful to take it away once people have come to rely on it. In the application case there are different design constraints, in that people may not have total control over their directory structures. In the package case, they do.

So I'm convinced that "I am going to have a really hard time using elm-test in my application if this is how the directory structure has to be" justifies adding this flexibility, especially since applications aren't shared the way packages are.

However, I don't feel confident I grasp the full implications of that choice for packages. The consequences of making that change are serious; if I add the flexibility and then end up having to take it away, I will have caused a lot of pain by saying yes right now. That's why I want to leave packages as-is.

I hear you that removing the tests/ requirement sounds nice for package authors, and I'll keep it in mind for the future! This does not close the door to making that change later.

Co-Authored-By: jackfranklin <[email protected]>

jackfranklin · 2018-10-25T13:32:59Z

@rtfeldman nice comments, thank you! Updated the PR :)

mgold · 2018-10-26T05:21:13Z

lib/elm-test.js

@@ -302,15 +302,15 @@ if (args._[0] === "make") {
        ? 'No tests found for the file pattern "' +
          fileGlobs.toString() +
          '"\n\nMaybe try running elm-test with no arguments?'
-        : "No tests found in the tests/ directory.\n\nNOTE: Make sure you're running elm-test from your project's root directory, where its elm.json lives.\n\nTo generate some initial tests to get things going, run elm-test init";
+        : "No tests found in the tests/ directory.\n\nNOTE: Make sure you're running elm-test from your project's root directory, where its elm.json lives.\n\nTo generate some initial tests to get things going, run elm-test init.\n\nAlternatively, if your application has tests in a different directory, try calling elm-test with a glob: elm-test 'frontend-app/**/*Tests.elm'.";


I know you say "your application" but I think we should hide this when testing a package. (It's possible the surrounding code already does that.)

I don't think it does, but that seems like a good idea. 👍

jackfranklin · 2018-10-26T16:10:17Z

@mgold nice idea - just to check, is 58db81a what you had in mind?

rtfeldman · 2018-10-27T00:29:19Z

I'm assuming this is what you had in mind @mgold; I'm gonna merge so people can try beta11 over the weekend...got the last release blocker bug merged too, so I think this might be the last beta release!

mgold · 2018-10-27T03:53:27Z

I'm assuming this is what you had in mind

Yup!

This was referenced Oct 24, 2018

Collocate tests with src #291

Closed

Organizational options for elm tests in other opinionated ecosystems? #281

Open

jackfranklin added 4 commits October 25, 2018 07:38

Improve error message to hint at globs

f5a7e4e

Pass into generateElmJson if we were given globs

4953d64

Always add tests/ to source-directories

d34174e

Only change globbing for applications

62d2d88

Fix incorrect code and comment

eec3452

rtfeldman reviewed Oct 25, 2018

View reviewed changes

lib/Generate.js Outdated Show resolved Hide resolved

rtfeldman reviewed Oct 25, 2018

View reviewed changes

Richard Feldman and others added 2 commits October 25, 2018 14:31

Update lib/Generate.js

e6b5d28

Co-Authored-By: jackfranklin <[email protected]>

Make glob error message for elm apps

247d4d1

mgold reviewed Oct 26, 2018

View reviewed changes

only show extra test error for apps

58db81a

rtfeldman merged commit 00fcf1f into rtfeldman:master Oct 27, 2018

jackfranklin mentioned this pull request Nov 5, 2018

Fix bug adding tests/ to source-directories #315

Merged

yonigibbs mentioned this pull request Dec 24, 2019

Does folder containing tests have to be called "tests"? intellij-elm/intellij-elm#563

Closed

i-am-the-slime mentioned this pull request Jun 13, 2023

Support "Colocation" purescript/registry-dev#602

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Only care about tests/ if no globs given #306

Only care about tests/ if no globs given #306

jackfranklin commented Oct 24, 2018

rtfeldman commented Oct 24, 2018

jackfranklin commented Oct 24, 2018

jackfranklin commented Oct 24, 2018

rtfeldman commented Oct 24, 2018

sporto commented Oct 25, 2018

jackfranklin commented Oct 25, 2018

rtfeldman Oct 25, 2018

jackfranklin Oct 25, 2018

rtfeldman commented Oct 25, 2018

jackfranklin commented Oct 25, 2018

mgold Oct 26, 2018

rtfeldman Oct 26, 2018

jackfranklin commented Oct 26, 2018

rtfeldman commented Oct 27, 2018

mgold commented Oct 27, 2018

Only care about tests/ if no globs given #306

Only care about tests/ if no globs given #306

Conversation

jackfranklin commented Oct 24, 2018

rtfeldman commented Oct 24, 2018

jackfranklin commented Oct 24, 2018

jackfranklin commented Oct 24, 2018

rtfeldman commented Oct 24, 2018

sporto commented Oct 25, 2018

jackfranklin commented Oct 25, 2018

rtfeldman Oct 25, 2018

Choose a reason for hiding this comment

jackfranklin Oct 25, 2018

Choose a reason for hiding this comment

rtfeldman commented Oct 25, 2018

jackfranklin commented Oct 25, 2018

mgold Oct 26, 2018

Choose a reason for hiding this comment

rtfeldman Oct 26, 2018

Choose a reason for hiding this comment

jackfranklin commented Oct 26, 2018

rtfeldman commented Oct 27, 2018

mgold commented Oct 27, 2018