tsconfig.json is not applied in test folder in VSCode

[merisbahti]

Current Behavior

Extremely confusing, if I set strict: true in tsconfig, there’s no error for this under the tests/ in vscode:

const f = (input?: number | null | undefined) => {                                                                       
   return input + 1                                                                                                                            
 }

No error in the editor, but error when running tsdx test, this is super confusing, I kept looking for errors in my editor...

Expected behavior

I want it to give me the exact same error, as if it had been used in my code.

Suggested solution(s)

I don’t know, I just feel betrayed and have been searching for a solution for so long. It’s been a huge journey, how do you feel?

Maybe include a warning?

Maybe we strict should be the default, and we should include the tests folder in the include array?

Should we make strict mode the default?

Additional context

Really I might just have needed to vent.

Your environment

Hehe im coding through mosh... but here’s the specs:

System:
    OS: Linux 5.6 Arch Linux
    CPU: (2) x64 Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
    Memory: 466.16 MB / 3.85 GB
    Container: Yes
    Shell: 3.1.2 - /usr/bin/fish
  Binaries:
    Node: 14.3.0 - /usr/bin/node
    Yarn: 1.22.4 - /usr/bin/yarn
    npm: 6.14.5 - /usr/bin/npm
  npmPackages:
    tsdx: ^0.13.3 => 0.13.3 
    typescript: ^4.0.2 => 4.0.2 

[merisbahti]

Here’s the commit where my libs tests were fixed.

jotaijs/jotai-optics@2a427cf

Is there any reason we do not do the strict typechecking in tests? I cannot imagine any other formally typechecked language where the default typechecking is turned off by default in tests (scala, swift, purescript, haskell, etc) so I cannot imagine why we’d do that here?

[agilgur5]

Reproduced

Ah, the VSCode vs tsdx test difference is very important. This is an issue I was strongly suspecting, but could not reproduce in my VSCode for some reason -- hence the existence of #650 . I copied your code and was able to successfully reproduce:

tsdx test:

$ yarn test
yarn run v1.22.4
$ tsdx test
 FAIL  test/blah.test.ts
  ● Test suite failed to run

    TypeScript diagnostics (customize using `[jest-config].globals.ts-jest.diagnostics` option):
    test/blah.test.ts:10:10 - error TS2533: Object is possibly 'null' or 'undefined'.

    10   return input + 1
                ~~~~~

Test Suites: 1 failed, 1 total
[...]

VSCode:

Thanks for reproducing this @merisbahti, this is very helpful to verify my suspicions as well as add a test for this!

Explanation of differences

VScode reads your root tsconfig.json and uses that for type-checking. That one happens to not include the test/ dir. ts-jest, on the other hand, processes all sorts of test files so it seems to ignore the include and so runs against the root tsconfig.json and all its set options on all your test files.

Historical Context

This section may not be useful to anyone other than me, but it's important context to understand the problem at hand:

1. Initially reported in vscode doesn't use tsconfig.json settings for files in test folder #84 but was never responded to by maintainers at that time
   • vscode doesn't use tsconfig.json settings for files in test folder #84 (comment) has a simple workaround for this too
2. Duplicate was filed in tsconfig.json not applied to test directory #225 , with different wording there
3. A "fix" was applied in Add "test" directory to template tsconfigs' includes #226
4. This "fix" was quite short-sighted, because it created lots of other, more problematic issues, like test/ dir is in include, non-spec TS files (e.g. test utils) break build with v0.13 rootDir: ./src #638.
   • It probably also contributed to the rootDir issues of (fix): set rootDir to './src', not './'. deprecate moveTypes #504
   • test/ dir is in include, non-spec TS files (e.g. test utils) break build with v0.13 rootDir: ./src #638 (comment) has a better workaround for this, but it is more complex.
5. I reverted that "fix" in (fix): remove faulty tsconfig.json include of test dir #646 in order to fix test/ dir is in include, non-spec TS files (e.g. test utils) break build with v0.13 rootDir: ./src #638.
   • As I wrote there, I was unable to reproduce tsconfig.json not applied to test directory #225 and vscode doesn't use tsconfig.json settings for files in test folder #84 in my VSCode, so I did not add more changes along the lines of test/ dir is in include, non-spec TS files (e.g. test utils) break build with v0.13 rootDir: ./src #638 (comment)
     • Here's a screenshot of what I failed to repro:
       
       • Not sure if I had the exact same repro back then (on a totally different computer and VSCode install) ...But looking at it now I also might have just misread my own repro... it's giving a noImplictAny warning, but not an error -- it should be an error with strict
6. I created Add VS Code Integration Tests #650 as a request for VSCode integration tests to make this more reproducible and doable in CI (as opposed to local, personal IDEs that may be heavily configured)
7. I mentioned this in stories dir is not being type checked during Storybook build #668 (comment) with regard to the stories dir instead of the test dir, but that actually seemed to be asking about Storybook itself (or Webpack running underneath actually) not type-checking, not VSCode.
   • change rootDir to ./src cause trouble for storybook custom /.storybook/config.ts with require.context (Storybook <5.3) #641 mentions it too, but it was using an outdated version of the templates / Storybook, as was use @storybook/preset-typescript and support MDX on Storybook template #440 (comment)

Might be missing some other places I've mentioned this. Has been something I've been looking out for in the issues, but hadn't seen a clear repro until now.

Have labeled as both regression and feature as it's somewhere in between. #84 and #225 were features, but #226 was buggy so was reverted by #646. So in that way you can either say it was never truly fixed (since the fix was buggy), or that #646 created an intentional (and at the time, unreproducible) regression in order to fix a worse bug.

Workaround

So there are several workarounds for this.

1. vscode doesn't use tsconfig.json settings for files in test folder #84 (comment) is the easiest, but not the most complete.
2. test/ dir is in include, non-spec TS files (e.g. test utils) break build with v0.13 rootDir: ./src #638 (comment) is more complete, but also a bit more complex. It currently requires passing a flag to tsdx build, but I'd like to add support for that by default.

Next Steps

1. Add a failing test(s) for this in the test suite now that it has been successfully reproduced.
   • I don't think this is possible without Add VS Code Integration Tests #650 though. We could test tsc itself since VSCode seems to follow the same behavior under-the-hood, but it's a little opaque. That being said, we do also want tsc --noEmit to error out here by default when used as a type-checker. Very related is to implement Linting should include type checking #352 (comment)
2. Change the templates to use test/ dir is in include, non-spec TS files (e.g. test utils) break build with v0.13 rootDir: ./src #638 (comment)
3. Add out-of-the-box support for tsconfig.build.json so that it doesn't require using the --tsconfig flag since it'll be the default and recommended in the templates.

[merisbahti]

This is the best response I’ve ever read, thank you.