alternative idea: zero dependency ts parser / typechecker

[mrazauskas]

From #25

@SimenB Just to leave a note. I was thinking / working on a ts parser, which reads a file like this one:

test('is deprecated', () => {
  expect(user.id).type.not.toBeDeprecated();
  expect(user.accountName).type.toBeDeprecated('Use `id` property instead.'); // should fail
});

test('type equality', () => {
  expect({ a: 'b' }).type.toEqual<{ a: string }>();
  expect({ a: 'b' }).type.toEqual<{ a: number | string }>(); // should fail
});

test('type match', () => {
  expect('ten').type.toMatch<string | number>();
  expect({ a: 'b' }).type.not.toMatch<Record<string, number>>();
});

test('type error', () => {
  expect(one('fail')).type.toThrow('Expected 0 arguments, but got 1.');
  expect(one('fail')).type.not.toThrow(); // should fail
});

And returns:

{
  testPath: 'path/to/some.test.ts',
  testResult: [
    {
      failureReport: {
        expected: 'Use `id` property instead.',
        matcherName: 'toBeDeprecated',
        isNot: false,
        type: 'assertion',
        received: '',
      },
      fileName: 'path/to/some.test.ts',
      // fileText - full file source
      result: 'fail',
      testTitle: 'is deprecated',
    },
    {
      failureReport: {
        expected: '{ a: string | number; }',
        matcherName: 'toEqual',
        isNot: false,
        type: 'assertion',
        received: '{ a: string; }',
      },
      fileName: 'path/to/some.test.ts',
      // fileText - full file source
      result: 'fail',
      testTitle: 'type equality',
    },
    {
      result: 'pass',
      testTitle: 'type match',
    },
    {
      failureReport: {
        expected: '',
        matcherName: 'toThrow',
        isNot: true,
        type: 'assertion',
        received: '',
      },
      fileName: 'path/to/some.test.ts',
      // fileText - full file source
      result: 'fail',
      testTitle: 'type error',
    },
  ],
};

Not yet finished, but it works! Reporter and runner are the missing parts.

While working on this, I was thinking if having tests is really useful? It is, of course! Few reasons:

• scoped variables;
• .skip, .only, .todo
• no need of comments to organised tests;
• notes for future readers;
• etc

The solution I have at this moment:

• familiar jestish API;
• zero dependencies (only jest, its dependencies and typescript);
• zero config (jest.config and tsconfig.json do the job); (edit: it will have config to allow testing upon different versions of TypeScript's compiler)

Working on it. Will be posting updates here. This is not tsd based solution anymore, but I think it is related to jest-runner-tsd.

[SimenB]

If that works that'd make me very happy 🙂 Happy to host it here in the org if it works out!

[mrazauskas]

Yes, would great to host it in jest-community. I think it belongs here the best. Just have to finish the code (;

[mrazauskas]

Moving forward. I was stuck with error messages. Had to rewrite matcher’s logic twice to get what I wanted. The code base looks way better and here is very first fully working matcher:

[mrazauskas]

Will it also check deprecation messages? Sure!

[crutchcorn]

@mrazauskas this is incredible work - so excited to see it in action.

Is the source for what your working on available anywhere by chance? I'd love to see internally how you're doing things

[mrazauskas]

@crutchcorn Thanks. Currently code lives in private repo, but it will become public at some point. All updates will be posted here (;

[mrazauskas]

.toMatch is now .toBeAssignable. Still thinking which name is better for this matcher? In a way .toMatch is more Jest, but .toBeAssignable is more TypeScript.

At some point I added a comment next to .toMatch: // is assignable to. Did not like that, so just changed the name and got rid of the comment.

In the other hand, .type.toMatchSnapshot() could be added at some point. Why not, right? But will this look good with .toBeAssignable? Might be .toMatch was better idea.

The matcher works, but needs to be tested with more complex examples. So I will have time to think.

[TexKiller]

Hey, thank you all for working on this!

I just published something similar, but I used the Jest transform approach so Jest can run typing tests alongside regular tests with tsd under the hood: https://github.com/TexKiller/jest-tsd-transform
It is very simple right now, but it seems to be working well enough. Feel free to check it out and let me know what you think :)

I'm also looking forward to checking out @mrazauskas's approach

[mrazauskas]

Just to leave a note. I was working on the library recently. Here is what it currently does:

---

As you see, it has built-in mechanism to run tests upon different versions of TypeScript. For me that was a must feature to have. Very happy to look at it ;D

That was tricky to implement. I had to deal with CLI and config. The initial idea was to delegate those tasks to Jest, but user needs a way to define TS version somehow.

The code still needs work. Some parts are sketchy and must be cleaned up. Moving forward.

[SimenB]

This is super exciting stuff 😀

[mrazauskas]

One year later and... a lot have happened!

The library is nearly finished. Already some time ago it felt finished, but I wanted to integrate it with VS Code via an extension. That was a disappointment, because the public APIs I had were not doing right job. In parallel I was trying to run type tests in a larger repo and must admit that performance was just horrible.

There was no other way just to rewrite everything. The code is in better shape now. Before it took 3.2 secs to run all the type tests in libraries own repo, now that is just 0.7 sec. Almost hard to believe how slow it was ;D

[mrazauskas]

And here it is!

TSTyche – The Essential Type Testing Tool.

Repo: https://github.com/tstyche/tstyche
Documentation: https://tstyche.org

Few notable features of TSTyche:

• possibility to specify TypeScript versions to test on: tstyche --target 5.0,latest;
• helpers like test() and describe() which support .only, .skip run mode flags;
• expect style assertions to can compare two types, or two expressions, or an expression with a type;
• .toRaiseError() matcher to capture exact error message.

---

Currently it is published as beta. I would like to collect some feedback before releasing a stable release (planning to do so early next year). Please, give it try. If something looks clumsy, drop me a line here or open an issue in the repo.

In general the plan is to replace tsd-lite with tstyche. They both wrap around the same TypeScript's API to compare types. This is the only similarity, the rest is different.

The architecture of tstyche has much more capabilities. This was my goal. Also I spend lots of time to thoroughly document all what it can do. Hope this helps to migrate.

As a reference, here are the first migration PRs:

Jest
jestjs/jest#14687

RedwoodJS
redwoodjs/redwood#9394

Typegoose
typegoose/typegoose#894

.d.ts declaration testing in a JavaScript package:

docusaurus-remark-plugin-tab-blocks
mrazauskas/docusaurus-remark-plugin-tab-blocks#212

[PindaPixel]

And here it is!

TSTyche – The Essential Type Testing Tool.

Repo: https://github.com/tstyche/tstyche Documentation: https://tstyche.org

Few notable features of TSTyche:

* possibility to specify TypeScript versions to test on: `tstyche --target 5.0,latest`;

* helpers like `test()` and `describe()` which support `.only`, `.skip` run mode flags;

* `expect` style assertions to can compare two types, or two expressions, or an expression with a type;

* `.toRaiseError()` matcher to capture exact error message.

Currently it is published as beta. I would like to collect some feedback before releasing a stable release (planning to do so early next year). Please, give it try. If something looks clumsy, drop me a line here or open an issue in the repo.

In general the plan is to replace tsd-lite with tstyche. They both wrap around the same TypeScript's API to compare types. This is the only similarity, the rest is different.

The architecture of tstyche has much more capabilities. This was my goal. Also I spend lots of time to thoroughly document all what it can do. Hope this helps to migrate.

As a reference, here are the first migration PRs:

Jest jestjs/jest#14687

RedwoodJS redwoodjs/redwood#9394

Looks good! In the meantime I've migrated to Vitest, which provides ExpectTypeOf (https://vitest.dev/api/expect-typeof.html).

[mrazauskas]

expect-type is a nice library, but by design it can not capture error messages or run just one test file or have something like .only / .skip, etc.

Vitest's implementation is somewhat similar to jest-runner-tsd. Simply put, Vitest is not a type test runner. Can it run your tests on different versions of TypeScript? Nope. It just spawns tsc for you. This comes with issue like vitest-dev/vitest#3752 (not a problem for TSTyche)

Admittedly Vitest makes it possible to select files and provides some .only / .skip experience. But it is not a type test runner, so the skips are somewhat soft-skips.

In TSTyche skip means skip:

import { expect, test } from "tstyche";

test.skip("is skipped?", () => {
  const a: string = 123; // obviously a type error, but it will be skipped

  // ...
});

[mrazauskas]

Also worth to add that jest-runner-tsd or Vitest's expectTypeOf() are only available for users of one or another JavaScript testing framework. Why? What so special does a JavaScript framework provide for type testing?

In the beginning TSTyche was a plugable runner too. I wanted to make it to work with Jest, and Vitest, and any other framework which can have runners. This was interesting experiment, but I was struggling with that Why? all the time.

Why to spend time creating haste map? Why to have Vite next to tsc? Why a JavaScript testing framework should know how to install different versions of TypeScript?

At some point I realised that all what a type testing tool needs is to wrap around typescript! Simple and obvious. Just walk the AST to collect test nodes, retrieve type information, pass it to type checker, etc. Ah.. And what about those different TypeScript versions? npm is installing them (;

[mrazauskas]

TSTyche reached almost 1000 downloads on npm. Given that type testing is a very niche thing (hard to find a blogpost about it), that is way more than I expected. Thanks to everyone for giving it a try!

Time to close this issue.