Entirely remake typia for TS 5.2 transform API

[samchon]

microsoft/TypeScript#54276

TypeScript team had decided to support formal transform API since v5.2 update.

However, as TS v5.2 transform API provides types of typescript/lib/tsclibrary instead of typescript, typia must be entirely rewritten. Also, internal API of typia would be (not means global functions like typia.assert<T>(), but means internal modules like AssertProgrammer) entirely changed, therefore 3rd party libraries like nestia also need break change, too.

In such reason, supporting TS 5.2 transform API, typia would have another major version v4.

Release date of TS 5.2 may 2023-08, but typia v4 would be released a little bit earlier. It's because next typia will support both ts-patch and TS 5.2 transform API. When you run npx typia setup, it will configure tsconfig.json file properly considering TS version. If TS version is lower than 5.2, ts-patch would be utilized. Otherwise, pure typescript would be utilized.

[samchon]

@jakebailey As you don't want to fill more question contents in the TS issue, I ask you in here issue.

You told me to use ts.getJSDocTags() function instead of ts.Symbol.getCommentTags() function. However, the ts.getJSDocTags() function removes comment that is not matched with formal JSDoc spec. For example, when I write @type int comment tag, the int text would be erased in the ts.getJSDocTags() API.

This is not what I want, as typia is utilizing comment tags for validation. I can detour it just by parsing comments by myself with ts.getCommentRange() function. Of course, copying and pasting from typescript/src/services/jsDoc.ts is also possible. However, it seems something inefficient.

Is there any other way to solve this problem cleary? Am I missing something?

• https://typia.io/docs/validators/comment-tags/

[jakebailey]

TypeScript team had decided to support formal transform API since v5.2 update.

Just to temper expectations, it's just me proposing this, and it may not actually happen at all. I would't spend too much time on testing this unless you're really bored.

Is there any other way to solve this problem cleary? Am I missing something?

Are you just looking to get the raw JSDoc text?

[samchon]

const range: ts.TextRange = tsc.getCommentRange(node);
const text: string = TsNodeUtil.getSourceFile(tsc)(
    node,
).text.substring(range.pos, range.end);
return filter(text).join("\n");

@jakebailey I'm getting raw comment text through getCommentRage() function like above code. But if it is supported by TS API, it would be much better for typia, especially generating JSON schema's description field. By the way, what typia really wants is Symbol.getCommentTags() similar function, which does not erase comment like getJSDocTags() function.

• About @type int comment:
  • Symbol.getCommentTags() do not erase int text
  • However, getJSDocTags() function erase the int text, because it does not follow JsDoc spec.

[jakebailey]

JSDoc is a property of the declaration, so you should be able to just call ts.getJSDocCommentsAndTags on the whichever element of symbol.declarations is the right one. IIRC this is what TypeDoc does, and we made it public in TS 5.1. That's the PR I linked in the proposal thread.

[samchon]

JSDoc is a property of the declaration, so you should be able to just call ts.getJSDocCommentsAndTags on the whichever element of symbol.declarations is the right one. IIRC this is what TypeDoc does, and we made it public in TS 5.1. That's the PR I linked in the proposal thread.

I tried the ts.getJSDocCommentsAndTags() function following your guide, but it also erases int text from @type int tag. Currently, I'm avoiding such removal problem through hard coding like below. If try ts.getJSDocCommentsAndTags() function restoring commented codes, only [ '@type' ] be printed in console.

https://github.com/samchon/typia/blob/features/ts5.2/src/utils/TsSymbolUtil.ts

[jakebailey]

I see, then if you want to get at the text, you could just slice the SourceFile based on node.jsDoc.pos and node.jsDoc.end. I am not experienced in our JSDoc APIs to know what the right answer is when attempting to use TS to parse out non-standard tags.

[samchon]

Yes, as you say, I'm using String.substring() on SourceFile to avoid the problem.

Do you think erasing comment is a spec, or bug? If bug, I'll make a reproducable repo and report by writing issue on TS repo.

[jakebailey]

The tags we handle are the ones we have types for and store, so I don't know if this is really a bug. @sandersn would know better.

[sandersn]

Neither comments nor tags should be erased. Here's an example I tried:

interface Foo {
  /** @type int */
  type: number
}

and then at the REPL:

> var f = ts.createSourceFile('welove.ts', fs.readFileSync('welove.ts', 'utf8'), ts.ScriptTarget.ESNext, true)
> var jsdoc = ts.getJSDocCommentsAndTags(f.statements[0].members[0])
> jsdoc[0].tags[0].typeExpression.type.typeName.escapedText
'int'

A few comments:

1. createSourceFile needs a fourth argument true in order to set parent pointers; getJSDocCommentsAndTags relies on parent pointers being set. (Now that it's public the JSDoc should say that.) Thanks @jakebailey for helping me with this. If you're creating a full program, you have to request diagnostics since parent pointers are set as part of binding.
2. @type is parsed according to its normal rules, so int, even though it won't be resolved by the checker, is available as typeExpression.type.
3. Unrecognised tags are parsed as JSDocUnknownTag, which has tagName and comment properties. Then you can parse the comment yourself.
4. When parsing comment, remember that its type is string | NodeArray<JSDocComment>; you get the second alternative when the comment contains a @link tag, which is rare but could lead to confusing bugs.
5. You can double check whether jsdoc is actually getting parsed by looking at the internal jsDoc property on nodes. getJSDocCommentsAndTags do some additional lookup for @param tags and parameters, but for custom tags it's basically just returns the internal property.

interface Foo {
  /** @foundation int */
  type: number
}

and then at the REPL:

> var f = ts.createSourceFile('welove.ts', fs.readFileSync('welove.ts', 'utf8'), ts.ScriptTarget.ESNext, true)
> var jsdoc = ts.getJSDocCommentsAndTags(f.statements[0].members[0])
> jsdoc[0].tags[0].comment
'int'

[samchon]

@sandersn I wrote same code with you, but the comment value still be undefined.

As you know, I can't controll the createSourceFile() function, because I'm just utilizing TS 5.2 transform API.

May I believe and just wait for @jakebailey to configure setParentNodes to be true?

Or is there something else I should do to help?

• Reproducable repo: https://github.com/samchon/reproduce-TS5.2-transform-API-comment-bug
• Test Result: https://github.com/samchon/reproduce-TS5.2-transform-API-comment-bug/actions/runs/5030943564/jobs/9023700967#step:5:9
• Principal code: https://github.com/samchon/reproduce-TS5.2-transform-API-comment-bug/blob/master/lib/transform.js#L1-L19

const entire = tsc.getJSDocCommentsAndTags(node);
for (const elem of entire)
    if (tsc.isJSDoc(elem)) {
        const typeTag = elem.tags?.[0];
        if (
            typeTag &&
            tsc.isJSDocTypeTag(typeTag) &&
            tsc.isTypeReferenceNode(typeTag.typeExpression.type) &&
            tsc.isIdentifier(typeTag.typeExpression.type.typeName)
        )
            console.log({
                comment: typeTag.comment,
                expression:
                    typeTag.typeExpression.type.typeName.escapedText.toString(),
            });
    }

{ comment: undefined, expression: 'int' }

[sandersn]

I notice that you call program.getTypeChecker. Could you call getSemanticDiagnostics on that checker? That should also set parent pointers.

[samchon]

@sandersn

Found ts.Program.getSemanticDiagnostics() method, but it accepts only ts.SourceFile typed parameter. Therefore, I could not call ts.Program.getSemanticDiagnostics() function about the ts.TypeChecker instance. Instead, changed reproducable repo to call ts.Program.getSemanticDiagnostics() function about ts.SourceFile instance, and only empty array be returned.

Of course, if try to call ts.Program.getSemanticDiagnostics(ts.TypeChecker) function without considering type error, debug failure error be occured. Also, ts.TypeChecker does not have getSemanticDiagnostics() method, either. Am I missing something, or misunderstanding your order?

• Changed code: https://github.com/samchon/reproduce-TS5.2-transform-API-comment-bug/blob/master/lib/transform.js
• Result: https://github.com/samchon/reproduce-TS5.2-transform-API-comment-bug/actions/runs/5076147514/jobs/9118041392

$ npm start

> [email protected] start
> npx tsc --allowPlugins

checker has getSemanticDiagnostics() method? false
Try to call ts.Program.getSemanticDiagnostics(ts.TypeChecker) function...
Error: Debug Failure. False expression.
    at D:\github\contributions\reproduce-TS5.2-transform-API-comment-bug\node_modules\typescript\lib\tsc.js:119434:13
    at runWithCancellationToken (D:\github\contributions\reproduce-TS5.2-transform-API-comment-bug\node_modules\typescript\lib\tsc.js:119411:14)
    at getBindAndCheckDiagnosticsForFileNoCache (D:\github\contributions\reproduce-TS5.2-transform-API-comment-bug\node_modules\typescript\lib\tsc.js:119429:12)
    at getAndCacheDiagnostics (D:\github\contributions\reproduce-TS5.2-transform-API-comment-bug\node_modules\typescript\lib\tsc.js:119712:20)
    at getBindAndCheckDiagnosticsForFile (D:\github\contributions\reproduce-TS5.2-transform-API-comment-bug\node_modules\typescript\lib\tsc.js:119426:12)
    at getSemanticDiagnosticsForFile (D:\github\contributions\reproduce-TS5.2-transform-API-comment-bug\node_modules\typescript\lib\tsc.js:119421:33)
    at getDiagnosticsHelper (D:\github\contributions\reproduce-TS5.2-transform-API-comment-bug\node_modules\typescript\lib\tsc.js:119359:44)
    at Object.getSemanticDiagnostics (D:\github\contributions\reproduce-TS5.2-transform-API-comment-bug\node_modules\typescript\lib\tsc.js:119372:12)
    at Object.create (D:\github\contributions\reproduce-TS5.2-transform-API-comment-bug\lib\transform.js:29:21)
    at D:\github\contributions\reproduce-TS5.2-transform-API-comment-bug\node_modules\typescript\lib\tsc.js:119324:21
--------------------------------------------------
[]
{ comment: undefined, expression: 'int' }
[]
{ comment: undefined, expression: 'uint' }
[]
{ comment: undefined, expression: 'int64' }

[sandersn]

Looks like I mis-remembered the API. If you pass a SourceFile to getSemanticDiagnostics, are the jsdoc tags in that source file available afterward.

Here's some API examples: https://github.com/microsoft/TypeScript/wiki/Using-the-Compiler-API

[samchon]

@sandersn Tried, but comment be undefined, too.

Thanks for help, but I should just use ts.SourceFile.text.substring() method instead.