Resolve the modue names with "modulename.json" to json files when doing node module resolution #22167

sheetalkamat · 2018-02-24T18:21:34Z

Fixes #7071

sheetalkamat · 2018-02-26T22:41:15Z

@mhegazy c154b81 is the issue that was causing emptyArray to modify and my tests to fail.

calebboyd · 2018-03-08T04:59:54Z

This is great! Any chance it will make it into 2.8?

sheetalkamat · 2018-03-10T01:42:44Z

@mhegazy can you please take a look so we can get this in. Thank you.

sandersn

The binder and checker code look good. I'll have to look at the rest later.

One big question: why doesn't the flag default to true, at least for Javascript files? In fact, why is there a flag at all?

If there is a difference between JS and TS, then there should be a JS-specific test.

sandersn

Some comments and questions, plus I think this should default to true for JS. (Actually, it could probably default to true for TS if our main concern about this feature is about taking too long to compile very large json files.)

sandersn · 2018-04-26T21:13:13Z

src/compiler/commandLineParser.ts


        function convertObjectLiteralExpressionToJson(
            node: ObjectLiteralExpression,
            knownOptions: Map<CommandLineOption> | undefined,
            extraKeyDiagnosticMessage: DiagnosticMessage | undefined,
            parentOption: string | undefined
        ): any {
-            const result: any = {};
+            const result: any = returnValue ? {} : undefined;


is this returnValue check just for efficiency, to avoid building up a value that's not needed.

Actually, why isn't it needed, when existing callers needed it?

We dont need value when we are importing modules: We just want the error reporting on valid json syntax. So creating (potentially large) json object isnt needed hence the flag. In case of config file, we do need json object as well as checks which can be done simultaneously.

Could you add a comment on the function header?

sandersn · 2018-04-26T21:34:02Z

src/compiler/parser.ts

            // Set source file so that errors will be reported with this file name
            sourceFile = createSourceFile(fileName, ScriptTarget.ES2015, ScriptKind.JSON, /*isDeclaration*/ false);
-            const result = <JsonSourceFile>sourceFile;
+            const result = sourceFile as JsonSourceFile;


The new code modifies sourceFile directly. I don't think result is really used anymore, so I'd just return sourceFile at the end of this function and get rid of result..

sandersn · 2018-04-26T21:36:31Z

src/compiler/parser.ts

+            else {
+                const statement = createNode(SyntaxKind.ExpressionStatement) as JsonObjectExpressionStatement;
+                switch (token()) {
+                    case SyntaxKind.OpenBracketToken:


I take it that we didn't parse top-level arrays, etc, in json files until this change?

Would it be easier to parseExpression and then have an error for expression kinds that are not legal? I don't think we're checking the contents of, say, parseObjectLiteralExpression either, so there's not actually much point in being strict here.

The issue in that is we dont do better job with incorrect object literal expression .Eg. There is already test (https://github.com/Microsoft/TypeScript/blob/master/src/harness/unittests/projectErrors.ts#L116) for "files": something} where in the first bracket is missing. We cant default to object literal which would be very likely the contents in json

Oh, ok. I didn't realise that this approach gives better error recovery.

sandersn · 2018-04-26T21:43:28Z

src/compiler/program.ts

@@ -1983,7 +1983,7 @@ namespace ts {
                    }


interesting to see permissions changing from 755 to 644. Any idea why they were 755 before?

in the title for program.ts, it says that program.ts has 14 lines changed and also changed permissions from 100755 to 100644. I think that is safer since I don't think we can run program.ts from the command line, but I wasn't sure what prompted the change.

sandersn · 2018-04-26T21:46:25Z

src/compiler/types.ts

@@ -2608,10 +2609,23 @@ namespace ts {
    }

    export interface JsonSourceFile extends SourceFile {
-        jsonObject?: ObjectLiteralExpression;
+        statements: NodeArray<JsonObjectExpressionStatement>;


I don't get the reason for this -- ObjectLiteralExpression can contain methods, so JsonObjectExpressionStatement is not actually that restrictive. Why not just use ExpressionStatement here?

sheetalkamat · 2018-04-30T18:02:07Z

One big question: why doesn't the flag default to true, at least for Javascript files? In fact, why is there a flag at all?

Because we dont want to pull in big json files. Its almost too late most of the time when the json could result into big memory. It could be too many options or too long a string. it was discussed in design note to put this under the flag instead.
Having people to consciously opt into this would imply the user understands the cost.

sandersn

I think it's ready to go, although I'd still like to know whether JsonSourceFile.statements could have type NodeArray<ExpressionStatement>.

After our in-person discussion, I think it's ok to start with a flag required for both JS and TS files, and perhaps revisit this if there's enough demand for better types in config-free scenarios.

sandersn · 2018-04-30T22:40:58Z

src/compiler/parser.ts

+            else {
+                const statement = createNode(SyntaxKind.ExpressionStatement) as JsonObjectExpressionStatement;
+                switch (token()) {
+                    case SyntaxKind.OpenBracketToken:


Oh, ok. I didn't realise that this approach gives better error recovery.

sandersn · 2018-04-30T22:42:15Z

src/compiler/program.ts

@@ -1983,7 +1983,7 @@ namespace ts {
                    }


in the title for program.ts, it says that program.ts has 14 lines changed and also changed permissions from 100755 to 100644. I think that is safer since I don't think we can run program.ts from the command line, but I wasn't sure what prompted the change.

mhegazy · 2018-04-30T23:02:33Z

src/compiler/emitter.ts

@@ -1671,7 +1675,9 @@ namespace ts {

        function emitExpressionStatement(node: ExpressionStatement) {
            emitExpression(node.expression);
-            writeSemicolon();
+            if (!isJsonSourceFile(currentSourceFile)) {


why would this be called in the first place?

https://github.com/Microsoft/TypeScript/pull/22167/files#diff-9e36f50e46928676c6142f634cb7cd1bR18 when the output is copied into outDir just like any other js file ?

mhegazy · 2018-04-30T23:06:41Z

src/compiler/checker.ts

@@ -4685,6 +4685,11 @@ namespace ts {
                    return links.type = anyType;
                }
                // Handle export default expressions
+                if (declaration.kind === SyntaxKind.SourceFile) {


isSourceFile(declaration) and save the cast few lines in

mhegazy · 2018-04-30T23:08:25Z

src/compiler/types.ts

@@ -505,6 +505,7 @@ namespace ts {
        JSDoc                                         = 1 << 21, // If node was parsed inside jsdoc
        /* @internal */ Ambient                       = 1 << 22, // If node was inside an ambient context -- a declaration file, or inside something with the `declare` modifier.
        /* @internal */ InWithStatement               = 1 << 23, // If any ancestor of node was the `statement` of a WithStatement (not the `expression`)
+        JsonFile                                      = 1 << 24, // If node was parsed in a Json


i would just call it JSON

or Json for consistency

https://github.com/Microsoft/TypeScript/pull/22167/files#diff-4b8bd1eea29904f1be39cd864e1a45c0R489 has this as JavaScriptFile isnt it more consitent to use JsonFile ?

mhegazy · 2018-04-30T23:12:36Z

@andy-ms can you please review this change.

mhegazy · 2018-04-30T23:20:17Z

@sheetalkamat, a couple of comments:

I think we need a size limit as well. either in the module resolver or in the program. in .js files these can be big, and we do not want to kill the server if one (or few) were loaded.
we should enable it for js projects by default, in other words set --resolveJsonModules to true in getDefaultCompilerOptions in a jsconfig.json. and also in the server. we can do that in a separate PR if you want.

sheetalkamat · 2018-05-03T22:22:28Z

chatted with @mhegazy offline and the making --resolveJsonModules default to true for javascript files and limiting the size of files need to be handled separately since its going to involve effort thats generic even for javascript projects in general.

sheetalkamat · 2018-05-03T22:57:28Z

Ping @andy-ms @mhegazy for final review before merge

ghost · 2018-05-04T15:29:59Z

src/compiler/checker.ts

@@ -4718,6 +4718,11 @@ namespace ts {
                    return links.type = anyType;
                }
                // Handle export default expressions
+                if (isSourceFile(declaration)) {
+                    Debug.assert(isJsonSourceFile(declaration));
+                    const jsonSourceFile = <JsonSourceFile>declaration;


cast(declaration, isJsonSourceFile) helps avoid the type assertion

ghost · 2018-05-04T15:35:03Z

src/compiler/diagnosticMessages.json

@@ -3543,6 +3547,11 @@
        "code": 6196,
        "reportsUnnecessary": true
    },
+    "Resolve module name imported with '.json' extension to the json source file.": {


I think this message could be improved -- not obvious what the alternative is. "Type-check imported '.json' files" might be a better explanation. The same for the error message -- it's not the module resolution that users will care about, but the fact that we typecheck the contents of the file.

But that's not what it means though.. We wont even look for .json files if the flag is on. So its not just typechecking but picking up the modules?

How about "Include '.json' files in the program"?
The reason is that module resolution seems like more of an internal issue that most users won't care about. But they will care if their '.json' files are getting compiled or not.

somehow module needs to be part of the msg since aren't just including any json file?
Include modules imported with '.json' extension ?

Assuming that .json imports would have failed before, that's a better description.

ghost · 2018-05-04T15:44:02Z

src/compiler/commandLineParser.ts


        function convertObjectLiteralExpressionToJson(
            node: ObjectLiteralExpression,
            knownOptions: Map<CommandLineOption> | undefined,
            extraKeyDiagnosticMessage: DiagnosticMessage | undefined,
            parentOption: string | undefined
        ): any {
-            const result: any = {};
+            const result: any = returnValue ? {} : undefined;


Could you add a comment on the function header?

ghost · 2018-05-04T15:48:43Z

src/compiler/moduleNameResolver.ts

@@ -898,6 +903,11 @@ namespace ts {
     * in cases when we know upfront that all load attempts will fail (because containing folder does not exists) however we still need to record all failed lookup locations.
     */
    function loadModuleFromFile(extensions: Extensions, candidate: string, failedLookupLocations: Push<string>, onlyRecordFailures: boolean, state: ModuleResolutionState): PathAndExtension | undefined {
+        if (extensions === Extensions.Json) {


Seems like this could be handled along with the bottom case if (hasJavaScriptFileExtension)?

That just makes too many checks of extension !== Extensions.Json since we dont want to resolve "module" to module.json but only "module.json"

ghost · 2018-05-04T16:53:17Z

src/compiler/program.ts

@@ -2426,6 +2433,7 @@ namespace ts {
        switch (extension) {
            case Extension.Ts:
            case Extension.Dts:
+            case Extension.Json:
                // These are always allowed.


Assuming we would only ever get a .json module if the option was set? Maybe could put that in the comment.

ghost · 2018-05-04T16:55:56Z

src/compiler/utilities.ts

+        }
+    }
+
+    export function getTsConfigPropArrayElementValue(tsConfigSourceFile: TsConfigSourceFile, propKey: string, elementValue: string): StringLiteral {


Nit: StringLiteral | undefined

ghost · 2018-05-04T16:57:44Z

src/compiler/utilities.ts

+
+    export function getTsConfigPropArrayElementValue(tsConfigSourceFile: TsConfigSourceFile, propKey: string, elementValue: string): StringLiteral {
+        const jsonObjectLiteral = getTsConfigObjectLiteralExpression(tsConfigSourceFile);
+        if (jsonObjectLiteral) {


return jsonObjectLiteral && firstDefined(getPropertyAssignment(jsonObjectLiteral, propKey), property => isArrayLiteralExpression(property.initializer) ? find(property.initializer.elements, (element): element is StringLiteral => isStringLiteral(element) && element.text === elementValue) : undefined);

ghost · 2018-05-04T16:59:50Z

src/harness/harness.ts

@@ -1217,7 +1217,7 @@ namespace Harness {
            }

            const useCaseSensitiveFileNames = options.useCaseSensitiveFileNames !== undefined ? options.useCaseSensitiveFileNames : true;
-            const programFileNames = inputFiles.map(file => file.unitName);
+            const programFileNames = inputFiles.map(file => file.unitName).filter(fileName => !ts.fileExtensionIs(fileName, ts.Extension.Json));


Why filter these?

Because json files arent provided as fileNames but only picked during module resolution

ghost · 2018-05-04T17:01:22Z

src/services/getEditsForFileRename.ts

-        if (!cfg) return;
-        const oldFile = cfg.jsonObject && getFilesEntry(cfg.jsonObject, oldFilePath);
+        const configFile = program.getCompilerOptions().configFile;
+        const oldFile = getTsConfigPropArrayElementValue(configFile, "files", oldFilePath);


Nit: That function isn't declared to accept undefined inputs

mhegazy · 2018-05-04T19:29:14Z

src/compiler/program.ts

@@ -2200,6 +2200,12 @@ namespace ts {
                }
            }

+            if (options.resolveJsonModule) {
+                if (getEmitModuleResolutionKind(options) !== ModuleResolutionKind.NodeJs) {


what about --module? do not think we should allow this for --module ES2015 for example..

Clarified offline. At present --moduleResolution specified takes preferences for resolution irrespective of --module and since there is no change in there, no error needs to be reported specially.

mhegazy

One comment about --module and --resolveJsonModule relationship.

sheetalkamat added 10 commits February 23, 2018 13:54

Resolve Json file when module resolution strategy is node

5771adb

Replace usage of jsonObject on JsonSourceFile

ee2e267

Bind and resolve the module for json file

3572fad

Fix emit for json file

00b6c32

More tests and also do not add index signature for json module

a1922fd

Need allowJs to be true to use the json module resolution

ca590d6

Parse all json values at root

a790a92

Resolve the .json file only if module name contains the extension

4257c2f

Report errors about correctness of the json file

23a7e2f

Verify the output file paths for the json module emit

4f1640d

sheetalkamat requested a review from mhegazy February 24, 2018 18:21

sheetalkamat mentioned this pull request Feb 24, 2018

Support importing json files as modules with commonjs or node module resolution #13665

Closed

Ensure our readonly emptyArray stays non modified.

c154b81

Merge branch 'master' into requireJson

2071466

sheetalkamat force-pushed the requireJson branch from 256ef2a to 2071466 Compare March 8, 2018 21:25

Merge branch 'master' into requireJson

2d9af0b

sheetalkamat mentioned this pull request Mar 11, 2018

Enable the commented out test and convert folder entries to sorted list #22450

Merged

sheetalkamat added 2 commits March 12, 2018 16:22

Merge branch 'master' into requireJson

28a988d

Merge branch 'master' into requireJson

b4e0062

mhegazy mentioned this pull request Mar 31, 2018

Design Meeting Notes, 3/31/2018 #23045

Closed

sheetalkamat added 4 commits April 5, 2018 15:18

Merge branch 'master' into requireJson

9f72415

Resolve json modules only when --resolveJsonModule is specified

a143963

Reverted unintentional formatting changes

9d3ad54

Merge branch 'master' into requireJson

ce08af4

mhegazy mentioned this pull request Apr 26, 2018

In JS, importing JSON should not give an error #23712

Closed

sandersn reviewed Apr 26, 2018

View reviewed changes

sandersn mentioned this pull request Apr 26, 2018

Add prettier user test and fix associated crash #23715

Merged

sandersn requested changes Apr 26, 2018

View reviewed changes

sheetalkamat added 2 commits April 30, 2018 11:44

Merge branch 'master' into requireJson

579748b

PR feedback

c7b2d92

sandersn approved these changes Apr 30, 2018

View reviewed changes

mhegazy reviewed Apr 30, 2018

View reviewed changes

mhegazy requested a review from a user April 30, 2018 23:12

PR feedback

97df079

Merge branch 'master' into requireJson

15f9ea3

ghost reviewed May 4, 2018

View reviewed changes

ghost mentioned this pull request May 4, 2018

Simplify getOutputExtension #23889

Closed

ghost reviewed May 4, 2018

View reviewed changes

sheetalkamat added 2 commits May 4, 2018 10:45

PR feedback

4e6586d

Update the message for resolveJsonModule as per PR feedback

c4143ae

ghost approved these changes May 4, 2018

View reviewed changes

mhegazy reviewed May 4, 2018

View reviewed changes

mhegazy approved these changes May 4, 2018

View reviewed changes

mhegazy mentioned this pull request May 4, 2018

Support new flags (2.9) #23627

Closed

4 tasks

sheetalkamat merged commit 5133c70 into master May 4, 2018

sheetalkamat deleted the requireJson branch May 4, 2018 19:42

This was referenced May 4, 2018

Declare a type with a JSON file instead of an interface #23898

Closed

No completions for .json files with --resolveJsonModule #23899

Closed

Update schema for TS 2.9 flags SchemaStore/schemastore#451

Merged

microsoft locked and limited conversation to collaborators Jul 31, 2018

Resolve the modue names with "modulename.json" to json files when doing node module resolution #22167

Resolve the modue names with "modulename.json" to json files when doing node module resolution #22167

Conversation

sheetalkamat commented Feb 24, 2018

sheetalkamat commented Feb 26, 2018

calebboyd commented Mar 8, 2018

sheetalkamat commented Mar 10, 2018

sandersn left a comment

Choose a reason for hiding this comment

sandersn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sheetalkamat commented Apr 30, 2018

sandersn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhegazy commented Apr 30, 2018

mhegazy commented Apr 30, 2018

sheetalkamat commented May 3, 2018

sheetalkamat commented May 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhegazy May 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhegazy left a comment

Choose a reason for hiding this comment

mhegazy May 4, 2018 •

edited

Loading