JavaScript heap out of memory

[zernie]

I'm getting a JavaScript heap out of memory error when trying to run tsc --noEmit.
Node version: v10.16.0
TypeScript: 3.4.3
ts-loader: 5.4.5
ts-node: 8.1.0

I guess the problem can be fixed by upgrading the dependencies to the latest versions and increasing max-old-space-size, but that's unfortunately not possible yet in my current project :(

---

<--- Last few GCs --->

[23026:0x4400980]    40880 ms: Scavenge 1384.8 (1423.9) -> 1384.4 (1424.4) MB, 12.3 / 0.0 ms  (average mu = 0.165, current mu = 0.085) allocation failure
[23026:0x4400980]    40893 ms: Scavenge 1385.1 (1424.4) -> 1384.7 (1424.9) MB, 7.9 / 0.0 ms  (average mu = 0.165, current mu = 0.085) allocation failure
[23026:0x4400980]    41493 ms: Mark-sweep 1385.4 (1424.9) -> 1384.9 (1424.9) MB, 595.0 / 0.0 ms  (average mu = 0.113, current mu = 0.056) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0xb91310dbe1d]
Security context: 0x1e0c9099e6e9 <JSObject>
    1: addPropertyToElementList(aka addPropertyToElementList) [0x21d642b90aa9] [/mnt/c/dev/node_modules/typescript/lib/tsc.js:~28448] [pc=0xb9131ad340a](this=0x0c2a182026f1 <undefined>,propertySymbol=0x16d1c4d04069 <Symbol map = 0x6cba608dd59>,context=0x262e61498bc1 <Object map = 0x15291fe33dc9>,typeElements=0x332e3ae4a389 <JSArray[0]>)
    2: createTy...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8f9d10 node::Abort() [node]
 2: 0x8f9d5c  [node]
 3: 0xaffd0e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xafff44 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xef4152  [node]
 6: 0xef4258 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xf00332 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xf00c64 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf038d1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xeccd54 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x116cede v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0xb91310dbe1d
[1]    23026 abort (core dumped)  npx tsc --noEmit

[Eyas]

Thanks for the report. Can you add a few details about how this is triggered:

1. Are you running TSC on this project, or a project that transitively includes schema-dts?
2. If the latter, are you importing "schema-dts" or "schema-dts-gen"?

[zernie]

@Eyas

1. yea
2. yea

It's as simple as

import { WithContext, Organization } from "schema-dts";

/*
 * @typedef {WithContext<Organization>} Schema
 */

[Eyas]

Hmm. I'm not having any luck repro-ing in newer versions, but I'll try with the exact Node/tsc versions to see if that would do it

[kkak10]

+1

I'm going through the same issue.

1. yes, i'm use this lib so simple way

just import and use typing

import { WithContext, Article } from "schema-dts";

const article: Article = {
...
};

2. also yes.

Is there anything I can do to help for this issue?

[kkak10]

My guess is that it issue when many types(maybe over 10 types??) are import and used.

It doesn't happen when you first use it, but it happens when you bring in a lot of types.

[zernie]

@Eyas here's the tsconfig and a snippet from the webpack config, hope that helps

{
  "compilerOptions": {
    "allowJs": true,
    "allowSyntheticDefaultImports": true,
    "checkJs": false,
    "diagnostics": false,
    "downlevelIteration": true,
    "esModuleInterop": true,
    "incremental": true,
    "importHelpers": true,
    "jsx": "react",
    "lib": ["dom", "es2017"],
    "module": "commonjs",
    "moduleResolution": "node",
    "noEmit": false,
    "noEmitHelpers": false,
    "noEmitOnError": false,
    "noErrorTruncation": true,
    "noImplicitAny": false,
    "noImplicitThis": false,
    "noImplicitUseStrict": false,
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "outDir": "dist",
    "preserveConstEnums": true,
    "removeComments": false,
    "resolveJsonModule": true,
    "rootDir": ".",
    "skipDefaultLibCheck": true,
    "skipLibCheck": true,
    "sourceMap": true,
    "strict": true,
    "target": "es5",
    "baseUrl": ".",
    "paths": {
      "@app/*": ["app/*"]
    }
  },
  "include": [
    "app/types/**/*",
    "app/index.js",
    "storybook/**/*",
    "tests/**/*"
  ],
  "exclude": ["code-generators", "dist"]
}
// webpack config:
rules: [
      {
        test: /\.[tj]sx?$/,
        include: path.resolve(process.cwd()),
        use: [
          {
            loader: "ts-loader",
            options: {
              transpileOnly: true,
              getCustomTransformers: () => ({
                before: [
                  createTransformer([
                    {
                      libraryName: "lodash",
                      libraryDirectory: null,
                      camel2DashComponentName: false,
                      style: false
                    }
                  ])
                ]
              }),
              compilerOptions: {
                module: "es2015"
              }
            }
          }
        ]
      },
]

[zernie]

@Eyas btw, the resulting typings file for schema-dts is huge - ~10k LOC. I wonder if breaking it into a smaller not interconnected pieces can help.

[Eyas]

Yeah, I'll need to see how it could be broken up. The thing is, it's auto-generated and a lot of the fields in schema.org reference each other liberally, we'll need to figure out a neat abstraction boundary where splitting makes sense.

[Eyas]

Hm. Another theory is that it isn't the file being too large, but the WithContext (which uses a generic upper-bound for it's T type parameter to extend Thing) might be too expensive. I'd be curious if:

import { Article } from "schema-dts";
type WithContext2<T extends {"@type": string}> = {
  "@context": "https://schema.org",
} & T;
const article: WithContext2<Article> = {
...
};

performed differently.

[iflix-erwin]

@Eyas We're getting the same issue even declaring our own WithContext2 same as what you mentioned. any updates on this?

[Eyas]

I'd love ideas to fix this, if anyone has time to dig. The issue here is that breaking the file into smaller files isn't very possible without creating some loopy dependencies. Schema.org schema can be very circular, so finding the right way to break it up isn't immediately clear to me.

[zernie]

@iflix-erwin what version of TS and node are you using? Have you tried upgrading?

[iflix-erwin]

@iflix-erwin what version of TS and node are you using? Have you tried upgrading?

@zernie Node: 12.4.0, TS: 3.5.3

[zernie]

@iflix-erwin what version of TS and node are you using? Have you tried upgrading?

@zernie Node: 12.4.0, TS: 3.5.3

Well that's unfortunate. I think we gonna need some help from TS developers

[simenbrekken]

Chiming in with the same problem reported from our CI. I'm importing and using the following types:

import { ItemList, ListItem, VideoObject, WithContext } from 'schema-dts';

[download13]

Right now this is a blocking issue for me. I'm wondering if there's any way we can build a version of the file that only contains the types we require for our project. It's a little clunky but would be better than our tools just crashing.

Is there a way to use schema-dts-gen to make a custom schema file with only a subset of types and their dependencies? I tried schema-dts-gen --ontology https://schema.org/Recipe.nt to just get the Recipe type for example, but it crashed with:

Error: No type found for Subject http://schema.org/recipe. Triples include:
http://schema.org/rangeIncludes: {"name":"rangeIncludes","context":{"href":"http://schema.org/","protocol":"http:","hostname":"schema.org","path":[""],"search":""},"href":"http://schema.org/rangeIncludes","type":"UrlNode"}
        => http://schema.org/Recipe

[Eyas]

@download13 Interesting, I was going to suggest this, but it looks like Recipe.nt doesn't just contain Recipe and its properties, but also properties who have the range Recipe (i.e. https://schema.org/recipe).

One way to fix this is to make sure that schema-dts-gen can ignore some properties. An --ignore_subjects= probably helps here.

You might still have issues here because Recipe.nt won't include definitions for the types of properties on Recipe, e.g. CreativeWork, MediaObject, Person, AlignmentObject, Thing, etc.

That last one there, Thing, is particularly problematic, since it results in potentially any other type being included. (FWIW, recipe.about, recipe.mainEntity, and recipe.mentions are all Things)

Let me think about a schema-dts-gen approach to this. I imagine some combination of:

• Specifying an "entry point" (e.g. Recipe)
• Tree Shaking (which alone doesn't help because anything that Includes Thing will result in nothing being pruned)
• Specifying wanted/unwanted types to be included.

might work. But it would be kind of hard to get set up.

[Eyas]

v0.5.0 tries to trim down some excess stuff there. Removes some ~800 lines of declaraitons, and makes some types readonly that in theory could help.

Those of you who have been blocked by this might want to try again.

[download13]

For me at least, the TS language server still won't return results to intellisense. It just keeps saying
Property '@type' does not exist on type 'Thing'

[matheusrezende]

I'm having the same problem, the TS server just crashes if I load this library and builds will timeout

[Eyas]

Do you have any information on your environment to reproduce it? e.g. TS version, OS, is this running in a container, etc?

Still haven't been able to repro this...

[download13]

Looks like this might've been fixed for us in typescript 3.9.0-beta

[Eyas]

Interesting! I think some of the type tightening around intersections dramatically reduces the number of different types TS needs to consider when evaluating

[jasongerbes]

Any updates?

I seem to be experiencing similar issues where importing types from this package stops intellisense from working and causes build timeouts

[Eyas]

Can you try Typescript 3.9? The current assumption is that this would be fixed by it

…

On Tue, Apr 28, 2020, 9:11 AM Jason Gerbes ***@***.***> wrote: Any updates? I seem to be experiencing similar issues where importing types from this package stops intellisense from working and causes build timeouts — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEHLOHPA2OAPRHMO4ULBJLRO3IYHANCNFSM4I32YGCA> .

[download13]

I was wrong. 3.9 definitely improved the performance, and it gets a lot farther before failing, but the language server still crashes

[Eyas]

@download13 @jasongerbes if you're using VS Code you might also want to try increasing memory for the TS Language Server, see microsoft/TypeScript#37214.

[Eyas]

I just pushed [email protected], I got some advice it might help.

[jasongerbes]

Hey @Eyas, I have just tested [email protected] and unfortunately the TypeScript compiler is still hanging

[Eyas]

@jasongerbes do you mind letting me know your:

• NodeJS version
• TypeScript version
• Whether tsc hangs or just tsserver

[jasongerbes]

@Eyas:

• NodeJS v12.16.1
• TypeScript v3.8.3
• Yeah, tsc hangs (seemingly) indefinitely

[Eyas]

thanks, and one more thing: is this on your local machine or some CI environment?

[jasongerbes]

Both locally (Windows and macOS) and on our CI environment

[Eyas]

Does this reliably repro with a minimal repro as simple as the one @kkak10 mentioned in #34 (comment) ? If the there's a different minimal repro for this, can you share it?

[nedkelly]

I've been having trouble with TypeDoc and up until today I was under the impression that typedoc was at fault, but after a lot of digging I reduced my failing docs build down to a small subset of components, they all had one thing in common, schema-dts.

So I created a minimal repo to test against and found that I could easily reproduce the issue. My project was using [email protected] and I was getting a JavaScript heap out of memory error, upgrading to [email protected] fixed a number of my files but still hangs on some.

TS: 3.9.7
Node: v12.18.0
Ubuntu 20.04 (Windows 10 WSL2)

Related TypeDoc issues:
TypeStrong/typedoc#835
TypeStrong/typedoc#1150

Minimal Example:
https://github.com/nedkelly/typedocissue

I hope this helps. We're actually almost at the point of stripping schema-dts out of our project due to its size and rigidity, it'd be good to see it broken down into more versatile chunks.

[Eyas]

Thanks @nedkelly -- My first thought is that this should still be a TypeDoc issue.

schema-dts has a very large type tree in part because it is trying to express a very and messy type system (schema.org). I got some help from the TS team to find a few ways to make the type tree more friendly to the TS compiler, and this will continue to be an area of work. But TypeDoc being able to traverse the existing type tree without OOM-ing probably also includes some optimizations/improvements on their own.

Your minimal example is probably helpful for TypeDoc themselves. And if someone from TypeDoc looks into this and has suggestions on reducing the cost of the schema-dts type tree, I'm all ears. But they might also identify areas for improvement within TypeDoc.

It might be helpful opening a separate issue against typedoc with your minimal example

[nedkelly]

@Eyas I agree, TypeDoc should definitely be able to traverse large type trees, I'll open an issue with them and see what sort of feedback I can get.

Opened here: TypeStrong/typedoc#1346

[Eyas]

Hi all -- I managed to reproduce a related issue more reliably, and changes in 0.8.1 should address it. I'm not sure if it's enough per se, but I'm noticing notable improvements on my end.

[Eyas]

This hasn't come up for the past few years. TS changes + typing simplifications have likely mitigated this over the long term. Let's file a new issue if something along these lines comes up for people in modern stacks.