Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[2.4.1] stricter generic checks causes out-of-memory crash #16777

Closed
emmanueltouzery opened this issue Jun 27, 2017 · 26 comments · Fixed by #17984
Closed

[2.4.1] stricter generic checks causes out-of-memory crash #16777

emmanueltouzery opened this issue Jun 27, 2017 · 26 comments · Fixed by #17984
Assignees
Labels
Bug A bug in TypeScript Fixed A PR has been merged for this issue

Comments

@emmanueltouzery
Copy link

emmanueltouzery commented Jun 27, 2017

TypeScript Version: 2.4.1
node version: v6.10.3, linux x86-64

Code
https://github.com/emmanueltouzery/ng-typeview

clone the project, run:

  • npm install
  • tsc

Expected behavior:
the project compiled just fine up to typescript 2.3.

Actual behavior:
with 2.4.1, I get an out of memory at every run:

<--- Last few GCs --->

   44131 ms: Mark-sweep 1338.5 (1416.3) -> 1334.0 (1420.3) MB, 2112.0 / 0.0 ms [allocation failure] [GC in old space requested].
   46537 ms: Mark-sweep 1334.0 (1420.3) -> 1334.0 (1384.3) MB, 2406.1 / 0.0 ms [last resort gc].
   48675 ms: Mark-sweep 1334.0 (1384.3) -> 1334.0 (1384.3) MB, 2137.8 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x38bd260cfb39 <JS Object>
    1: set [native collection.js:~252] [pc=0x1a7da7795be3] (this=0x237efe5f2b39 <a Map with map 0x16e45b60a191>,r=0x3ebe4546afa1 <String[15]: 1359681,1359683>,z=1)
    2: /* anonymous */ [/home/emmanuel/.npm-packages/lib/node_modules/typescript/lib/tsc.js:~977] [pc=0x1a7da7953990] (this=0x163899f5ce41 <JS Global Object>,value=1,key=0x3ebe4546afa1 <String[15]: 1359681,1359683>)
    3: arguments ...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0xde09ee [node]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
 7: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [node]
 8: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::EnsureGrowable(v8::internal::Handle<v8::internal::OrderedHashMap>) [node]
 9: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [node]
10: 0x1a7da6c092a7
tsc --version
[1]    5209 abort (core dumped)  tsc

I've seen similar bug reports but no obvious solution (a bug suggested to increase the typescript dependency version, tried to increase that to 2.4.1, didn't help, tried to remove the typedoc dependency which has a transitive dependency to typescript, that didn't help either), and in some of these bugs typescript maintainers ask for the source and reporters couldn't provide it: here I can.

@IrickNcqa
Copy link

Same problem - different project. I'm using the angular template from the "Microsoft ASP.NET Core JavaScript Services" @ https://github.com/aspnet/JavaScriptServices, and was getting the exact same error while using Webpack's HMR in Node.

In trying to reproduce the problem from a minimal startup, I may have solved the problem in my case.

It appears that the rxjs/Subject.d.ts file no longer has valid definitions with the improved generic typechecking in v2.4.1. When I try a stock template, I get the following error in the browser:

ERROR in [at-loader] ./node_modules/rxjs/Subject.d.ts:16:22 TS2415: Class 'Subject<T>' incorrectly extends base class 'Observable<T>'. Types of property 'lift' are incompatible. Type '<R>(operator: Operator<T, R>) => Observable<T>' is not assignable to type '<R>(operator: Operator<T, R>) => Observable<R>'. Type 'Observable<T>' is not assignable to type 'Observable<R>'. Type 'T' is not assignable to type 'R'.

Alright, so that leads me to the skipLibCheck flag in the tsconfig.json file. The template includes the skipDefaultLibCheck flag set to true, but doesn't include the skipLibCheck flag. Double checking the flags at https://www.typescriptlang.org/docs/handbook/compiler-options.html, skipDefaultLibCheck is now deprecated in favor of skipLibCheck.

So, I simply replaced skipDefaultLibCheck with skipLibCheck (still set as true). That fixed the browser error about the d.ts file, and it also fixed the heap out of memory error in node (I'm guessing Webpack was silently failing on build because the now invalid d.ts file was being included, which led to the memory issue in node).

I checked your tsconfig.json file, and you don't have the skipLibCheck flag set to true. Maybe give that a shot and see if it fixes your problem too?

@myknbani
Copy link

myknbani commented Jun 28, 2017

Affects me too when I bumped TS to 2.4.1. Works fine with 2.3.4

@emmanueltouzery
Copy link
Author

@IrickNcqa you're correct! Setting up skipLibCheck in my tsconfig.json gives me now normal build errors, which I assume are correct and due to the increased strictness of TS2.4.
My dependencies are quite up to date though. But I guess I must live with the flag for some time if I want to use 2.4.

@myknbani
Copy link

I can confirm that setting skipLibCheck to true works.

@myknbani
Copy link

Nevertheless, shouldn't this be considered a regression, since we did not need the skipLibCheck option before 2.4.1?

@emmanueltouzery
Copy link
Author

@myknbani regardless of anything, tsc should never crash like it does in this bug. at the very least, even if the type definitions are invalid, it should display a proper error message.

That said, from my understanding, skipLibCheck won't be needed anymore once the libraries are updated for typescript 2.4. I do wonder whether we have worst type-safety for the broken libraries compared to TS2.3.x in the meantime though?

@IrickNcqa
Copy link

I'd say it's to be expected that stricter type checking would result in some new errors in existing type definition files. Given the above, it's clear rxjs needs to update their d.ts file, and in another thread it looks like the Redux d.ts is going to need an update also. There will probably be a lot of others.

So this will probably boil down to filing issues with the third parties to update their definition files or fixing ourselves and submitting. The major packages will probably be updated fairly quickly. TS v2.4.1 only dropped 20 hours ago - so they definitely haven't had time yet.

A bad d.ts file leading to a out of memory error is bizarre. Agree that needs improvement.

@IrickNcqa
Copy link

Looks like the heap error has been around. I just saw it in the v2.4.2 milestone, but no discussion of the skipLibCheck there. https://github.com/Microsoft/TypeScript/issues/16234

@emmanueltouzery
Copy link
Author

for my app i'm not even sure which dependencies cause the issue. I suspect maybe immutable but I reproduce also removing that dependency. So a good error message should tell us which dependency is to blame otherwise it's going to be complicated to fix.

what I'd really like to know as well is, what is the downside of skipLibCheck?

does that make the type-checking of the libraries "lazy" and... if you don't use the feature which is badly mapped in the type definition then you have no problem and no regression in type-safety from 2.3. If you do use the feature which is badly mapped, then you still get the crash..

OR does it disable some type-safety and you are worse off on 2.4 with that option compared to staying on 2.3?

but yeah whether it's #1 or #2 it's probably better not enabling the option and sticking to 2.3 until the type definitions are ported to 2.4...

@sandersn
Copy link
Member

sandersn commented Jun 28, 2017

I did a git bisect and found that it is a bug in the stricter generic checks introduced after 2.4.0 (at commit 368e5af).

For me, --noStrictGenericChecks also circumvents the crash, without sacrificing any other kinds of type checking. That's the workaround I'd recommend until the bug is fixed.

@sandersn sandersn changed the title [2.4.1] FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory [2.4.1] stricter generic checks causes out-of-memory crash Jun 28, 2017
@sandersn
Copy link
Member

sandersn commented Jun 28, 2017

By the way @IrickNcqa, #16234 is almost certainly a different issues since it predates the stricter generic type checking. There are lots of ways the Typescript compiler can run out of memory!

@axefrog
Copy link

axefrog commented Jun 29, 2017

@sandersn Just upgraded from 2.3.x to 2.4.1 and started getting the same error. I can confirm that adding the noStrictGenericChecks flag to my tsconfig successfully circumvents the issue.

@emmanueltouzery
Copy link
Author

I see this bug is not in the 2.4.2 milestone. I think it should be, because it hampers libraries becoming compatible to 2.4, since it makes it hard for them to understand what's wrong with their type definitions. It seems like a pretty big regression.

@benjamincombes
Copy link

Same problem for me. I tried with version 2.3.4, 2.4.1, notStrictGenericChecks and skipLibCheck, I still get the crash.
Here is the error I get:


[52324:0x104001600]   121844 ms: Mark-sweep 1418.7 (1657.5) -> 1418.7 (1657.5) MB, 399.3 / 0.0 ms  allocation failure scavenge might not succeed
[52324:0x104001600]   122261 ms: Mark-sweep 1418.7 (1657.5) -> 1418.8 (1641.5) MB, 416.5 / 0.0 ms  last resort
[52324:0x104001600]   122698 ms: Mark-sweep 1418.8 (1641.5) -> 1418.8 (1641.5) MB, 435.3 / 0.0 ms  last resort


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3a13bc129891 <JS Object>
    1: DoJoin(aka DoJoin) [native array.js:~96] [pc=0x7c3faf560ed](this=0x2c06d9182311 <undefined>,p=0x3dd6a137341 <JS Array[150]>,q=150,E=0x2c06d91823b1 <true>,A=0x2c06d91dc5f1 <String[1]:  >,z=0x2c06d9182421 <false>)
    2: Join(aka Join) [native array.js:~121] [pc=0x7c3fc83aee8](this=0x2c06d9182311 <undefined>,p=0x3dd6a137341 <JS Array[150]>,q=150,A=0x2c06d91dc5f1 <String[1]:  >,z=0x2c06d918...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 6: v8::internal::Runtime_StringBuilderJoin(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
 7: 0x7c3f978437d
Abort trap: 6```

@sandersn
Copy link
Member

#17984 fixes the crashes I observed trying to compile ng-typeview and monet. For the others, you can typescript@next starting tomorrow to see whether the fix applies.

@mhegazy mhegazy added the Fixed A PR has been merged for this issue label Aug 23, 2017
@myitcv
Copy link

myitcv commented Aug 24, 2017

@sandersn - using:

$ tsc --version
Version 2.6.0-dev.2017082

I'm getting an a "heap out of memory" crash:

<--- Last few GCs --->

[92152:0x2d18600]    75250 ms: Mark-sweep 1408.2 (1510.9) -> 1408.1 (1491.9) MB, 3022.1 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 3022 ms) last resort
[92152:0x2d18600]    78273 ms: Mark-sweep 1408.1 (1491.9) -> 1408.1 (1491.9) MB, 3022.4 / 0.0 ms  last resort


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x246a2c41bbd9 <JS Object>
    1: set [native collection.js:~247] [pc=0x340ce51f52ba](this=0xeb0a3e61ec9 <a Map with map 0x226b3f014319>,p=0x3c6f900dd919 <String[8]: 889900:2>,x=0x3401e985a811 <a Type with map 0x226b3f0585d1>)
    2: instantiateTypeNoAlias(aka instantiateTypeNoAlias) [/home/myitcv/mgo/src/modelogiq.com/frontend/node_modules/typescript/lib/tsc.js:~24926] [pc=0x340ce58be0ae](this=0x246a2c402241 <undefined>...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x136849c [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
 7: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [node]
 8: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::EnsureGrowable(v8::internal::Handle<v8::internal::OrderedHashMap>) [node]
 9: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [node]
10: 0x340ce50040bd
Aborted (core dumped)

If I set noStrictGenericChecks to true then all seems well. Is this the right "fix"?

@mhegazy
Copy link
Contributor

mhegazy commented Aug 24, 2017

@myitcv mind filing a new issue and sharing your project

@myitcv
Copy link

myitcv commented Aug 24, 2017

@mhegazy - happy to file a new issue, but we can't share the code unfortunately.

@mhegazy
Copy link
Contributor

mhegazy commented Aug 24, 2017

We would be happy to sign any NDA required to get access to the source. without the sources, or a sample reproduction of the issue, i am afraid there is not much we can do.

@sandersn
Copy link
Member

@myitcv please do file a new issue to track this. Even if you can't share the code, we are planning to improve the way the compiler fails when the cache grows too large, so you may be able to debug the problem yourself.

In the meantime, --noStrictGenericChecks is the best workaround.

@myitcv
Copy link

myitcv commented Sep 18, 2017

@sandersn - since #18231 landed (although I haven't narrowed it down to that exact PR, that just happened to be a significant PR that I saw land) the problem has gone away; no --noStrictGenericChecks required. Hence no need to open another issue.

At this point, the only outstanding issue prevent us moving back to the nightly build is something else you're working on, namely immutable-js/immutable-js#1285. I wonder if you or @leebyron can help move that along?

@sandersn
Copy link
Member

Immutable is looking for new maintainers right now so I'm not sure @leebyron will have time. One possibility is to decouple the types and put immutable-js's types back on DefinitelyTyped. But that's not a good long-term strategy without agreement from the maintainers of immutable.

@myitcv
Copy link

myitcv commented Sep 18, 2017

Immutable is looking for new maintainers right now...

That's not the conclusion I took from @leebyron's comments in immutable-js/immutable-js#1215 (comment).

There are of course ways in which I can use your proposed PR changes for now.

But I was hoping given that we're on Typescript 2.5 and that these breaking changes landed in 2.4 that we'd be able to move things along.

@amitm02
Copy link

amitm02 commented Apr 3, 2018

I experience this issue in [email protected] when using [email protected]

@mhegazy
Copy link
Contributor

mhegazy commented Apr 3, 2018

@amitm02 this is rather an old issue. please file a new ticket, and give us enough context to be able to diagnose the issue.

@microsoft microsoft locked and limited conversation to collaborators Jul 25, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Bug A bug in TypeScript Fixed A PR has been merged for this issue
Projects
None yet
9 participants