ts-loader is hanging while consuming huge amounts of memory

[pelotom]

As of version 1.32 and 1.33, ts-loader hangs while consuming much more memory than usual when I try to compile my relatively large TypeScript project. Running tsc directly works fine, as does ts-loader v1.31 and earlier.

[johnnyreilly]

Thats interesting. My own seem to be fine. I would speculate this is related to using webpack's newer enhanced-resolve. Since we have a need to use enhanced-resolve I'm not sure what to suggest..

[johnnyreilly]

If you add in the loader options plugin as suggested in #421 does this resolve the issue?

[HerringtonDarkholme]

@pelotom can you provide TSC's diagnostics with tsc --diagnostics?

If your project is large, it's probable that tsc marginally get compilation done with 1GB heap memory limit. ts-loader usually requires more memory than plain tsc.

[pelotom]

@johnnyreilly I'm still using WebPack 1.14.0, and LoaderOptionsPlugin doesn't appear to exist for that?

@HerringtonDarkholme tsc --diagnostics outputs the following:

Files:          1299
Lines:        152445
Nodes:        841958
Identifiers:  289186
Symbols:      299124
Types:         84776
Memory used: 393890K
I/O read:      0.08s
I/O write:     0.00s
Parse time:    9.63s
Bind time:     1.29s
Check time:    6.15s
Emit time:     0.00s
Total time:   17.07s

[johnnyreilly]

Yes - loader op

[johnnyreilly]

Yes loader options is for WebPack 2. Could you share a minimal repro?

[pelotom]

Unfortunately this is a private company repo. I'm not sure how to make a minimal example from it that doesn't effectively open source the project 😛

[johnnyreilly]

Tricky but I understand. I suggest you fork, have a play around and attempt to fix. I recommend taking a look at the part of the code that uses enhanced-resolve as it's likely that's the issue. We'd certainly appreciate a PR if you're successful.

[johnnyreilly]

This is where you probably want to focus:

https://github.com/TypeStrong/ts-loader/blob/master/src/resolver.ts

https://github.com/TypeStrong/ts-loader/blob/master/src/servicesHost.ts

[johnnyreilly]

One other thing to try; might be worth commenting out these lines: https://github.com/TypeStrong/ts-loader/blob/master/src/index.ts#L72-L74

See if that makes a difference before you try the other bits. This was a micro optimisation I added in 1.3.3; it probably wasn't worth it anyway but it might be a cause to look at.

[pelotom]

@johnnyreilly It can't be related to anything that was added or removed in 1.3.3 because the problem was introduced in 1.3.2.

[HerringtonDarkholme]

@pelotom Thanks for the diagnostics. It shows totally reasonable memory consumption. So that's really resolve problem I think.

I would suggest you to clone the repository, npm link the local repository to your project. And git bisect the problematic commit.

[pelotom]

Good idea, I'll do that and report back.

[pelotom]

Bisecting tells me 3b4b83d introduced the problem.

[johnnyreilly]

Thats interesting. So the problem actually lies with enhanced resolve by the sounds of it. Could you raise an issue here: https://github.com/webpack/enhanced-resolve

[pelotom]

I figured out a workaround which may help illuminate what's going on. As part of our build process we npm linkthe top-level directories under src/, e.g. src/foo and src/bar, so that we can then import things using "absolute" paths, e.g.

import { something } from 'foo/some/module/path'

On a hunch that this unorthodox practice of ours might be why no one else has noticed this, I got rid of the npm links and used TypeScript 2.0's path-mapping feature:

// tsconfig.json
{
  "compilerOptions": {
    // ...
    "baseUrl": ".",
    "paths": {
      "foo/*": ["src/foo/*"],
      "bar/*": ["src/bar/*"]
    }
  }
}

in conjunction with WebPack's resolve.root:

{ // webpack.config.js
  // ...
  resolve: {
    extensions: ['.js', '.jsx', '.ts', '.tsx', ''],
    root: path.resolve('./src'),
  },
  // ...
}

This achieved the same effect as the npm link trick... moreover I was able to upgrade ts-loader to 1.3.3 and compilation no longer hangs, woohoo!

So there is apparently some bad interaction between enhanced-resolve and NPM links in moderately large projects, but having found a way to circumvent the issue I'm probably not going to pursue it any further.

[HerringtonDarkholme]

I guess this might be a cyclic directory problem for enhanced-resolve, but not sure.
Given webpack already has resolve.root in conjunction with path mapping in TypeScript I do agree that npm link might not be a wise choice.

[pelotom]

Thanks @HerringtonDarkholme and @johnnyreilly. Closing as I think this can be considered resolved as far as ts-loader is concerned.

[johnnyreilly]

Thanks for investigating and reporting back @pelotom - that's always helpful. 👍