Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt to reduce types which are about to produce a complexity error #53739

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

weswigham
Copy link
Member

@weswigham weswigham commented Apr 11, 2023

Basically #42772 but synced with main, but also improves how often we defer union/intersection member types; namely with this we now always defer construction of a union or intersection's symbol's type if one of its' constituent symbols is itself deferred. I had to add this in to the original change, as in the intervening time between the original PR and now, we got noticeably worse at deferring union members, and actually started manufacturing members like ref even if they weren't explicitly referenced (in this case, to check if the member can be used for contextual type discimination, which jsx now tries).

The original PR description (which is also still true) for reference:

When an intersection is going to produce an expression too complex error, eagerly perform reductions in a last-ditch effort to avoid the error. I've wanted to do something like this since #29949 was opened, and since we added getReducedType and associated machinery, alongside machinery for bailing when expressions are producing unions/intersections are too complex. I feel like we now have a scenario where it's justifiable to invoke the reduction machinery earlier than we normally do in order to try to save us from resorting to an "Expression produces a union type that is too complex to represent" error, since that error is ultimately heuristic and, in fact, the union produced is not too complex if significant reductions take place.

#29949 has had acceptable perf for awhile, it just also had a complexity error in it, so could already be considered "fixed". This really fixes #29949, IMO, by finally removing that "expression too complex" error once react's types are adjusted to make the two ref object types mutually exclusive by adding a current?: undefined property to the RefCallback type and the callback in ForwardedRef.

This shouldn't impact perf generally, since this fallback heuristic only takes effect when we would already issue an error.

Fixes #53234

@weswigham
Copy link
Member Author

@typescript-bot test this
@typescript-bot run dt
@typescript-bot test top100
@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Heya @weswigham, I've started to run the extended test suite on this PR at 2d21364. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Heya @weswigham, I've started to run the parallelized Definitely Typed test suite on this PR at 2d21364. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Heya @weswigham, I've started to run the diff-based top-repos suite on this PR at 2d21364. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Heya @weswigham, I've started to run the perf test suite on this PR at 2d21364. You can monitor the build here.

@weswigham
Copy link
Member Author

Modified the implementation slightly to reduce each element being considered a bit more aggressively and then the test case from #53234.

@weswigham
Copy link
Member Author

@typescript-bot test this
@typescript-bot run dt
@typescript-bot test top100
@typescript-bot perf test this
@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Heya @weswigham, I've started to run the tarball bundle task on this PR at 757c56a. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Heya @weswigham, I've started to run the perf test suite on this PR at 757c56a. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Heya @weswigham, I've started to run the diff-based top-repos suite on this PR at 757c56a. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Heya @weswigham, I've started to run the extended test suite on this PR at 757c56a. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Heya @weswigham, I've started to run the parallelized Definitely Typed test suite on this PR at 757c56a. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Hey @weswigham, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/152162/artifacts?artifactName=tgz&fileId=BEADF2FF396A56967CF73BDF9503B12030501FCC7EEFB858C5748DE53A77ED5C02&fileName=/typescript-5.1.0-insiders.20230411.tgz"
    }
}

and then running npm install.


There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/[email protected]".;

@typescript-bot
Copy link
Collaborator

@weswigham
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - main..53739
Metric main 53739 Delta Best Worst p-value
Angular - node (v18.10.0, x64)
Memory used 365,353k (± 0.01%) 365,345k (± 0.01%) ~ 365,314k 365,386k p=0.471 n=6
Parse Time 3.38s (± 0.36%) 3.38s (± 0.85%) ~ 3.33s 3.40s p=1.000 n=6
Bind Time 1.12s (± 0.73%) 1.12s (± 0.56%) ~ 1.11s 1.13s p=0.432 n=6
Check Time 8.74s (± 0.61%) 8.71s (± 0.51%) ~ 8.65s 8.78s p=0.334 n=6
Emit Time 7.48s (± 0.83%) 7.48s (± 0.77%) ~ 7.43s 7.59s p=0.810 n=6
Total Time 20.73s (± 0.54%) 20.68s (± 0.52%) ~ 20.58s 20.85s p=0.423 n=6
Compiler-Unions - node (v18.10.0, x64)
Memory used 190,836k (± 0.01%) 190,862k (± 0.02%) ~ 190,820k 190,930k p=0.199 n=6
Parse Time 1.50s (± 0.70%) 1.50s (± 0.73%) ~ 1.49s 1.51s p=0.498 n=6
Bind Time 0.77s (± 1.28%) 0.77s (± 1.06%) ~ 0.76s 0.78s p=0.862 n=6
Check Time 9.57s (± 0.38%) 9.58s (± 0.66%) ~ 9.50s 9.69s p=0.871 n=6
Emit Time 2.73s (± 1.20%) 2.72s (± 0.59%) ~ 2.71s 2.75s p=0.745 n=6
Total Time 14.57s (± 0.40%) 14.57s (± 0.51%) ~ 14.47s 14.70s p=0.936 n=6
Monaco - node (v18.10.0, x64)
Memory used 346,475k (± 0.01%) 346,502k (± 0.01%) ~ 346,477k 346,531k p=0.093 n=6
Parse Time 2.58s (± 0.87%) 2.59s (± 1.02%) ~ 2.56s 2.63s p=0.686 n=6
Bind Time 1.00s (± 0.89%) 1.01s (± 1.37%) ~ 0.99s 1.03s p=0.673 n=6
Check Time 7.11s (± 0.82%) 7.11s (± 0.18%) ~ 7.10s 7.13s p=0.745 n=6
Emit Time 4.26s (± 0.53%) 4.28s (± 0.73%) ~ 4.25s 4.33s p=0.329 n=6
Total Time 14.95s (± 0.53%) 14.98s (± 0.34%) ~ 14.94s 15.07s p=0.872 n=6
TFS - node (v18.10.0, x64)
Memory used 300,737k (± 0.01%) 300,737k (± 0.00%) ~ 300,717k 300,756k p=0.936 n=6
Parse Time 2.06s (± 0.85%) 2.06s (± 1.83%) ~ 2.01s 2.11s p=0.870 n=6
Bind Time 1.14s (± 0.92%) 1.14s (± 0.45%) ~ 1.13s 1.14s p=0.142 n=6
Check Time 6.59s (± 0.68%) 6.54s (± 0.45%) -0.06s (- 0.86%) 6.50s 6.58s p=0.045 n=6
Emit Time 3.85s (± 0.75%) 3.83s (± 0.86%) ~ 3.78s 3.87s p=0.196 n=6
Total Time 13.66s (± 0.50%) 13.56s (± 0.46%) -0.10s (- 0.72%) 13.46s 13.61s p=0.035 n=6
material-ui - node (v18.10.0, x64)
Memory used 480,129k (± 0.00%) 480,211k (± 0.02%) +82k (+ 0.02%) 480,155k 480,366k p=0.008 n=6
Parse Time 3.02s (± 0.45%) 3.04s (± 0.71%) ~ 3.01s 3.07s p=0.288 n=6
Bind Time 0.91s (± 0.82%) 0.92s (± 2.02%) ~ 0.91s 0.96s p=0.150 n=6
Check Time 17.16s (± 0.28%) 17.36s (± 0.70%) +0.20s (+ 1.15%) 17.17s 17.49s p=0.013 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 21.09s (± 0.17%) 21.32s (± 0.64%) +0.22s (+ 1.04%) 21.09s 21.46s p=0.019 n=6
xstate - node (v18.10.0, x64)
Memory used 562,090k (± 0.02%) 562,131k (± 0.03%) ~ 561,932k 562,341k p=0.748 n=6
Parse Time 3.82s (± 0.48%) 3.81s (± 0.44%) ~ 3.79s 3.83s p=0.282 n=6
Bind Time 1.65s (± 0.33%) 1.64s (± 0.81%) ~ 1.62s 1.66s p=0.663 n=6
Check Time 2.83s (± 0.47%) 2.88s (± 0.61%) +0.06s (+ 2.00%) 2.86s 2.90s p=0.004 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 0.00%) ~ 0.08s 0.08s p=1.000 n=6
Total Time 8.38s (± 0.23%) 8.41s (± 0.39%) ~ 8.38s 8.46s p=0.064 n=6
Angular - node (v16.17.1, x64)
Memory used 364,726k (± 0.01%) 364,742k (± 0.01%) ~ 364,705k 364,776k p=0.575 n=6
Parse Time 3.54s (± 0.97%) 3.54s (± 0.70%) ~ 3.51s 3.57s p=1.000 n=6
Bind Time 1.18s (± 0.69%) 1.18s (± 0.76%) ~ 1.17s 1.19s p=0.550 n=6
Check Time 9.50s (± 0.46%) 9.54s (± 0.35%) ~ 9.49s 9.58s p=0.170 n=6
Emit Time 7.93s (± 0.58%) 7.94s (± 0.45%) ~ 7.90s 7.99s p=1.000 n=6
Total Time 22.16s (± 0.38%) 22.20s (± 0.28%) ~ 22.07s 22.23s p=0.572 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 192,535k (± 0.02%) 192,540k (± 0.01%) ~ 192,514k 192,588k p=0.471 n=6
Parse Time 1.59s (± 0.76%) 1.58s (± 0.95%) ~ 1.55s 1.59s p=0.323 n=6
Bind Time 0.83s (± 0.66%) 0.82s (± 0.63%) ~ 0.82s 0.83s p=0.640 n=6
Check Time 10.28s (± 0.54%) 10.30s (± 0.65%) ~ 10.23s 10.43s p=0.746 n=6
Emit Time 2.99s (± 0.25%) 2.97s (± 0.81%) ~ 2.93s 3.00s p=0.161 n=6
Total Time 15.68s (± 0.41%) 15.68s (± 0.52%) ~ 15.58s 15.82s p=0.808 n=6
Monaco - node (v16.17.1, x64)
Memory used 345,754k (± 0.01%) 345,770k (± 0.00%) +16k (+ 0.00%) 345,758k 345,779k p=0.031 n=6
Parse Time 2.71s (± 0.43%) 2.71s (± 0.30%) ~ 2.70s 2.72s p=0.738 n=6
Bind Time 1.08s (± 0.38%) 1.09s (± 0.69%) ~ 1.08s 1.10s p=0.100 n=6
Check Time 7.83s (± 0.63%) 7.79s (± 0.57%) ~ 7.74s 7.87s p=0.196 n=6
Emit Time 4.48s (± 0.63%) 4.47s (± 0.52%) ~ 4.45s 4.51s p=0.517 n=6
Total Time 16.10s (± 0.44%) 16.06s (± 0.32%) ~ 15.99s 16.13s p=0.148 n=6
TFS - node (v16.17.1, x64)
Memory used 300,062k (± 0.01%) 300,062k (± 0.01%) ~ 300,019k 300,097k p=0.936 n=6
Parse Time 2.17s (± 0.64%) 2.16s (± 0.38%) ~ 2.15s 2.17s p=0.314 n=6
Bind Time 1.23s (± 0.84%) 1.24s (± 0.51%) ~ 1.23s 1.25s p=0.203 n=6
Check Time 7.21s (± 0.57%) 7.24s (± 0.46%) ~ 7.20s 7.29s p=0.373 n=6
Emit Time 4.33s (± 0.74%) 4.35s (± 0.53%) ~ 4.32s 4.38s p=0.260 n=6
Total Time 14.94s (± 0.42%) 14.98s (± 0.24%) ~ 14.93s 15.02s p=0.293 n=6
material-ui - node (v16.17.1, x64)
Memory used 479,410k (± 0.00%) 479,448k (± 0.00%) +38k (+ 0.01%) 479,421k 479,476k p=0.020 n=6
Parse Time 3.23s (± 0.44%) 3.24s (± 0.25%) ~ 3.23s 3.25s p=0.402 n=6
Bind Time 0.95s (± 0.00%) 0.95s (± 0.54%) ~ 0.95s 0.96s p=0.174 n=6
Check Time 18.31s (± 0.73%) 18.29s (± 0.35%) ~ 18.18s 18.35s p=1.000 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.49s (± 0.60%) 22.47s (± 0.30%) ~ 22.36s 22.55s p=1.000 n=6
xstate - node (v16.17.1, x64)
Memory used 559,701k (± 0.02%) 559,764k (± 0.03%) ~ 559,506k 559,942k p=0.521 n=6
Parse Time 3.99s (± 0.39%) 3.98s (± 0.16%) ~ 3.97s 3.99s p=0.180 n=6
Bind Time 1.75s (± 0.60%) 1.74s (± 0.23%) ~ 1.73s 1.74s p=0.209 n=6
Check Time 3.05s (± 0.41%) 3.10s (± 0.55%) +0.05s (+ 1.69%) 3.08s 3.12s p=0.005 n=6
Emit Time 0.09s (± 5.76%) 0.10s (± 5.76%) ~ 0.09s 0.10s p=1.000 n=6
Total Time 8.88s (± 0.17%) 8.92s (± 0.28%) +0.04s (+ 0.47%) 8.88s 8.95s p=0.020 n=6
Angular - node (v14.15.1, x64)
Memory used 358,496k (± 0.00%) 358,510k (± 0.00%) +14k (+ 0.00%) 358,498k 358,524k p=0.045 n=6
Parse Time 3.64s (± 0.59%) 3.62s (± 0.51%) ~ 3.60s 3.65s p=0.249 n=6
Bind Time 1.21s (± 0.43%) 1.21s (± 0.34%) ~ 1.21s 1.22s p=0.595 n=6
Check Time 9.92s (± 0.37%) 9.89s (± 0.25%) ~ 9.86s 9.93s p=0.171 n=6
Emit Time 8.33s (± 1.03%) 8.27s (± 0.75%) ~ 8.18s 8.35s p=0.260 n=6
Total Time 23.10s (± 0.41%) 22.99s (± 0.28%) -0.11s (- 0.48%) 22.93s 23.10s p=0.044 n=6
Compiler-Unions - node (v14.15.1, x64)
Memory used 187,795k (± 0.05%) 187,838k (± 0.01%) ~ 187,781k 187,857k p=0.149 n=6
Parse Time 1.60s (± 0.83%) 1.60s (± 0.61%) ~ 1.58s 1.61s p=0.739 n=6
Bind Time 0.85s (± 0.89%) 0.85s (± 0.61%) ~ 0.84s 0.85s p=0.784 n=6
Check Time 10.38s (± 0.56%) 10.35s (± 0.54%) ~ 10.28s 10.43s p=0.378 n=6
Emit Time 3.23s (± 4.30%) 3.12s (± 0.51%) 🟩-0.11s (- 3.30%) 3.09s 3.13s p=0.004 n=6
Total Time 16.06s (± 1.19%) 15.91s (± 0.39%) -0.15s (- 0.91%) 15.83s 16.00s p=0.030 n=6
Monaco - node (v14.15.1, x64)
Memory used 340,659k (± 0.00%) 340,672k (± 0.01%) ~ 340,645k 340,694k p=0.173 n=6
Parse Time 2.81s (± 0.43%) 2.82s (± 0.93%) ~ 2.80s 2.87s p=0.805 n=6
Bind Time 1.10s (± 0.68%) 1.10s (± 1.06%) ~ 1.09s 1.12s p=0.734 n=6
Check Time 8.14s (± 0.34%) 8.13s (± 0.21%) ~ 8.11s 8.16s p=0.570 n=6
Emit Time 4.68s (± 0.45%) 4.68s (± 0.47%) ~ 4.65s 4.71s p=1.000 n=6
Total Time 16.73s (± 0.20%) 16.74s (± 0.15%) ~ 16.70s 16.77s p=0.935 n=6
TFS - node (v14.15.1, x64)
Memory used 295,136k (± 0.00%) 295,131k (± 0.00%) ~ 295,117k 295,142k p=0.378 n=6
Parse Time 2.40s (± 0.73%) 2.39s (± 0.97%) ~ 2.36s 2.42s p=0.682 n=6
Bind Time 1.07s (± 0.97%) 1.06s (± 0.49%) ~ 1.06s 1.07s p=0.774 n=6
Check Time 7.54s (± 0.32%) 7.55s (± 0.56%) ~ 7.49s 7.60s p=0.687 n=6
Emit Time 4.29s (± 0.90%) 4.28s (± 0.49%) ~ 4.26s 4.31s p=0.808 n=6
Total Time 15.30s (± 0.34%) 15.28s (± 0.35%) ~ 15.18s 15.33s p=0.936 n=6
material-ui - node (v14.15.1, x64)
Memory used 474,969k (± 0.01%) 475,030k (± 0.01%) +62k (+ 0.01%) 474,992k 475,063k p=0.020 n=6
Parse Time 3.36s (± 1.00%) 3.35s (± 0.36%) ~ 3.34s 3.37s p=0.935 n=6
Bind Time 1.00s (± 0.82%) 1.00s (± 0.98%) ~ 0.99s 1.01s p=0.862 n=6
Check Time 19.24s (± 0.43%) 19.20s (± 0.71%) ~ 19.04s 19.43s p=0.423 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 23.60s (± 0.42%) 23.56s (± 0.59%) ~ 23.43s 23.81s p=0.377 n=6
xstate - node (v14.15.1, x64)
Memory used 548,279k (± 0.00%) 548,231k (± 0.01%) -49k (- 0.01%) 548,175k 548,265k p=0.013 n=6
Parse Time 4.26s (± 1.41%) 4.27s (± 1.67%) ~ 4.22s 4.41s p=0.871 n=6
Bind Time 1.61s (± 3.10%) 1.63s (± 3.52%) ~ 1.52s 1.67s p=0.517 n=6
Check Time 3.15s (± 0.77%) 3.23s (± 0.60%) +0.08s (+ 2.38%) 3.21s 3.26s p=0.005 n=6
Emit Time 0.09s (± 4.45%) 0.09s (± 0.00%) ~ 0.09s 0.09s p=0.405 n=6
Total Time 9.12s (± 0.49%) 9.22s (± 0.55%) +0.10s (+ 1.13%) 9.14s 9.28s p=0.016 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-135-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.15.1, x64)
Scenarios
  • Angular - node (v18.10.0, x64)
  • Angular - node (v16.17.1, x64)
  • Angular - node (v14.15.1, x64)
  • Compiler-Unions - node (v18.10.0, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Compiler-Unions - node (v14.15.1, x64)
  • Monaco - node (v18.10.0, x64)
  • Monaco - node (v16.17.1, x64)
  • Monaco - node (v14.15.1, x64)
  • TFS - node (v18.10.0, x64)
  • TFS - node (v16.17.1, x64)
  • TFS - node (v14.15.1, x64)
  • material-ui - node (v18.10.0, x64)
  • material-ui - node (v16.17.1, x64)
  • material-ui - node (v14.15.1, x64)
  • xstate - node (v18.10.0, x64)
  • xstate - node (v16.17.1, x64)
  • xstate - node (v14.15.1, x64)
Benchmark Name Iterations
Current 53739 6
Baseline main 6

TSServer

Comparison Report - main..53739
Metric main 53739 Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,395ms (± 0.57%) 2,395ms (± 0.56%) ~ 2,384ms 2,420ms p=0.809 n=6
Req 2 - geterr 5,734ms (± 0.38%) 5,718ms (± 0.46%) ~ 5,670ms 5,743ms p=0.470 n=6
Req 3 - references 336ms (± 0.59%) 334ms (± 0.35%) ~ 333ms 336ms p=0.164 n=6
Req 4 - navto 280ms (± 0.93%) 278ms (± 0.43%) ~ 277ms 280ms p=0.289 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 86ms (± 2.70%) 84ms (± 0.62%) ~ 83ms 84ms p=0.079 n=6
CompilerTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,501ms (± 0.53%) 2,522ms (± 1.12%) ~ 2,492ms 2,562ms p=0.230 n=6
Req 2 - geterr 4,255ms (± 0.81%) 4,228ms (± 0.53%) ~ 4,201ms 4,266ms p=0.126 n=6
Req 3 - references 344ms (± 0.86%) 345ms (± 0.57%) ~ 342ms 348ms p=1.000 n=6
Req 4 - navto 291ms (± 0.99%) 290ms (± 0.96%) ~ 287ms 295ms p=0.566 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 65ms (± 6.39%) 65ms (± 4.86%) ~ 62ms 69ms p=0.934 n=6
xstateTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 3,015ms (± 0.80%) 3,013ms (± 0.34%) ~ 3,004ms 3,033ms p=1.000 n=6
Req 2 - geterr 1,625ms (± 0.70%) 1,579ms (± 1.69%) -46ms (- 2.84%) 1,553ms 1,620ms p=0.020 n=6
Req 3 - references 107ms (± 2.54%) 106ms (± 1.41%) ~ 104ms 108ms p=0.744 n=6
Req 4 - navto 362ms (± 0.49%) 368ms (± 2.69%) ~ 363ms 388ms p=0.050 n=6
Req 5 - completionInfo count 2,862 (± 0.00%) 2,862 (± 0.00%) ~ 2,862 2,862 p=1.000 n=6
Req 5 - completionInfo 370ms (± 1.58%) 370ms (± 1.68%) ~ 361ms 377ms p=1.000 n=6
Compiler-UnionsTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,504ms (± 0.66%) 2,510ms (± 0.30%) ~ 2,502ms 2,519ms p=0.575 n=6
Req 2 - geterr 6,154ms (± 0.43%) 6,134ms (± 0.67%) ~ 6,101ms 6,210ms p=0.230 n=6
Req 3 - references 344ms (± 0.45%) 345ms (± 0.56%) ~ 342ms 347ms p=0.566 n=6
Req 4 - navto 278ms (± 0.49%) 279ms (± 0.37%) ~ 277ms 280ms p=0.737 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 81ms (± 1.82%) 80ms (± 0.79%) ~ 79ms 81ms p=0.345 n=6
CompilerTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,660ms (± 0.87%) 2,668ms (± 1.18%) ~ 2,628ms 2,712ms p=0.575 n=6
Req 2 - geterr 4,647ms (± 0.63%) 4,599ms (± 0.52%) -48ms (- 1.03%) 4,565ms 4,627ms p=0.031 n=6
Req 3 - references 359ms (± 0.54%) 357ms (± 0.42%) ~ 356ms 360ms p=0.164 n=6
Req 4 - navto 284ms (± 0.72%) 284ms (± 0.59%) ~ 282ms 287ms p=0.934 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 68ms (± 1.21%) 67ms (± 0.77%) ~ 67ms 68ms p=0.523 n=6
xstateTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 3,169ms (± 0.30%) 3,178ms (± 0.31%) ~ 3,161ms 3,190ms p=0.260 n=6
Req 2 - geterr 1,766ms (± 1.34%) 1,744ms (± 1.22%) ~ 1,717ms 1,775ms p=0.148 n=6
Req 3 - references 115ms (± 2.49%) 113ms (± 1.82%) ~ 111ms 117ms p=0.329 n=6
Req 4 - navto 348ms (± 1.42%) 352ms (± 0.42%) +4ms (+ 1.20%) 350ms 354ms p=0.029 n=6
Req 5 - completionInfo count 2,862 (± 0.00%) 2,862 (± 0.00%) ~ 2,862 2,862 p=1.000 n=6
Req 5 - completionInfo 388ms (± 2.20%) 385ms (± 1.18%) ~ 378ms 391ms p=0.574 n=6
Compiler-UnionsTSServer - node (v14.15.1, x64)
Req 1 - updateOpen 2,607ms (± 0.74%) 2,600ms (± 0.42%) ~ 2,585ms 2,614ms p=0.810 n=6
Req 2 - geterr 6,423ms (± 1.69%) 6,369ms (± 1.78%) ~ 6,240ms 6,482ms p=0.378 n=6
Req 3 - references 366ms (± 1.09%) 361ms (± 0.98%) ~ 356ms 365ms p=0.107 n=6
Req 4 - navto 282ms (± 0.27%) 283ms (± 0.88%) ~ 280ms 285ms p=0.742 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 99ms (± 8.23%) 96ms (± 6.80%) ~ 89ms 103ms p=0.331 n=6
CompilerTSServer - node (v14.15.1, x64)
Req 1 - updateOpen 2,808ms (± 0.17%) 2,822ms (± 1.23%) ~ 2,797ms 2,889ms p=0.809 n=6
Req 2 - geterr 4,725ms (± 0.53%) 4,733ms (± 1.22%) ~ 4,695ms 4,850ms p=0.936 n=6
Req 3 - references 377ms (± 1.15%) 378ms (± 1.37%) ~ 370ms 385ms p=0.574 n=6
Req 4 - navto 292ms (± 0.36%) 293ms (± 0.73%) ~ 290ms 296ms p=0.254 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 80ms (± 4.17%) 82ms (± 3.40%) ~ 81ms 88ms p=0.210 n=6
xstateTSServer - node (v14.15.1, x64)
Req 1 - updateOpen 3,402ms (± 0.43%) 3,381ms (± 0.32%) -21ms (- 0.63%) 3,368ms 3,400ms p=0.013 n=6
Req 2 - geterr 1,877ms (± 0.49%) 1,857ms (± 0.71%) -20ms (- 1.05%) 1,841ms 1,879ms p=0.045 n=6
Req 3 - references 126ms (± 2.10%) 125ms (± 1.43%) ~ 123ms 128ms p=0.622 n=6
Req 4 - navto 378ms (± 1.37%) 376ms (± 1.23%) ~ 370ms 380ms p=0.296 n=6
Req 5 - completionInfo count 2,862 (± 0.00%) 2,862 (± 0.00%) ~ 2,862 2,862 p=1.000 n=6
Req 5 - completionInfo 407ms (± 0.97%) 410ms (± 1.23%) ~ 400ms 413ms p=0.169 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-135-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.15.1, x64)
Scenarios
  • Compiler-UnionsTSServer - node (v18.10.0, x64)
  • Compiler-UnionsTSServer - node (v16.17.1, x64)
  • Compiler-UnionsTSServer - node (v14.15.1, x64)
  • CompilerTSServer - node (v18.10.0, x64)
  • CompilerTSServer - node (v16.17.1, x64)
  • CompilerTSServer - node (v14.15.1, x64)
  • xstateTSServer - node (v18.10.0, x64)
  • xstateTSServer - node (v16.17.1, x64)
  • xstateTSServer - node (v14.15.1, x64)
Benchmark Name Iterations
Current 53739 6
Baseline main 6

Startup

Comparison Report - main..53739
Metric main 53739 Delta Best Worst p-value
tsc-startup - node (v16.17.1, x64)
Execution time 141.69ms (± 0.20%) 141.45ms (± 0.18%) -0.24ms (- 0.17%) 140.78ms 144.45ms p=0.000 n=600
tsserver-startup - node (v16.17.1, x64)
Execution time 219.87ms (± 0.18%) 219.52ms (± 0.18%) -0.36ms (- 0.16%) 218.62ms 224.87ms p=0.000 n=600
tsserverlibrary-startup - node (v16.17.1, x64)
Execution time 221.38ms (± 0.13%) 221.06ms (± 0.18%) -0.32ms (- 0.14%) 220.20ms 228.84ms p=0.000 n=600
typescript-startup - node (v16.17.1, x64)
Execution time 203.25ms (± 0.15%) 202.95ms (± 0.17%) -0.31ms (- 0.15%) 202.20ms 207.70ms p=0.000 n=600
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-135-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • tsc-startup - node (v16.17.1, x64)
  • tsserver-startup - node (v16.17.1, x64)
  • tsserverlibrary-startup - node (v16.17.1, x64)
  • typescript-startup - node (v16.17.1, x64)
Benchmark Name Iterations
Current 53739 6
Baseline main 6

Developer Information:

Download Benchmark

@gabritto
Copy link
Member

@typescript-bot perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 11, 2023

Heya @gabritto, I've started to run the abridged perf test suite on this PR at 757c56a. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

Hey @weswigham, the results of running the DT tests are ready.
Everything looks the same!
You can check the log here.

@typescript-bot
Copy link
Collaborator

@weswigham Here are the results of running the top-repos suite comparing main and refs/pull/53739/merge:

Everything looks good!

if (i === typeSet.length - 1 || isTypeAny(runningResult) || runningResult.flags & TypeFlags.Never) {
return runningResult;
}
if (!(runningResult.flags & TypeFlags.Union) || (runningResult as UnionType).types.length > typeSet.length) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we bail if the second condition here is true ((runningResult as UnionType).types.length > typeSet.length)? Is it because then the heuristic is not being helpful in reducing the number of types to intersect?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. If going element-by-element is growing the type, this probably isn't helping, so we give up on it.

@typescript-bot
Copy link
Collaborator

@gabritto
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..53739

Metric main 53739 Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 364,726k (± 0.01%) 364,744k (± 0.00%) ~ 364,727k 364,767k p=0.378 n=6
Parse Time 3.54s (± 0.97%) 3.55s (± 0.21%) ~ 3.54s 3.56s p=0.566 n=6
Bind Time 1.18s (± 0.69%) 1.18s (± 1.07%) ~ 1.17s 1.20s p=0.558 n=6
Check Time 9.50s (± 0.46%) 9.53s (± 0.32%) ~ 9.48s 9.57s p=0.375 n=6
Emit Time 7.93s (± 0.58%) 8.01s (± 0.73%) +0.07s (+ 0.92%) 7.95s 8.11s p=0.029 n=6
Total Time 22.16s (± 0.38%) 22.26s (± 0.25%) ~ 22.21s 22.37s p=0.065 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 192,535k (± 0.02%) 193,041k (± 0.73%) ~ 192,194k 195,913k p=0.810 n=6
Parse Time 1.59s (± 0.76%) 1.59s (± 1.22%) ~ 1.55s 1.60s p=0.557 n=6
Bind Time 0.83s (± 0.66%) 0.82s (± 0.99%) ~ 0.81s 0.83s p=0.859 n=6
Check Time 10.28s (± 0.54%) 10.33s (± 0.39%) ~ 10.29s 10.39s p=0.146 n=6
Emit Time 2.99s (± 0.25%) 3.04s (± 3.99%) ~ 2.94s 3.27s p=0.373 n=6
Total Time 15.68s (± 0.41%) 15.78s (± 0.94%) ~ 15.66s 16.08s p=0.128 n=6
Monaco - node (v16.17.1, x64)
Memory used 345,754k (± 0.01%) 345,755k (± 0.00%) ~ 345,748k 345,764k p=0.574 n=6
Parse Time 2.71s (± 0.43%) 2.71s (± 0.19%) ~ 2.71s 2.72s p=0.666 n=6
Bind Time 1.08s (± 0.38%) 1.08s (± 0.48%) ~ 1.08s 1.09s p=0.595 n=6
Check Time 7.83s (± 0.63%) 7.80s (± 0.67%) ~ 7.74s 7.88s p=0.258 n=6
Emit Time 4.48s (± 0.63%) 4.44s (± 0.18%) ~ 4.43s 4.45s p=0.052 n=6
Total Time 16.10s (± 0.44%) 16.03s (± 0.37%) ~ 15.97s 16.12s p=0.127 n=6
TFS - node (v16.17.1, x64)
Memory used 300,062k (± 0.01%) 300,038k (± 0.01%) ~ 299,999k 300,087k p=0.261 n=6
Parse Time 2.17s (± 0.64%) 2.17s (± 0.48%) ~ 2.15s 2.18s p=1.000 n=6
Bind Time 1.23s (± 0.84%) 1.24s (± 0.33%) ~ 1.23s 1.24s p=0.257 n=6
Check Time 7.21s (± 0.57%) 7.25s (± 0.32%) ~ 7.22s 7.28s p=0.077 n=6
Emit Time 4.33s (± 0.74%) 4.31s (± 0.56%) ~ 4.27s 4.34s p=0.225 n=6
Total Time 14.94s (± 0.42%) 14.96s (± 0.27%) ~ 14.90s 15.02s p=0.809 n=6
material-ui - node (v16.17.1, x64)
Memory used 479,410k (± 0.00%) 479,452k (± 0.00%) +42k (+ 0.01%) 479,427k 479,478k p=0.008 n=6
Parse Time 3.23s (± 0.44%) 3.25s (± 0.42%) +0.02s (+ 0.72%) 3.24s 3.28s p=0.020 n=6
Bind Time 0.95s (± 0.00%) 0.96s (± 0.88%) ~ 0.95s 0.97s p=0.176 n=6
Check Time 18.31s (± 0.73%) 18.43s (± 1.94%) ~ 18.22s 19.14s p=0.689 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.49s (± 0.60%) 22.63s (± 1.59%) ~ 22.42s 23.35s p=0.378 n=6
xstate - node (v16.17.1, x64)
Memory used 559,701k (± 0.02%) 559,723k (± 0.02%) ~ 559,619k 559,877k p=0.689 n=6
Parse Time 3.99s (± 0.39%) 3.99s (± 0.13%) ~ 3.98s 3.99s p=1.000 n=6
Bind Time 1.75s (± 0.60%) 1.74s (± 0.47%) ~ 1.73s 1.75s p=0.181 n=6
Check Time 3.05s (± 0.41%) 3.12s (± 0.55%) +0.07s (+ 2.35%) 3.10s 3.15s p=0.005 n=6
Emit Time 0.09s (± 5.76%) 0.10s (± 5.76%) ~ 0.09s 0.10s p=1.000 n=6
Total Time 8.88s (± 0.17%) 8.94s (± 0.24%) +0.06s (+ 0.64%) 8.91s 8.97s p=0.005 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-135-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
Benchmark Name Iterations
Current 53739 6
Baseline main 6

Developer Information:

Download Benchmark

Copy link
Member

@gabritto gabritto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. It would be nice though to confirm that the perf regression in xstate happens because there's an expression too complex compilation error.

@sandersn sandersn added the For Backlog Bug PRs that fix a backlog bug label May 2, 2023
@sandersn
Copy link
Member

@weswigham Is this ready to go? I don't know whether you want to double check xstate results first or not.

@weswigham weswigham marked this pull request as draft August 13, 2024 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
For Backlog Bug PRs that fix a backlog bug
Projects
Status: Needs merge
4 participants