Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fail to allocate memory when run rustup update #2759

Closed
wongsyrone opened this issue May 7, 2021 · 32 comments
Closed

fail to allocate memory when run rustup update #2759

wongsyrone opened this issue May 7, 2021 · 32 comments
Labels

Comments

@wongsyrone
Copy link

wongsyrone commented May 7, 2021

Problem
Fail to allocate memory when I run rustup update on Windows 10 x64 using cmd.exe

C:\Users\home>rustup update
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'
info: syncing channel updates for 'nightly-x86_64-pc-windows-gnu'
info: latest update on 2021-05-07, rust version 1.54.0-nightly (676ee1472 2021-05-06)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
 16.2 MiB /  16.2 MiB (100 %)   8.0 MiB/s in  2s ETA:  0s
info: downloading component 'rust-mingw'
info: downloading component 'rust-std'
 23.6 MiB /  23.6 MiB (100 %)   8.0 MiB/s in  2s ETA:  0s
info: downloading component 'rustc'
 79.8 MiB /  79.8 MiB (100 %)  14.0 MiB/s in  7s ETA:  0s
info: downloading component 'rustfmt'
info: removing previous version of component 'cargo'
info: removing previous version of component 'clippy'
info: removing previous version of component 'rust-docs'
info: removing previous version of component 'rust-mingw'
info: removing previous version of component 'rust-std'
info: removing previous version of component 'rustc'
warning: during uninstall component rustc was not found
info: removing previous version of component 'rustfmt'
warning: during uninstall component rustfmt was not found
info: installing component 'cargo'
info: using up to 500.0 MiB of RAM to unpack components
info: installing component 'clippy'
info: installing component 'rust-docs'
 16.2 MiB /  16.2 MiB (100 %)   3.1 MiB/s in  7s ETA:  0s
info: installing component 'rust-mingw'
info: installing component 'rust-std'
 23.6 MiB /  23.6 MiB (100 %)  16.1 MiB/s in  1s ETA:  0s
info: installing component 'rustc'
 79.8 MiB /  79.8 MiB (100 %)  14.4 MiB/s in 19s ETA:  0s
info: installing component 'rustfmt'
info: syncing channel updates for 'nightly-x86_64-pc-windows-msvc'
info: latest update on 2021-05-07, rust version 1.54.0-nightly (676ee1472 2021-05-06)
info: downloading component 'rust-src'
info: downloading component 'rust-analysis'
info: downloading component 'rust-analyzer-preview'
info: downloading component 'rustfmt'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
 16.2 MiB /  16.2 MiB (100 %)  14.4 MiB/s in  2s ETA:  0s
info: downloading component 'rust-std'
 22.9 MiB /  22.9 MiB (100 %)  13.6 MiB/s in  2s ETA:  0s
info: downloading component 'rustc'
 59.6 MiB /  59.6 MiB (100 %)   9.6 MiB/s in  8s ETA:  0s
info: removing previous version of component 'rust-src'
info: removing previous version of component 'rust-analysis'
info: removing previous version of component 'rust-analyzer-preview'
info: removing previous version of component 'rustfmt'
info: removing previous version of component 'cargo'
info: removing previous version of component 'clippy'
info: removing previous version of component 'rust-docs'
info: removing previous version of component 'rust-std'
info: removing previous version of component 'rustc'
info: installing component 'rust-src'
info: installing component 'rust-analysis'
info: installing component 'rust-analyzer-preview'
info: installing component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
 16.2 MiB /  16.2 MiB (100 %)   3.0 MiB/s in  7s ETA:  0s
info: installing component 'rust-std'
 22.9 MiB /  22.9 MiB (100 %)   8.0 MiB/s in 11s ETA:  0s
info: installing component 'rustc'
memory allocation of 16777216 bytes failed

C:\Users\home>

Steps

  1. run rustup update

Possible Solution(s)

setting the environment variable RUSTUP_UNPACK_RAM to 21474836480 does NOT solve the problem

Notes

Output of rustup --version:

C:\Users\home>rustup --version
rustup 1.24.1 (a01bd6b0d 2021-04-27)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.52.0 (88f19c6da 2021-05-03)`

Output of rustup show:

C:\Users\home>rustup show
Default host: x86_64-pc-windows-msvc
rustup home:  D:\.rustup_home

installed toolchains
--------------------

stable-x86_64-pc-windows-msvc (default)
nightly-x86_64-pc-windows-gnu
nightly-x86_64-pc-windows-msvc

installed targets for active toolchain
--------------------------------------

x86_64-pc-windows-gnu
x86_64-pc-windows-msvc

active toolchain
----------------

stable-x86_64-pc-windows-msvc (default)
rustc 1.52.0 (88f19c6da 2021-05-03)
@wongsyrone wongsyrone added the bug label May 7, 2021
@wongsyrone
Copy link
Author

setting the environment variable RUSTUP_UNPACK_RAM to 21474836480 does NOT solve the problem.

How can I get my toolchain back as I cannot install it anymore?

@wongsyrone
Copy link
Author

I changed the value, it looks like working now.

C:\Users\home>set | findstr /I rustup
RUSTUP_HOME=D:\.rustup_home
RUSTUP_UNPACK_RAM=20000000

@kinnison
Copy link
Contributor

kinnison commented May 8, 2021

We think there's likely a problem with memory fragmentation on Windows' allocator. We are working on a fix to this in #2756 and I hope we will resolve this fairly soon.

@gruber123
Copy link

gruber123 commented May 8, 2021

In the nighltly toolchain It fails on rustc unpack allocation cannot allocate 16 mb of memory [crash],

@kinnison
Copy link
Contributor

kinnison commented May 8, 2021

It fails on rustc unpack allocation cannot allocate 16 mb of memory [crash],

@gruber123 what platform are you using?

@gruber123
Copy link

It fails on rustc unpack allocation cannot allocate 16 mb of memory [crash],

@gruber123 what platform are you using?

windows 7 x64

@kinnison
Copy link
Contributor

kinnison commented May 8, 2021

@gruber123 Cool, could you download and install https://github.com/rust-lang/rustup/suites/2685811520/artifacts/59213037 (you must be logged into github to download it) and then try again to do the install of nightly? (You may need to remove the incomplete/broken version first)

@kinnison kinnison mentioned this issue May 8, 2021
@gruber123
Copy link

Thanks, I've already installed it after closing some programs, but failing message for next updates is frustrating.
With first nigthly release after stable release (v1.52) was appeared I've tried to update nightly release (before updating to stable) and failed all the time no matter how many or less programs (memory) are opened (same memory fail on rustc), after some failure I've made full update rustup update and this time it works, today rustup update nightly same story.

@kinnison
Copy link
Contributor

kinnison commented May 8, 2021

Can you please confirm - did you install the rustup-init which I linked to above? If so, could you please run rustup dump-testament and paste the output of that into a comment here; and also run rustup toolchain remove nightly then rustup toolchain install nightly --no-self-update and confirm that either (a) it works or (b) it fails in which case also paste the log of doing that.

@rbtcollins
Copy link
Contributor

@gruber123 the custom build of rustup that @kinnison has linked may fix the problem for you, but you'll need to follow these steps:

  1. download the URL he linked, which is a zip file.
  2. inside the zip is a rustup-init.exe
  3. run that rustup-init.exe; it will re-install rustup on top of your exinsting install
  4. try uninstalling and re-installing your toolchain. Please add the option --no-self-update when doing that, as otherwise rustup will download the current released version and re-introduce the problem.

@gruber123
Copy link

C:\Users\User>rustup dump-testament
Rustup version renders as: 1.24.1+433 (5360c02 2021-05-08)
Current crate version: 1.24.1
Built from branch: remotes/pull/2756/merge
Commit info: 1.24.1+433 (5360c02 2021-05-08)
Working tree is clean

C:\Users\User>rustup toolchain remove nightly
info: uninstalling toolchain 'nightly-x86_64-pc-windows-msvc'
info: toolchain 'nightly-x86_64-pc-windows-msvc' uninstalled

C:\Users\User>rustup toolchain install nightly --no-self-update
info: syncing channel updates for 'nightly-x86_64-pc-windows-msvc'
info: latest update on 2021-05-09, rust version 1.54.0-nightly (881c1ac40 2021-05-08)
info: downloading component 'cargo'
3.6 MiB / 3.6 MiB (100 %) 2.8 MiB/s in 1s ETA: 0s
info: downloading component 'clippy'
info: downloading component 'rust-docs'
16.6 MiB / 16.6 MiB (100 %) 2.0 MiB/s in 8s ETA: 0s
info: downloading component 'rust-std'
19.3 MiB / 19.3 MiB (100 %) 2.2 MiB/s in 7s ETA: 0s
info: downloading component 'rustc'
59.6 MiB / 59.6 MiB (100 %) 2.8 MiB/s in 27s ETA: 0s
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
16.6 MiB / 16.6 MiB (100 %) 1.3 MiB/s in 1m 0s ETA: 0s
info: installing component 'rust-std'
19.3 MiB / 19.3 MiB (100 %) 9.2 MiB/s in 1m 46s ETA: 0s
info: installing component 'rustc'
59.6 MiB / 59.6 MiB (100 %) 9.3 MiB/s in 6s ETA: 0s
info: installing component 'rustfmt'

nightly-x86_64-pc-windows-msvc installed - rustc 1.54.0-nightly (881c1ac40 2021-05-08)

C:\Users\User>rustc --version
rustc 1.54.0-nightly (881c1ac40 2021-05-08)

@gruber123
Copy link

Ok, it worked without closing extra programs

So what is custom build of rustup-init, how it's different from official version on main page?
Also on failure cases temp directory not cleaned.

@kinnison
Copy link
Contributor

kinnison commented May 9, 2021

The version we asked you to try is a work-in-progress to fix the issue on the main version. I will look to merge that tomorrow and then we can try and prepare a new beta release. I'll keep you updated when that's done and I'll ask you to install and try the beta release then. I'll provide instructions at that point.

@gruber123
Copy link

again some problem:

C:\Users\User>rustup update nightly --no-self-update
info: syncing channel updates for 'nightly-x86_64-pc-windows-msvc'
info: latest update on 2021-05-10, rust version 1.54.0-nightly (ca82264ec 2021-05-09)
info: downloading component 'rust-analysis'
info: downloading component 'rls'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
16.6 MiB / 16.6 MiB (100 %) 6.1 MiB/s in 3s ETA: 0s
info: downloading component 'rust-std'
19.3 MiB / 19.3 MiB (100 %) 8.6 MiB/s in 3s ETA: 0s
info: downloading component 'rustc'
59.7 MiB / 59.7 MiB (100 %) 6.7 MiB/s in 12s ETA: 0s
info: downloading component 'rustfmt'
info: removing previous version of component 'rust-analysis'
info: removing previous version of component 'rls'
info: removing previous version of component 'cargo'
info: removing previous version of component 'clippy'
info: removing previous version of component 'rust-docs'
info: removing previous version of component 'rust-std'
info: removing previous version of component 'rustc'
info: removing previous version of component 'rustfmt'
info: installing component 'rust-analysis'
info: installing component 'rls'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
16.6 MiB / 16.6 MiB (100 %) 382.4 KiB/s in 1m 43s ETA: 0s
info: installing component 'rust-std'
19.3 MiB / 19.3 MiB (100 %) 7.8 MiB/s in 2m 20s ETA: 0s
info: installing component 'rustc'
59.7 MiB / 59.7 MiB (100 %) 9.0 MiB/s in 9s ETA: 0s
info: installing component 'rustfmt'
warning: could not delete temp file: C:\Users\Igor.rustup\tmp\qcng398bkm269lgw_file
warning: could not delete temp file: C:\Users\Igor.rustup\tmp\9hmu8aii0zdyoj5m_file
warning: could not delete temp file: C:\Users\Igor.rustup\tmp\n5m5iu6vc0nnrjmq_file

nightly-x86_64-pc-windows-msvc updated - rustc 1.54.0-nightly (ca82264ec 2021-05-09) (from rustc 1.54.0-nightly (881c1ac40 2021-05-08))

@gruber123
Copy link

gruber123 commented May 10, 2021

2021-05-10_183311

after update rustup still in processes of tasks list

I think it's because rls.exe was in processes also when also vs code was opened, It's not the big problem but could be better to ask for close programs that interfere with.

@FoxKyong
Copy link

I have exactly the same problem but on Raspberry Pi 3. Unitl now I was able to update with 'rustup update' without a problem. Now I can't update it nor do a clean install.

info: profile set to 'default'
info: default host triple is armv7-unknown-linux-gnueabihf
info: syncing channel updates for 'stable-armv7-unknown-linux-gnueabihf'
info: latest update on 2021-05-10, rust version 1.52.1 (9bc8c42bb 2021-05-09)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-std'
 18.8 MiB /  18.8 MiB (100 %)   5.8 MiB/s in  4s ETA:  0s
info: downloading component 'rustc'
 84.0 MiB /  84.0 MiB (100 %)   4.6 MiB/s in 23s ETA:  0s
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: using up to 500.0 MiB of RAM to unpack components
  5.5 MiB /   5.5 MiB (100 %)   3.2 MiB/s in  1s ETA:  0s
info: installing component 'clippy'
info: installing component 'rust-std'
 18.8 MiB /  18.8 MiB (100 %)   3.1 MiB/s in  6s ETA:  0s
  9 IO-ops /   9 IO-ops (100 %)   0 IOPS in  3s ETA: Unknown
info: installing component 'rustc'
 18.2 MiB /  84.0 MiB ( 22 %)   3.7 MiB/s in  9s ETA: 17smemory allocation of 16777216 bytes failed
Aborted

@kinnison
Copy link
Contributor

@gruber123 As we state in the release announcements, you must ensure there are no IDEs etc open when updating rust toolchains, for exactly that reason.

@rbtcollins
Copy link
Contributor

@FoxKyong if you wish to try a build of master with our candidate fix in it, you can find it here https://github.com/rust-lang/rustup/suites/2703495866/artifacts/59599296

@kinnison
Copy link
Contributor

I've updated the beta testing post on the users forum with details of how to test our proposed fix - https://internals.rust-lang.org/t/seeking-testers-for-rustup-1-24-2-beta/14634/6

@tatsuya6502
Copy link

Thank you for addressing this issue and letting us know about the beta testing. I asked one of the users who reported a duplicate issue #2764 on Raspberry Pi 3 to join the beta testing. So far, so good; he verified that Rustup v1.24.2 beta can install the latest Rust stable tool chain without problem.

Raspberry Pi 3 running Raspbian Buster arm32 (armv7-unknown-linux-gnueabihf)

$ rustup self uninstall
$ export RUSTUP_UPDATE_ROOT=https://dev-static.rust-lang.org/rustup
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ rustup dump-testament
Rustup version renders as: 1.24.2 (34cd8a5b7 2021-05-10)
Current crate version: 1.24.2
Built from branch: stable
Commit info: 1.24.1+76 (34cd8a5b7 2021-05-10)
Working tree is clean
$ rustc -V
rustc 1.52.1 (9bc8c42bb 2021-05-09)

@kinnison
Copy link
Contributor

That's great news, thank you for that. We'll hold off a little while in case other reports come in, but I think we have a candidate for 1.24.2 :D

@ghost
Copy link

ghost commented May 11, 2021

I also got this problem randomly with newly installed Windows 10 21H1 and WSL. I saw an allocation error on the first try, and the second was successful.

@ghost
Copy link

ghost commented May 11, 2021

info: profile set to 'default'
info: default host triple is x86_64-pc-windows-msvc
info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'
info: latest update on 2021-05-10, rust version 1.52.1 (9bc8c42bb 2021-05-09)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: using up to 500.0 MiB of RAM to unpack components
info: installing component 'clippy'
info: installing component 'rust-docs'
info: installing component 'rust-std'
memory allocation of 16777216 bytes failed

@kinnison
Copy link
Contributor

kinnison commented May 11, 2021

@ghost Are you using the 1.24.2 beta release as per the above comments?

@ghost
Copy link

ghost commented May 11, 2021

@kinnison I downloaded from https://win.rustup.rs.

@ghost
Copy link

ghost commented May 11, 2021

@kinnison I'm using 1.24.1.

$ rustup -v
rustup 1.24.1 (a01bd6b0d 2021-04-27)
The Rust toolchain installer
...

@kinnison
Copy link
Contributor

kinnison commented May 11, 2021

@ghost Could you please follow the instructions in https://internals.rust-lang.org/t/seeking-testers-for-rustup-1-24-2-beta/14634/6 to get the beta copy and try that?

@ghost
Copy link

ghost commented May 11, 2021

After changing the version to beta, I was able to see a significant difference. When running rust-analyzer in vscode with rustup version 1.24.1 it took a very long time than usual. With version 1.24.2 beta it took me the usual time again. Maybe rust-analyzer uses rustup internally.

@FoxKyong
Copy link

FoxKyong commented May 11, 2021

@FoxKyong if you wish to try a build of master with our candidate fix in it, you can find it here https://github.com/rust-lang/rustup/suites/2703495866/artifacts/59599296

Thanks, I just tried it and it works. Installation works but I noticed another very minor issue. It adds . "$HOME/.cargo/env" to both .profile and .bashrc. It should by only at one place and I think it was before only in .profile.

@kinnison
Copy link
Contributor

@FoxKyong Because the sourcing of the env file is designed to be safe to do multiple times, we now scattergun the line into as many places as possible to cover as many different shells as we can.

@wongsyrone
Copy link
Author

I tried the beta version, works smoothly. Great job.

@wongsyrone
Copy link
Author

Closing as the problem has been fixed.

hkratz added a commit to hkratz/simd-json that referenced this issue May 13, 2021
hkratz added a commit to rusticstuff/simdutf8 that referenced this issue May 13, 2021
Licenser pushed a commit to simd-lite/simd-json that referenced this issue May 13, 2021
bors added a commit to rust-lang/cargo that referenced this issue May 18, 2021
Add temporary fix for rustup on windows in CI.

This adds a temporary fix for rustup on the Windows CI runners. The GitHub image updated to rustup 1.24.1 which has an issue (rust-lang/rustup#2759) causing them to fail.  This is fixed in 1.24.2.  I believe the images are updated on roughly a weekly basis, and from what I can see the image just downloads the most recently release, so hopefully this will be fixed on GitHub's side in roughly a week. Until then, this will force rustup to update.

Note: Self updates aren't available on macOS images, but they seem to work on the others.
weiznich added a commit to diesel-rs/diesel that referenced this issue May 31, 2021
bors added a commit to rust-lang/cargo that referenced this issue May 25, 2023
chore(ci): remove temporary fix for rustup 1.24.1

rust-lang/rustup#2759 was fixed in 1.24.2 and now GitHub ships 1.26.0 [^1]. The temporary workaround is not needed anymore. Let's see if everything goes well!

[^1]: https://github.com/actions/runner-images/blob/156ad8a318284c617f76cef1077cd0d64aeb758e/images/linux/Ubuntu2204-Readme.md?plain=1#L151
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants