Get rid of sha1 for fixed-output derivations in nixpkgs #77238

grahamc · 2020-01-07T15:45:08Z

Issue description

We're in 2020 and:

chosen-prefix sha1 collisions can now be found for as little as $45,000
a classic collision can be found for as little as $11k

Biggest usage of sha1 is files generated for Node packages:

2372	pkgs/development/node-packages/node-packages-v10.nix
1741	pkgs/servers/web-apps/codimd/yarn.nix
1644	pkgs/applications/version-management/gitlab/yarnPkgs.nix
1485	pkgs/servers/monitoring/prometheus/webui-yarndeps.nix
1481	pkgs/servers/gotify/yarndeps.nix
476	pkgs/development/tools/yarn2nix-moretea/yarn2nix/yarn.nix
421	pkgs/applications/networking/cluster/spacegun/node-packages.nix
290	pkgs/servers/rippled/package.nix
213	pkgs/development/compilers/elm/packages/node-packages.nix
189	pkgs/development/web/remarkjs/node-packages.nix
169	pkgs/development/mobile/androidenv/generated/packages.nix
141	pkgs/development/node-packages/node-packages-v12.nix
127	pkgs/applications/networking/instant-messengers/riot/riot-desktop-yarndeps.nix
116	pkgs/servers/matrix-synapse/matrix-appservice-slack/node-packages.nix
88	pkgs/misc/base16-builder/node-packages-generated.nix
74	pkgs/development/mobile/androidenv/generated/addons.nix
73	pkgs/development/node-packages/node-packages-v13.nix
70	pkgs/development/compilers/graalvm/default.nix
66	pkgs/applications/networking/instant-messengers/matrix-recorder/node-packages.nix
48	pkgs/tools/networking/airfield/node-packages.nix
48	pkgs/development/misc/google-clasp/node-packages.nix
36	pkgs/development/mobile/androidenv/generated/system-images-android.nix
33	pkgs/servers/web-apps/cryptpad/node-packages-generated.nix
33	pkgs/development/mobile/androidenv/generated/system-images-google_apis.nix
10	pkgs/development/mobile/androidenv/generated/system-images-android-tv.nix
6	pkgs/tools/package-management/nixui/node-packages.nix
6	pkgs/development/mobile/androidenv/generated/system-images-google_apis_playstore.nix
6	pkgs/development/mobile/androidenv/generated/system-images-android-wear.nix
6	pkgs/development/mobile/androidenv/convertaddons.xsl
4	pkgs/development/mobile/androidenv/generated/system-images-android-wear-cn.nix
2	pkgs/development/mobile/androidenv/convertpackages.xsl
2	pkgs/applications/networking/browsers/mozilla-plugins/google-talk-plugin/default.nix
1	pkgs/tools/typesetting/tex/texlive/default.nix
1	pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/generateNix.js
1	pkgs/development/tools/unity3d/default.nix
1	pkgs/development/mobile/androidenv/convertsystemimages.xsl
1	pkgs/development/libraries/wxsqliteplus/default.nix
1	pkgs/development/libraries/wxsqlite3/default.nix
1	pkgs/development/haskell-modules/configuration-hackage2nix.yaml
1	pkgs/development/haskell-modules/configuration-common.nix
1	pkgs/build-support/vm/rpm/rpm-closure.pl
1	pkgs/applications/office/grisbi/default.nix
1	pkgs/applications/graphics/gcolor2/default.nix
1	doc/release-notes.xml

Steps to resolve, higher level

generator tools should be updated to use better hashes:
- node2nix
- yarn2nix
generator tools should be updated to print out how they were generated:
- yarn2nix-moretea.yarn2nix
update all the packages which use generated dependency files
update all the packages which use sha1 without a generator

Files to address

The text was updated successfully, but these errors were encountered:

related to: NixOS#77238

related NixOS#77238

edolstra · 2020-01-07T20:55:21Z

We can script most of this by using maintainers/scripts/find-tarballs.nix to get all the URLs/hashes, fetching them and then doing s/$sha1/$sha256 on the Nix expression. Or use tarballs.nixos.org to map sha1 hashes to sha512:

$ curl -v https://tarballs.nixos.org/sha1/e410a52dcff3d5c6c3d448b68a026d04ccd744be
location: /sha512/90ae204071cdab3b5e0da28709afa8ac0a5ee13737254cded76a6b8cf323ed2b42390aaa10dbb7935b2aaa57504ceba21306a7eaeea088ab85ee9bdb0a05b530

cdepillabout · 2020-01-08T03:25:13Z

@grahamc Thanks for looking into this. I think these types of security related fixes are important!

I was wondering how you determined the list of files using sha1?

I wanted to fix the haskell-related files (pkgs/development/haskell-modules/configuration-common.nix and pkgs/development/haskell-modules/configuration-hackage2nix.yaml), so I grepped through the files looking for the strings sha1 and hash, but I couldn't find anything suspicious. Also configuration-hackage2nix.yaml doesn't appear to have any hashes at all.

grahamc · 2020-01-08T03:26:12Z

I did a simple search of sha1 =. It has a few false positives, for sure :) Please tick them as done if you find them!

7c6f434c · 2020-01-08T11:14:12Z

git grep -l -E '(^| )sha1\> *= *' should catch less false positives, I believe.

Once we clean up all the generators, we can rerun with \<sha1\> *= as a safety check.

kira-bruneau · 2020-01-10T01:36:12Z

All the packages in node-packages-v10.nix that use sha1, only do so because they don't list a sha512 hash in the npm registry. Regenerating this file is already slow enough, if we wanted to use sha256 or sha512, node2nix (and yarn2nix) would also have to download the files.

For example, acorn 4.0.13 doesn't have a sha512 hash:

nix-shell -p curl jq --run 'curl http://registry.npmjs.org/acorn/4.0.13 | jq .dist'

{
  "shasum": "105495ae5361d697bd195c825192e1ad7f253787",
  "tarball": "https://registry.npmjs.org/acorn/-/acorn-4.0.13.tgz"
}

but version 7.1.0 does:

nix-shell -p curl jq --run 'curl http://registry.npmjs.org/acorn/7.1.0 | jq .dist'

{
  "integrity": "sha512-kL5CuoXA/dgxlBbVrflsflzQ3PAas7RYZB52NOm/6839iVYJgKMJ3cQJD+t2i5+qFa8h3MDpEOJiS64E8JLnSQ==",
  "shasum": "949d36f2c292535da602283586c2477c57eb2d6c",
  "tarball": "https://registry.npmjs.org/acorn/-/acorn-7.1.0.tgz",
  "fileCount": 11,
  "unpackedSize": 1104477,
  "npm-signature": "-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJdibjuCRA9TVsSAnZWagAAmQgQAIkJPo9i+wdBLLZY8yjT\nHI82RsWU5OvIQxhE6H9qLiEfF57ZSn3LfKVugsEocc84loCiJkkd7vsGsidJ\nRMbkEIK2ENjH1VcUHdHq8PnA/HgnMgHNq6nGg+h6aXLsN/H5SFdyvWedoLXw\n5bwhLbfURV/00bx2himrt/RBNL8SxfowdpQ/ps5+mAWUzLTf7D2+PchYktve\nRJ1lSwpshrgu4AxUdbRmqDEH7+Wm5SutB/EMkQPxQV/sJviFzB8qqFlQopJS\n83RuvIhQb7ZNUQeh5A1+pQocFU+yd227m9tAr69UeES8vinI+0FYVh5GNh0y\nk5MoiQiG3LROtO1RNea0uYGv8W7cU9CK/6MnHbDMUgr+fpEeU/VGItZfOLhB\nTg10PpYCKtQCsF+Z36YLTXgxeWEoRvvKWmJET/MDQzBM/pRzMRpPHiWn6E9T\nl9wt1WLix2qM1jymwaq+0H6hcCtu0ioY7qn4GGeCAfVlBxWYcydNR11Rb++n\nO3oXEqhc8oxKy+hIDHIMDDz/YfCrJ9q7ZzMO9Ie5dzOqeLxty4fbuDeLcD9U\nkbuA+I8dfEFmO2UtRwyx3k9yjX2Tsf2oAC8VOTINN8TXiauyTjWyU//lSKDX\nmUHBDXXZhKWgygi3TMNW7gvmzYuzmDcbkienQeoADvONpCuJNmraVaC33n1h\ncQmj\r\n=blZm\r\n-----END PGP SIGNATURE-----\r\n"
}

related NixOS#77238 (cherry picked from commit 741db8c)

vcunat · 2020-01-11T08:51:04Z

texlive: it has one "sha1 =" string but the code is generic – pkgs/tools/typesetting/tex/texlive/fixedHashes.nix has over 8k sha1 hashes. I haven't looked into complications when generating these, but if I look right, the file's size would increase roughly 460k -> 624k. (Seems acceptable, I suppose, given the circumstances.)

worldofpeace · 2020-02-05T20:36:34Z

@grahamc I removed this as a blocker because I'm not convinced it's something we should block release on. Thoughts?

prusnak · 2020-05-05T17:11:43Z

pkgs/applications/networking/browsers/mozilla-plugins/google-talk-plugin/default.nix is not an issue since c415d67 has been merged

prusnak · 2020-05-05T17:15:51Z

pkgs/build-support/vm/rpm/rpm-closure.pl is a false positive

prusnak · 2020-05-05T17:22:52Z

pkgs/applications/office/grisbi/default.nix -> grisbi: get rid of sha1 + package cleanup #86966

prusnak · 2020-05-05T17:46:35Z

pkgs/development/libraries/wxsqliteplus/default.nix -> wxsqlite3: 3.3.1 -> 4.5.1 #86969
pkgs/development/libraries/wxsqlite3/default.nix -> wxsqlite3: 3.3.1 -> 4.5.1 #86969

xworld21 · 2021-06-19T12:37:12Z

texlive: it has one "sha1 =" string but the code is generic – pkgs/tools/typesetting/tex/texlive/fixedHashes.nix has over 8k sha1 hashes. I haven't looked into complications when generating these, but if I look right, the file's size would increase roughly 460k -> 624k. (Seems acceptable, I suppose, given the circumstances.)

@vcunat (edit: I should be asking @veprbl) I can do the sha1 -> sha256 conversion for texlive. Now the increase is 498k -> 676k. Just give a thumbs up if you'd like a PR (or a thumbs down!).

vcunat · 2021-06-19T17:03:31Z

Well, I don't maintain texlive anymore, for years already. But I assume that what I wrote above still holds (approximately).

Stunkymonkey · 2021-07-14T17:26:04Z

i think the remaining sha1-packages are all related to node2nix or yarn2nix.

Whenever available use the SRI hashes from the integrity field to create the fetchurl calls instead of entirely relying on the `resolved` sha1 which may or may not exist with recent yarn versions. Related issues: - nix-community/yarn2nix#125 - NixOS#77238

jtojnar · 2021-12-09T00:52:14Z

yarn2nix will now use the integrity field from yarn.lock when present (available from at least 2018), which is typically sha512 for packages ¿uploaded to npm in the last X years? – for example, yarn.lock of a random modern project contains 319 sha1 and 1624 sha512 entries.

We probably still want to port the TOFU script https://github.com/NixOS/nixpkgs/blob/bb9bd465b625bfc971908c5d3d84ce517e1c0691/pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/fixPkgAddMissingSha1.js to a different hash schema and maybe even switch to TOFU for packages using sha1 in integrity field.

prusnak · 2021-12-09T11:23:35Z

We probably still want to port the TOFU script https://github.com/NixOS/nixpkgs/blob/bb9bd465b625bfc971908c5d3d84ce517e1c0691/pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/fixPkgAddMissingSha1.js to a different hash schema

I sent a draft PR #149834

Whenever available use the SRI hashes from the integrity field to create the fetchurl calls instead of entirely relying on the `resolved` sha1 which may or may not exist with recent yarn versions. Related issues: - nix-community/yarn2nix#125 - NixOS#77238

aikooo7 · 2024-02-10T21:54:23Z

Hello, I would like to help with this, any suggestions where I can get started?

nbraud · 2024-09-15T11:53:39Z

I went and updated the list in the issue, checking all files not yet marked as done (in some cases updating their path) and adding a few I found with rg -l -g '*.nix' -g '*.json' 'sha1 = "|"sha1": "|"sha1-' pkgs/.

PS: I looked through the remaining cases, and

androidenv generates those files based on upstream metadata from https://dl.google.com/android/repository/, which only contains SHA-1 hashes ;
same for minecraft-servers, getting metadata from https://launchermeta.mojang.com/ ;
rippled seems to be a false negative: SHA-1 hashes exist in its package.nix file... which nothing seems to refer to?
I opened rippled: drop dead package.nix file #342054
yarn2nix-moretea has been discussed above.

The most sensible approach for androidenv and minecraft-servers might be to get the upstream to publish metadata with non-broken hashes. Do we have relevant contacts at Google and Mojang?

grahamc added this to the 20.03 milestone Jan 7, 2020

grahamc changed the title ~~Get rid of sha1 support for fixed-output derivations~~ Get rid of sha1 for fixed-output derivations in nixpkgs Jan 7, 2020

This was referenced Jan 7, 2020

Force a move from sha1 to sha256 or sha512 svanderburg/node2nix#166

Open

Force a move from sha1 to sha256 or sha512 nix-community/yarn2nix#125

Open

turboMaCk added a commit to turboMaCk/nixpkgs that referenced this issue Jan 7, 2020

gcolor2: sha1 -> sha256

eb58df4

related to: NixOS#77238

turboMaCk mentioned this issue Jan 7, 2020

gcolor2: sha1 -> sha256 #77244

Merged

10 tasks

turboMaCk mentioned this issue Jan 7, 2020

google_talk_plugin: sha1 -> sha256 #77245

Merged

10 tasks

turboMaCk added a commit to turboMaCk/nixpkgs that referenced this issue Jan 7, 2020

google_talk_plugin: sha1 -> sha256

741db8c

related NixOS#77238

dtzWill pushed a commit to dtzWill/nixpkgs that referenced this issue Jan 11, 2020

google_talk_plugin: sha1 -> sha256

d59574a

related NixOS#77238 (cherry picked from commit 741db8c)

worldofpeace removed the 1.severity: blocker This is preventing another PR or issue from being completed label Feb 5, 2020

prusnak mentioned this issue May 5, 2020

grisbi: get rid of sha1 + package cleanup #86966

Merged

10 tasks

prusnak mentioned this issue May 5, 2020

wxsqlite3: 3.3.1 -> 4.5.1 #86969

Merged

10 tasks

prusnak mentioned this issue May 30, 2020

nodePackages: Switch to 12.x and 14.x, continue #85764 #89184

Merged

10 tasks

AmineChikhaoui mentioned this issue Apr 15, 2021

yarn2nix: use yarn lockfile integrity field whenever possible #119522

Merged

10 tasks

veprbl removed this from the 20.03 milestone May 31, 2021

Stunkymonkey mentioned this issue Jun 29, 2021

unity3d: update sha1 to sha256 #128631

Merged

11 tasks

Stunkymonkey mentioned this issue Jul 13, 2021

texlive: use sha256 #130097

Merged

11 tasks

bleys mentioned this issue Oct 12, 2021

Toolchain: Build pretty. Use xz, not gz. Use sha256, not md5 SerenityOS/serenity#10449

Closed

prusnak mentioned this issue Dec 9, 2021

yarn2nix: use sha512 by default #149834

Closed

13 tasks

vcunat mentioned this issue Jul 26, 2022

fetchurl: disallow specifying both sha256 and hash #182953

Merged

jtojnar mentioned this issue Oct 26, 2022

yarn2nix: Handle lockfile entries with multiple integrity hashes #197185

Merged

13 tasks

jtojnar mentioned this issue Feb 14, 2024

grafanaPlugins: fix update script/add some interesting data source plugins #288642

Merged

13 tasks

samueldr added the 5. scope: tracking Long-lived issue tracking long-term fixes or multiple sub-problems label Apr 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get rid of sha1 for fixed-output derivations in nixpkgs #77238

Get rid of sha1 for fixed-output derivations in nixpkgs #77238

grahamc commented Jan 7, 2020 •

edited by Artturin

Loading

edolstra commented Jan 7, 2020

cdepillabout commented Jan 8, 2020

grahamc commented Jan 8, 2020

7c6f434c commented Jan 8, 2020

kira-bruneau commented Jan 10, 2020 •

edited

Loading

vcunat commented Jan 11, 2020

worldofpeace commented Feb 5, 2020 •

edited

Loading

prusnak commented May 5, 2020 •

edited

Loading

prusnak commented May 5, 2020 •

edited

Loading

prusnak commented May 5, 2020 •

edited

Loading

prusnak commented May 5, 2020 •

edited by grahamc

Loading

xworld21 commented Jun 19, 2021 •

edited

Loading

vcunat commented Jun 19, 2021

Stunkymonkey commented Jul 14, 2021

jtojnar commented Dec 9, 2021

prusnak commented Dec 9, 2021

aikooo7 commented Feb 10, 2024

nbraud commented Sep 15, 2024 •

edited

Loading

Get rid of sha1 for fixed-output derivations in nixpkgs #77238

Get rid of sha1 for fixed-output derivations in nixpkgs #77238

Comments

grahamc commented Jan 7, 2020 • edited by Artturin Loading

Issue description

Steps to resolve, higher level

Files to address

edolstra commented Jan 7, 2020

cdepillabout commented Jan 8, 2020

grahamc commented Jan 8, 2020

7c6f434c commented Jan 8, 2020

kira-bruneau commented Jan 10, 2020 • edited Loading

vcunat commented Jan 11, 2020

worldofpeace commented Feb 5, 2020 • edited Loading

prusnak commented May 5, 2020 • edited Loading

prusnak commented May 5, 2020 • edited Loading

prusnak commented May 5, 2020 • edited Loading

prusnak commented May 5, 2020 • edited by grahamc Loading

xworld21 commented Jun 19, 2021 • edited Loading

vcunat commented Jun 19, 2021

Stunkymonkey commented Jul 14, 2021

jtojnar commented Dec 9, 2021

prusnak commented Dec 9, 2021

aikooo7 commented Feb 10, 2024

nbraud commented Sep 15, 2024 • edited Loading

grahamc commented Jan 7, 2020 •

edited by Artturin

Loading

kira-bruneau commented Jan 10, 2020 •

edited

Loading

worldofpeace commented Feb 5, 2020 •

edited

Loading

prusnak commented May 5, 2020 •

edited

Loading

prusnak commented May 5, 2020 •

edited

Loading

prusnak commented May 5, 2020 •

edited

Loading

prusnak commented May 5, 2020 •

edited by grahamc

Loading

xworld21 commented Jun 19, 2021 •

edited

Loading

nbraud commented Sep 15, 2024 •

edited

Loading