Get rid of md5 support for fixed-output derivations

[domenkozar]

We're in 2014 and even universities have a course where students forge md5 hashes of files.

Biggest usage of md5 hashes in nixpkgs is python, followed by libreoffice (scripted install).

$ git grep "md5 ="|grep -v libreoffice | grep -v python-|grep -v redhat|grep -v suse|wc -l                                                                                                        
141

$ git grep "md5 =" pkgs/top-level/python-packages.nix | wc -l
273

Observations:

nix-prefetch- should print out multiple hashes together with fetch functions supporting and verifying all of specified hashes

Q/A:

it is considered best practice to use it when that's what upstream provides

That's a very bad security practice. It trades user security for few seconds of maintainer time.

TODO

• libreoffice uses md5 during generation
• pkgs/games/steam/runtime-generated.nix uses md5 during generation
• 91 other derivations
• deprecate md5 support
• revert 2ca8833

[robberer]

This is not as trivial as i thought because nix-prefetch-url does not check if the downloaded archive is a valid archive. e.g. my update for cups is invalid because nix-prefetch-url just downloaded a HTML file instead of an tar.bz2. i will prepare a fix. At least for cups. Or should we revert commit ?

[domenkozar]

Fixing cups is OK.

[copumpkin]

👍

[copumpkin]

Can we broaden this to also deprecate sha1 for similar reasons? It's not as broken as md5, but it's also not considered particularly good when used for security purposes.

[copumpkin]

Perhaps fetchurl and friends could even display a warning when passed md5 or sha1s that tells you that the practice is deprecated and give you a sha256 to replace it with in the expression.

[7c6f434c]

Can we broaden this to also deprecate sha1 for similar reasons? It's not as broken as md5, but it's also not considered particularly good when used for security purposes.

Unfortunately, some projects still publish signed lists of SHA1s and
some Nix package maintainers seem to use these lists.

Another problem is that we don't seem to support SHA512: Firefox has
official SHA1 list and SHA512 list.

[copumpkin]

Perhaps we could then include both, making the sha256 attribute mandatory? The sha1/md5 can be used to check against upstream lists, whereas the sha256 serves to give you certainty that what you're deploying is what you think it is.

[7c6f434c]

Perhaps we could then include both, making the sha256 attribute mandatory? The sha1/md5 can be used to check against upstream lists, whereas the sha256 serves to give you certainty that what you're deploying is what you think it is.

For firefox-bin the whole feasibility of including all the languages
hinges on not downloading all the packages but using Mozilla-provided
checksum lists.

[vcunat]

I think forbidding sha1 is a huge overkill ATM. In most packages there's even no way of securely retrieving the hash/tarball (no signature or https at least).

If I wanted to attack nixpkgs, I'd become a contributor, so I'd get commit access relatively easily, and then I could sneak in whatever forged hashes I needed (unlikely to be noticed if done well).

[copumpkin]

NixOS/nix#802

[vcunat]

ATM the largest amount of md5 usage is from the new texlive, as its list of tarball hashes is taken from upstream and they only provide md5.

However, the transformed fixed-output derivations use sha1 already, so md5 isn't really used by regular users.

[7c6f434c]

ATM the largest amount of md5 usage is from the new texlive, as its list of tarball hashes is taken from upstream and they only provide md5.

However, the transformed fixed-output derivations use sha1 already, so md5 isn't really used by regular users.

We also have LibreOffice dependency list which is MD5-from-upstream.

[copumpkin]

It sounds like TeXLive is getting with the times: https://www.preining.info/blog/2016/01/tex-live-security-improvements/

[copumpkin]

@7c6f434c LibreOffice's main download page took me here: http://download.documentfoundation.org/libreoffice/stable/5.1.0/mac/x86_64/LibreOffice_5.1.0_MacOS_x86-64.dmg.mirrorlist which seems to provide a fine sha256.

[vcunat]

Yeah, LibreOffice provides those now; note that they also serve the hashes over https URL.

[7c6f434c]

@7c6f434c LibreOffice's main download page took me here: http://download.documentfoundation.org/libreoffice/stable/5.1.0/mac/x86_64/LibreOffice_5.1.0_MacOS_x86-64.dmg.mirrorlist which seems to provide a fine sha256.

It has around a metric ton of inner dependencies that are built in some
special ways inside LO build. They are listed in downloads.lst — with
md5 sums. And they are, ahem, quite numerous. Some of them have been
factored out, but I remember returning some of them back into the LO
build becuase of heavy breakage.

[copumpkin]

Has anyone raised the issue upstream to see if they can just switch that
process over to a more sensible hash?
On Mon, Feb 15, 2016 at 15:12 Michael Raskin [email protected]
wrote:

@7c6f434c LibreOffice's main download page took me here:
http://download.documentfoundation.org/libreoffice/stable/5.1.0/mac/x86_64/LibreOffice_5.1.0_MacOS_x86-64.dmg.mirrorlist
which seems to provide a fine sha256.

It has around a metric ton of inner dependencies that are built in some
special ways inside LO build. They are listed in downloads.lst — with
md5 sums. And they are, ahem, quite numerous. Some of them have been
factored out, but I remember returning some of them back into the LO
build becuase of heavy breakage.

—
Reply to this email directly or view it on GitHub
#4491 (comment).

[grahamc]

FWIW: I submitted a (long-merged) PR reducing the # of md5s down from 273 to 13, and sha1s from ?? to 0. I implemented it by taking the current URLs, fetching them, comparing md5s, then updating to sha256 only if they updated.

[pSub]

The situation of non-python packages actually got worse (compared the the initial values):

$ git grep "md5 ="|grep -v libreoffice | grep -v python-|grep -v redhat|grep -v suse|wc -l
516

Edit: It seems that is mainly because of pkgs/games/steam/runtime-generated.nix.

[domenkozar]

We still need to implement deprecation (reverting 2ca8833)

[7c6f434c]

Well, technically you want to revert the removal of warnings.

I think 2ca8833 has been effectively reverted in e3a8734

Is the plan now to apply 717ff85 once more?

@domenkozar do you have better ideas?
@edolstra would you object to such a plan?

[grahamc]

What if we just deleted support for specifying md5 = ?

[7c6f434c]

I think deprecation is supposed to annoy people with out-of-tree Nix packages to finally update with some time window, instead of abrupt disappearance of functionality?

(I do not have any personal interest in NixPkgs fetch* MD5 support, though)

[copumpkin]

We're going to need a new one of these for sha1...

[vcunat]

So signing git commits is useless now? EDIT: I just started doing so a couple months ago...

[copumpkin]

Not fully useless (it's collision not preimage), and the attack is still pretty expensive, but I expect cost to go down significantly soon given how much relies on SHA1 nowadays.

[vcunat]

Well, actually when we sign git commits, we certify rather just the changes and not the whole history. (Noone's verifying signatures on ancestors AFAIK.)

[jpierre03]

On 3fdd726

Total count of "md5 =" in nixpkgs is huge.

$ git grep "md5 ="|wc -l
225

Libreoffice is top 1 project using md5.

$ git grep "md5 ="|grep libreoffice|wc -l
221

Projects without libreoffice

$ git grep "md5 =" |grep -v libreoffice|wc -l
4

$ git grep "md5 =" |grep -v libreoffice
pkgs/development/haskell-modules/configuration-common.nix:  apache-md5 = dontCheck super.apache-md5;              # http://hydra.cryp.to/build/498709/nixlog/1/raw
pkgs/development/haskell-modules/configuration-hackage2nix.yaml:  - cryptohash-md5 ==0.11.100.1
pkgs/development/lisp-modules/lisp-packages.nix:  md5 = buildLispPackage rec {
pkgs/development/tools/egg2nix/chicken-eggs.nix:  md5 = eggDerivation {

None of them is a md5 hash. Good !

[jpierre03]

Should an issue for sha1 be opened ?

$ git grep "sha1 ="|wc -l
8839

$ git grep "sha256 ="|wc -l
39903

Node-packages are top 1 on sha1 ( svanderburg/node2nix#39 )

$ git grep "sha1 ="|grep node-packages|wc -l
7562

[globin]

Libreoffice uses sha256 to check the download:

(map (x : ((fetchurl {inherit (x) url sha256 name;}) // {inherit (x) md5name md5;})) (import ./libreoffice-srcs.nix));

Example entry:

  {
    name = "libabw-0.1.1.tar.bz2";
    url = "http://dev-www.libreoffice.org/src/libabw-0.1.1.tar.bz2";
    sha256 = "7a3d3415cf82ab9894f601d1b3057c4615060304d5279efdec6275e01b96a199";
    md5 = "7a3815b506d064313ba309617b6f5a0b";
    md5name = "7a3815b506d064313ba309617b6f5a0b-libabw-0.1.1.tar.bz2";
  }

I don't see any remaining packages using md5 so I'd propose removing the support in the fetchers on master, fixing this in projects outside of nixpkgs is easy and shouldn't be used anyway.

[Baughn]

This bit me with Curseforge, where I'm generating nix code by scraping their download pages. They only provide MD5, and I don't want to download the files themselves; it'd take far too much bandwidth, and might get me banned.

Could you reconsider disabling md5, or maybe gate it behind a __YES_I_REALLY_WANT_THIS type flag? The only alternatives I can see are to fork fetchurl, or to disable hash checking entirely, and I'd prefer to do neither. I've tried asking them to provide a better hash, but it doesn't seem like that will happen anytime soon, especially as the service has no official API.

[edolstra]

@Baughn Nix's builtin fetchurl still supports md5, so you may be able to do:

import <nix/fetchurl.nix> { url = ...; md5 = "7f8b9cc5953a21f2e0fa58547db7bb6c"; }