raspberrypi: update firmware, linux, tools to latest (~2020-12-01)

[drewrisinger]

Motivation for this change

Various packages in the Raspberry Pi ecosystem were out of date.

Some commits might need squashed, wanted to leave them separate for now to clarify why/what I did.

Potentially controversial: noticed duplicate raspberrypi-tools && libraspberrypi. These have the same source & practically the same derivation. I chose to keep libraspberrypi because it had a slightly cleaner derivation, but I'll leave it up to the maintainers to decide.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[drewrisinger]

This technically builds on regular Linux, but it's really targeted at ARM devices/the Pi, so I think this is reasonable.
https://github.com/raspberrypi/usbboot/blob/d3760e119385a179765f43a50a8e051a44127c25/Readme.md#building

[flokli]

Isn't this a tool I could run on my workstation to "fastload" a raspberry Pi? If so, building it on x86_64-linux too would be appreciated ;-)

[prusnak]

untested, changes look fine

[flokli]

Isn't this a tool I could run on my workstation to "fastload" a raspberry Pi? If so, building it on x86_64-linux too would be appreciated ;-)

[flokli]

I rebased this on latest master, incorporated the fixes from https://github.com/NixOS/nixpkgs/pull/107689/files and fixed the duplicate attribute in the kernel expression.

[ghost]

My i2s sound card does not work with the 5.4 rpi4 or rpi3 kernel. The device tree overlays fail to build:

Error at 'compatible': FDT_ERR_NOTFOUND
builder for '/nix/store/bk30gbx07bv0303ji08dzhblwrsc8jd8-device-tree-overlays.drv' failed with exit code 1

This can be reproduced with the following config lines:

  boot.kernelPackages = pkgs.linuxPackages_rpi4;
  hardware.deviceTree.enable = true;
  hardware.deviceTree.overlays = [ "${pkgs.linux_rpi4}/dtbs/overlays/hifiberry-amp.dtbo" ];

[flokli]

@petabyteboy yes, the raspberrypi device tree files unfortunately seem to use some custom format/extension which doesn't seem to be understood by the mainline dtc compiler, and its error messages are terrible too.

raspberrypi/linux#3198 (comment) claims it should work, but apparently it doesn't (or broke again, who knows).

We might want to open a new issue if it clearly doesn't work with the upstream dtc compiler.

The raspberrypi-specific dtmerge tool seems to be able to work with these overlays - I stumbled over kwohlfahrt@7ba456e, which allows using that for hardware.deviceTree instead of dtc - we might want to upstream to this nixpkgs.

I did some ground-work in fixing raspberrypi-tools to build on x86_64-linux (so I can still cross-compile raspberrypi closures from x86_64 (PR merged in #107689, moved to libraspberrypi in this PR), but moved over to a Pine64 for the time being.

Happy to review a PR that allows compiling devicetress with dtmerge :-)

[ghost]

I could make my Raspberry Pi 4 work with the HifiBerry Amp2 (which is using the hifiberry-dacplus overlay) by using dtmerge:

{ config, pkgs, lib, ... }:

let
  dtbsWithSymbols = pkgs.stdenv.mkDerivation {
    name = "dtbs-with-symbols";
    inherit (config.boot.kernelPackages.kernel) src nativeBuildInputs depsBuildBuild;
    patches = map (patch: patch.patch) config.boot.kernelPackages.kernel.kernelPatches;
    buildPhase = ''
      patchShebangs scripts/*
      substituteInPlace scripts/Makefile.lib \
        --replace 'DTC_FLAGS += $(DTC_FLAGS_$(basetarget))' 'DTC_FLAGS += $(DTC_FLAGS_$(basetarget)) -@'
      make ${pkgs.stdenv.hostPlatform.platform.kernelBaseConfig} ARCH="${pkgs.stdenv.hostPlatform.platform.kernelArch}"
      make dtbs ARCH="${pkgs.stdenv.hostPlatform.platform.kernelArch}"
    '';
    installPhase = ''
      make dtbs_install INSTALL_DTBS_PATH=$out/dtbs  ARCH="${pkgs.stdenv.hostPlatform.platform.kernelArch}"
    '';
  };

in {
  boot.kernelPackages = pkgs.linuxPackages_rpi4;
  
  hardware.deviceTree.enable = true;
  hardware.deviceTree.filter = "bcm2711-rpi-4-b.dtb";
  hardware.deviceTree.package = lib.mkForce (
    pkgs.runCommand "device-tree-overlays" {
      buildInputs = with pkgs; [ findutils libraspberrypi ];
    } ''
      cd ${dtbsWithSymbols}/dtbs
      for dtb in $(find . -type f -name "${config.hardware.deviceTree.filter}")
      do
        install -D $dtb $out/$dtb

        ${lib.concatMapStrings (param: ''
          dtmerge -d $out/$dtb{,-merged} - ${param}
          mv $out/$dtb{-merged,}
        '') [ "spi=on" ]}

        ${lib.concatMapStrings (overlay: ''
          dtmerge -d $out/$dtb{,-merged} ${overlay}
          mv $out/$dtb{-merged,}
        '') [ "${config.boot.kernelPackages.kernel}/dtbs/overlays/hifiberry-dacplus.dtbo" ]}
      done
    ''
  );
  
  [...]
}

However my Raspberry Pi 3 is out of luck. Changing boot.kernelPackages to pkgs.linuxPackages_rpi3 and hardware.deviceTree.filter to bcm2837-rpi-3-b-plus.dtb makes it fail even using dtmerge:

DTOVERLAY[debug]: using platform 'bcm2835'
DTOVERLAY[debug]: overlay map loaded
root module: vc4
DTOVERLAY[debug]: /__symbols__:i2c=i2c0
DTOVERLAY[debug]: /__symbols__:i2c_arm=i2c0
DTOVERLAY[debug]: /__symbols__:i2c_vc=i2c1
DTOVERLAY[error]: can't find symbol 'vdd_3v3_reg'

Or when applying the hifiberry-amp overlay:

DTOVERLAY[debug]: using platform 'bcm2835'
DTOVERLAY[debug]: overlay map loaded
DTOVERLAY[debug]: /__symbols__:i2c=i2c0
DTOVERLAY[debug]: /__symbols__:i2c_arm=i2c0
DTOVERLAY[debug]: /__symbols__:i2c_vc=i2c1
DTOVERLAY[error]: can't find symbol 'sound'

These are the same errors I get when trying to apply these overlays to a recent upstream linux kernel by the way.

[ghost]

I should first say a setup that was quite common with NixOS aarch64 users was actually never supported by upstream: The RPi3 is supposed to be used in 32-bit arm and work exactly like an RPi2, while we use it in aarch64 mode, where it has its own defconfig and so on.

From 4.19 to 5.4 this change broke raspberry pi specific device tree overlays for the RPi3 aarch64 config:

raspberrypi/linux@9a6d1cb#diff-0c179e16b6e7236b33310a13729679c1c136d2b7a216339ff652d5c4db3a1e21L33

Both the RPi4 aarch64 config and the RPi3 and lower 32-bit arm configs import bcm270x-rpi.dtsi, which specifies some things for raspberry-pi-specific overlays, like vdd_3v3_reg, that are not upstreamed:

result/arch/arm/boot/dts/bcm270x-rpi.dtsi: /* Downstream modifications to bcm2835-rpi.dtsi */
result/arch/arm/boot/dts/bcm270x-rpi.dtsi:      vdd_3v3_reg: fixedregulator_3v3 {
result/arch/arm/boot/dts/bcm2708-rpi.dtsi:#include "bcm270x-rpi.dtsi"
result/arch/arm/boot/dts/bcm2711-rpi.dtsi:#include "bcm270x-rpi.dtsi"

But bcm2835-rpi.dtsi does not include bcm270x-rpi.dtsi, so the RPi3 aarch64 config can not use raspberry pi specific overlays.

I will try to send a patch to the raspberrypi/linux to add back support for this use case.

[ghost]

Nevermind, this was already the case with the last 4.19 rpi3 aarch64 kernel, so it's not a problem that was introduced with this PR.

[flokli]

@petabyteboy I linked to here from https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi_4, cause others seem to be running into this issue. If you went into contact with upstream, can you please link from here?