Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

R4R: Infrastructure for reproducible builds #4262

Merged
merged 73 commits into from
May 13, 2019
Merged

Conversation

alessio
Copy link
Contributor

@alessio alessio commented May 2, 2019

This change set introduces support for building gaia with gitian
on the following GOOS/GOARCH pairs:

  • darwin/386
  • darwin/amd64
  • linux/386
  • linux/amd64
  • linux/arm
  • linux/arm64
  • windows/386
  • windows/amd64

cmd/gaia/contrib/gitian-descriptors/ contains gitian descriptor files.

cmd/gaia/contrib/gitian-keys/ contains:

  • a keys.txt file that is meant to list core developers and gitian builders PGP keys.
  • README.me to provide instructions on how to import the keys into one's
    personal GPG keyring.

The gosum utility is removed, so is the go.sum hashsum bit from gaiacli/gaiad
version string. It was meant to be a provisional mitigation to the lack of a
reproducible build process.

GOBIN is removed from all Makefiles. When GOBIN is set, go refuses to cross-compiles
binaries for foreign architectures. export GOBIN=$GOPATH/bin is unnecessary anyway
as by default go install places built binaries in $GOPATH/bin. Developers are required
to update their enviornment files and replace $GOBIN with $GOPATH/bin in PATH.

circleci configuration file is amended accordingly.

Instructions at https://github.com/cosmos/cosmos-sdk/wiki/Reproducible-builds

Closes: #4027
Closes: #4280


For Admin Use:

  • Added appropriate labels to PR (ex. wip, ready-for-review, docs)
  • Reviewers Assigned
  • Squashed all commits, uses message "Merge pull request #XYZ: [title]" (coding standards)

@codecov
Copy link

codecov bot commented May 3, 2019

Codecov Report

Merging #4262 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #4262   +/-   ##
======================================
  Coverage    59.1%   59.1%           
======================================
  Files         217     217           
  Lines       14595   14595           
======================================
  Hits         8627    8627           
  Misses       5330    5330           
  Partials      638     638

alessio added 13 commits May 3, 2019 18:18
Instructions follow:

Make sure ruby is installed in your system and lives in /usr/bin/ruby:

 [ `which ruby` = '/usr/bin/ruby' ] && echo yes || echo no

Clone gitian:

 git clone https://github.com/devrandom/gitian-builder

Prepare a docker image for the build:

 cd gitian-build
 bin/make-base-vm --docker
 bin/make-base-vm --docker --arch amd64
 export USE_DOCKER=1

Download go:

  mkdir inputs
  gopkg=go1.12.4.linux-amd64.tar.gz ; \
  curl -L https://dl.google.com/go/$gopkg > inputs/$gopkg

Build:

  bin/gbuild $SDK-PATH/cmd/gaia/contrib/gitian-linux.yml \
    --commit cosmos-sdk=v0.34.3
Update version package accordingly.
@alessio alessio force-pushed the alessio/reproducible-builds branch from 2022bb9 to 9f28890 Compare May 3, 2019 17:19
@alessio alessio changed the title Add Linux descriptor for Gitian builds Start building the infrastructure for reproducible builds May 3, 2019
@alessio alessio marked this pull request as ready for review May 3, 2019 18:53
@alessio alessio requested review from mircea-c and zmanian May 3, 2019 18:53
@alessio
Copy link
Contributor Author

alessio commented May 9, 2019

Bnaries are now built with CGO enabled. I've successfully tested ledger support on Linux amd64.

Copy link
Contributor

@liamsi liamsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I run ./cmd/gaia/contrib/gitian-build.sh on macosx, I get a ./cmd/gaia/contrib/gitian-build.sh: line 196: 1: unbound variable. Update: OK, I see this needs a param passed in, e.g. ./cmd/gaia/contrib/gitian-build.sh linux pwd.

Now:

Start the build
bin/gbuild:252:in ``': No such file or directory - sha256sum (Errno::ENOENT)
        from bin/gbuild:252:in `<main>'

OK, looks like coreutils installation failed. NVM.

@liamsi
Copy link
Contributor

liamsi commented May 10, 2019

The instructions should probably contain a sentence on how to verify that the build I've created matches the official build / I successfully reproduced it.

@alessio
Copy link
Contributor Author

alessio commented May 10, 2019

@liamsi dixit:

The instructions should probably contain a sentence on how to verify that the build I've created matches the official build / I successfully reproduced it.

I will certainly do! And I will include instructions on how to retrieve and import builder keys into your GPG keyring, etc.

@liamsi
Copy link
Contributor

liamsi commented May 10, 2019

Update: coreutils wasn't in PATH ... NVM

I ran git pull to get the latest changes (and deleted the gaia.sigs and the gitian-build-linux dirs, and restarted the build (via ./cmd/gaia/contrib/gitian-build.sh -s [email protected] linux `pwd`) and got:

Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... 
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
bin/gbuild:147:in `block (2 levels) in build_one_configuration': error looking up author date in cosmos-sdk (RuntimeError)
        from bin/gbuild:143:in `each'
        from bin/gbuild:143:in `block in build_one_configuration'
        from bin/gbuild:127:in `open'
        from bin/gbuild:127:in `build_one_configuration'
        from bin/gbuild:309:in `block (2 levels) in <main>'
        from bin/gbuild:304:in `each'
        from bin/gbuild:304:in `block in <main>'
        from bin/gbuild:302:in `each'
        from bin/gbuild:302:in `<main>'

@liamsi
Copy link
Contributor

liamsi commented May 10, 2019

Build reports: https://github.com/Liamsi/gaia.sigs

@alessio alessio requested a review from liamsi May 13, 2019 16:44
Copy link
Member

@jackzampolin jackzampolin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Builds the darwin for me!!

@alessio alessio requested a review from cwgoes May 13, 2019 19:22
@alessio alessio merged commit c0486aa into master May 13, 2019
@alessio alessio deleted the alessio/reproducible-builds branch May 13, 2019 22:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Drop CGO Reproducible Builds
6 participants