Re-enabling multiarch support

[martadinata666]

1. Reenable multiarch build
2. As we know github actions pretty limited, either killed OOM or 6hrs timeout

In this PR we do old way, build everything separately, then gather and push manifest. Some issue is the currecnt Dockerfile can't be used for this, as we will build js out side docker, and only build dim in container. I thought create Dockerfile.ci for this use case. Technically still like master Dockerfile but with UI build disabled.

Build proof :
https://github.com/martadinata666/dim/actions/runs/2565143979

Thoughts?

[vgarleanu]

Couldn't we use matrix to do multi arch builds?

Also we don't push to dockerhub anymore, only ghcr.

Also I don't fully understand why building the UI is disabled. I'm not a fan of server binaries that don't embed the UI, which might confuse downstream users when they run dim.

[vgarleanu]

Looking at the pipeline that you linked, the build times for different architectures is way too excessive. Why is this? I see that qemu is used in the workflow, although I'm not sure why. Are we emulating a different arch here?

Couldn't we make the multi arch build a lot simpler? Rust has very good support for cross compilation. We could just use something like cross.

[martadinata666]

Im not really sure about the dynamic linking libraries, Will it work? Like build on libva amd64 and cross arm64 with libva arm64? Afaik it only work if it fully static like musl target.

[vgarleanu]

As far as I know libva is either not supported on non-x86 hardware or there aren't any devices that support vaapi on non-x86 hardware, as such we should disable vaapi for other architectures.

That said I think we should still target musl for non-x86 architectures. What I had in mind is that we could use cross to compile the binaries for all other architectures targeting musl, then simply copy those binaries into the docker image.

I imagine most of the time in the arm job is spent compiling dim while also emulating arm on x86. This can be avoided as described above.

Since dim pretty much statically links everything, I doubt we will experience any hiccups here.

[martadinata666]

So i played a bit, looking how cross played out in the end stuck on openssl sys issue:

• Build Failed: openssl-sys cross-rs/cross#400
• Remove OpenSSL cross-rs/cross#229

There is downgrade method likely working?

• Remove OpenSSL cross-rs/cross#229 (comment)

workflow run:

• https://github.com/martadinata666/dim/runs/7066219905?check_suite_focus=true

and strategy on build image docker, cant really pass the matrix with special character, or maybe im missing something.
Another issue is if the matrix work, how it differentiate the tags, im not really sure.

  - name: Build and push based on matrix
        uses: docker/build-push-action@v2
        with:
          context: .
          push: ${{ github.event_name != 'pull_request' }} #push except PR.
          platforms: ${{ matrix.platforms }}
          tags: ${{ steps.metaid.outputs.tags }}
          labels: ${{ steps.metaid.outputs.labels }}

Update : matrix result oom killed, using matrix as it count as one job, i assume matrix still using single runner.

Im gonna revert to last working with nodejs build separately.

[vgarleanu]

Openssl with musl will be tricky. I wonder which dependency is using it and if we can replace it with rustls.

p.s. you can amend your previous commit and force push instead of creating a lot of small commits for small fixes. Something like git commit --amend && git push -f should work well.

[martadinata666]

openssl-sys break on gnu/musl target. such tricky issue. Im testing this and that 🤣 i hope some magic happens.

update:
https://github.com/martadinata666/dim/actions/runs/2569571677

docker build via artifact, maybe need to create release or so, but it your preferences.
https://github.com/martadinata666/dim/runs/7076707658

[vgarleanu]

Looks pretty good so far. The pipeline run you posted seems to build binaries but not images. Although this should be easy to fix.

I looked at the aarch64 binaries and it seems like it still loads libc.so. Are we not using musl here?

p.s. I've got a raspi laying around, Ill try to test the binaries on it.

[vgarleanu]

Also I'm not a fan of how the docker-ci workflow is a separate workflow. Could we combine it with the rust-target-multi workflow?

[vgarleanu]

FYI I managed to get rid of openssl-sys on one of my branches. Turns out reqwest was using openssl instead of rustls by default.

[martadinata666]

Yes, we can combine docker build with rust-target, currenctly just separated for faster testing.
The problem with alpine again about the SSL, im not really sure how to handle it, i think just change the target will do. 🤷🏼
Well we can just try it, to see where it fail.

Ref:
https://eighty-twenty.org/2019/10/15/cross-compiling-rust 🤞🏼

[vgarleanu]

Just tried to compile targetting musl and it fails to link because of anitomy-sys using some glibc functions that musl doesnt implement (I think). Looks like we have to stay with glibc until we can fix anitomy

[vgarleanu]

@martadinata666 We just need to replace

dim/dim/Cargo.toml

Line 37 in 3810fc1

reqwest = { version = "0.11.0", features = ["json", "default-tls"], default-features = false }

with

reqwest = { version = "0.11.0", features = ["json", "rustls-tls"], default-features = false }

Doing this should get rid of any dependency on openssl.

[martadinata666]

@martadinata666 We just need to replace

dim/dim/Cargo.toml

Line 37 in 3810fc1

reqwest = { version = "0.11.0", features = ["json", "default-tls"], default-features = false }

with

reqwest = { version = "0.11.0", features = ["json", "rustls-tls"], default-features = false }

Doing this should get rid of any dependency on openssl.

Not really sure about this, do we need it? Is this for supporting ssl protocol out of the box without reverse proxy?
Will the rustls-tls do the same? We already can build with openssl, so i think it ok for now.

Otherhand, i agree just stay at glibc, this alpine musl, we could but via qemu, that defeating the purpose fast binary build. I will revert dim binary build and combine with docker build tomorrow. 👍🏼

[vgarleanu]

I think we should still depend on rustls as I trust rust code more than C++ ;p. This will just affect the reqwest dependency which we use to fetch assets and make API calls. The api server already uses rustls (and supports tls without a reverse proxy).

[martadinata666]

I see, moving to rusttls pretty much not an issue then. 👍🏼

Last build success: ✅
https://github.com/martadinata666/dim/actions/runs/2573606377

[vgarleanu]

This looks good. Will give this another quick review Monday and then I'm happy to merge!

[vgarleanu]

Looks like building an aarch64 image is failing. The other CI jobs (docs / rust build) can be ignored

[martadinata666]

Let me sync up with new commits and see the result, https://github.com/martadinata666/dim/actions/runs/3123000524

[martadinata666]

Is it any specific reason that the tool chain locked in?

https://raw.githubusercontent.com/Dusk-Labs/dim/master/rust-toolchain

[vgarleanu]

Yes, the toolchain is currently pinned because rustc releases newer than that version (both stable and nightly) fail to compile dim in release mode due to an ICE caused by some sort of MIR bug.

[vgarleanu]

FYI if you rebase onto master, DATABASE_URL doesnt need to be set anymore, it will be automatically set by our build-scripts.

[martadinata666]

Rebase to master again, and last build success, https://github.com/martadinata666/dim/actions/runs/3124534900