Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build on MacOS M1 (Apple Silicon) #7104

Merged
merged 1 commit into from
Oct 19, 2021

Conversation

h7kanna
Copy link
Contributor

@h7kanna h7kanna commented Oct 18, 2021

What

Experimental Build on macOS M1 (Apple silicon)

How

Upgrade JDK version to 17 and parameterize through an environment variable
Upgrade Node version to 16 and parameterize through an environment variable
Parameterize the docker build platform

Example

export DOCKER_BUILD_PLATFORM=linux/arm64
export DOCKER_BUILD_ARCH=arm64
export JDK_VERSION=17
export NODE_VERSION=16.11.1
SUB_BUILD=PLATFORM ./gradlew build -x test

Recommended reading order

  1. build.gradle
  2. airbyte-workers/Dockerfile

Pre-merge Checklist

This is an experiment mainly to speed my dev cycle by solving #2017.
Currently, I have to build on a separate Linux machine which is a frustratingly very slow dev loop.
Changes include fixes for the below issues.

TODO

I am able to successfully run Airbyte on MacOS M1, created sample Connection, Destination, and Source.

Screen Shot 2021-10-18 at 12 35 53 AM

@CLAassistant
Copy link

CLAassistant commented Oct 18, 2021

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added area/connectors Connector related issues area/core area/documentation Improvements or additions to documentation area/worker Related to worker labels Oct 18, 2021
@h7kanna h7kanna force-pushed the jdk17-macos-m1 branch 2 times, most recently from 3804b2b to e3fb9a8 Compare October 18, 2021 06:15
@marcosmarxm
Copy link
Member

Thanks @h7kanna I asked @jrhizor and @tuliren to review your contribution! They already discuss about developing in M1 or upgrading to Java 17.

@tuliren
Copy link
Contributor

tuliren commented Oct 19, 2021

@h7kanna, huge thanks to you for working on this issue.

I tested it on a 16 GB M1 Mac Pro. The compilation works perfectly. I am able to launch the server and run a sync as well.

Some problems I have seen are:

  1. Unit tests fail because the current Mockito does not support Java 17.
  2. There are temporal problems from time to time (see an example below). Restarting the server can fix them. I suppose this will be fixed once temporal can pull the correct image?

airbyte-temporal | {"level":"error","ts":"2021-10-19T16:31:04.969Z","msg":"Operation failed with internal error.","service":"matching","error":"UpdateTaskQueue failed. Failed to start transaction. Error: context deadline exceeded","metric-scope":38,"logging-call-at":"persistenceMetricClients.go:804","stacktrace":"go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:804\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:787\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/taskReader.go:272\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:160"}`

@h7kanna, by the way, do you plan to work on the TODOs? It's ok if you don't. We just need to know what projects we need to prioritize for M1 support.

I also confirmed that this PR does not interfere with Intel mac. Build, unit test, and server can all run without any problem. So I think it is worth merging this PR as is (even if the unit test issue cannot be resolved soon), since it can unblock developers working on M1.

@h7kanna
Copy link
Contributor Author

h7kanna commented Oct 19, 2021

@tuliren My intention of this is to unblock myself to develop on M1 and not break anything for existing build.

for the TODO important one is temporal arm64 image. Better way is to engage with the Temporal team if they can include linux/arm64 images be pushed to Docker hub.

Do merge it.. Thanks.

@tuliren
Copy link
Contributor

tuliren commented Oct 19, 2021

A small change has been made in the master. Our CI runner has an old version of docker-compose. So we added a M1 specific docker compose build file: docker-compose.build-m1.yaml, which will be used when the arch is arm64.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation area/worker Related to worker community
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants