Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

M1 support, upgrade to newer RocksDB and AWS Java SDK #8533

Merged

Conversation

DevinTDHa
Copy link
Member

@DevinTDHa DevinTDHa commented May 20, 2022

Description

This PR enables Spark NLP to be built with M1 support. Most of the required packages have already been published and have been integrated into the build.

This PR fixes the remaining compatibility issues:

  • Add another condition in build.sbt to use a newer rocksdbjni version if m1 was selected for the build.
  • Add compatible libraries for libsentencepiece_jni and _sentencepiece_tokenizer with respective conditions for loading

TODO:

  • Update docs with user guide on how to use m1 jar.

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Code improvements with no or little impact
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

@DevinTDHa DevinTDHa added enhancement dependencies Pull requests that update a dependency file labels May 20, 2022
@DevinTDHa DevinTDHa requested a review from maziyarpanahi May 20, 2022 09:53
@maziyarpanahi maziyarpanahi added the DON'T MERGE Do not merge this PR label May 20, 2022
@maziyarpanahi
Copy link
Member

@DevinTDHa Can we not call those native files aarch64? (I am not sure if these files are compatible with Linux aarch64 which Databricks/AWS are offering now, so when we start supporting those it maybe confusing - but if they are compatible with all aarch64 and not just M1 we can leave it like this, otherwise let's replace aarch64 with m1)

@maziyarpanahi maziyarpanahi added the new-feature Introducing a new feature label May 20, 2022
@DevinTDHa
Copy link
Member Author

@DevinTDHa Can we not call those native files aarch64? (I am not sure if these files are compatible with Linux aarch64 which Databricks/AWS are offering now, so when we start supporting those it maybe confusing - but if they are compatible with all aarch64 and not just M1 we can leave it like this, otherwise let's replace aarch64 with m1)

Yes, let me change it actually to M1 so it is clearer. Regarding Linux and Windows compatibility there shouldn't be an issue, as this is for the mac dylibs only (i'll add another condition for SentencePieceJNI.java though)

@maziyarpanahi maziyarpanahi changed the base branch from release/400-release-candidate to master May 20, 2022 12:15
@maziyarpanahi maziyarpanahi changed the title M1 support For Spark NLP M1 support, upgrade to newer RocksDB and AWS Java SDK May 20, 2022
@maziyarpanahi maziyarpanahi changed the base branch from master to release/400-release-candidate May 20, 2022 12:55
@coveralls
Copy link

Coverage Status

Coverage increased (+0.02%) to 74.376% when pulling 4b0dede on DevinTDHa:m1_support into 5ea6c91 on JohnSnowLabs:release/400-release-candidate.

@maziyarpanahi maziyarpanahi merged commit b529aa5 into JohnSnowLabs:release/400-release-candidate May 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file DON'T MERGE Do not merge this PR enhancement new-feature Introducing a new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants