Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Introduce optional dependencies to streamline platform-specific installation #63

Open
wants to merge 2 commits into
base: staging
Choose a base branch
from

Conversation

CMCDragonkai
Copy link
Member

@CMCDragonkai CMCDragonkai commented Jun 25, 2023

Description

Our js-quic now published optional packages

  "optionalDependencies": {
    "@matrixai/quic-darwin-arm64": "0.0.7-alpha.0",
    "@matrixai/quic-darwin-x64": "0.0.7-alpha.0",
    "@matrixai/quic-linux-x64": "0.0.7-alpha.0",
    "@matrixai/quic-win32-x64": "0.0.7-alpha.0"
  },

I want to apply this to js-db too, so we can do streamline the installation and also be able to self-publish at least just the Linux one.

  "optionalDependencies": {
    "@matrixai/db-darwin-arm64": "5.1.0",
    "@matrixai/db-darwin-x64": "5.1.0",
    "@matrixai/db-linux-x64": "5.1.0",
    "@matrixai/db-win32-x64": "5.1.0"
  },

This involves replacing how we do our prebuild.js in particular and bringing in the scripts/prebuild.js, scripts/prepublishOnly.js and scripts/version.js.

We may also need to fix up our CI/CD scripts too.

Without the GitLab CI/CD we can only do the linux build and windows builds. So we can delay having the macos ones.

Note that we rely on node-gyp-build package for both detection and auto-building the local dependency when installed. We don't have node-gyp-build in js-quic, thus there's no automatic compilation upon installation. Instead if none of the native dependencies can be found, it's necessary for the user to compile their own (which is a bit of a pain)... since compilation requires running a certain script, specifically npm run prebuild. However that's not really possible while we are installing js-quic as a dependency. We could do something similar with our own install script, but such a script/command has to exist in the packaged distribution and we actually ignore all /scripts in our .npmignore.

At the end of the day, I'm not even sure if it makes sense to enable automatic compilation. Really if it is not available, they have to source it from somewhere and place it in the right location. Specifically the custom loader will look for it in just the local prebuild/ directory. So figure out a compilation routine and then distribute it during npm install. The problem with relying on npm install to compile the native binary is that there's no guarantee that all the tools are available to do this for the end user. So therefore one should think of npm install as the final distribution here, for power users, they should be using the gitlab repository, possibly directly refer to it as a dependency, or submit a build for this opensource repo.

Fixes #43
Fixes #44

Tasks

  • 1. Bring in scripts from js-quic
  • 2. Run a compilation on Linux
  • 3. Replace the node-gyp-build with just your own loader script, copied from js-quic.
  • 4. Remove the install command of node-gyp-build. It would not be necessary... as it simply detects if the binary already exists, and if not ends up calling a build to do a local compilation.

Final checklist

  • Domain specific tests
  • Full tests
  • Updated inline-comment documentation
  • Lint fixed
  • Squash and rebased
  • Sanity check the final build

@ghost
Copy link

ghost commented Jun 25, 2023

👇 Click on the image for a new way to code review

Review these changes using an interactive CodeSee Map

Legend

CodeSee Map legend

@CMCDragonkai
Copy link
Member Author

Ok turns out it's a little bit more complicated than just copying over the prebuild.js from js-quic. The reason is that the binding.gyp indicates that we end up creating a fat executable on MacOS, that is both x86_64 and arm64. In js-quic, we separated these 2 out and now build 1 specific binary.

I wonder if it is possible to get binding.gyp to specify to outputs instead of just 1 output?

@CMCDragonkai CMCDragonkai self-assigned this Jun 25, 2023
@CMCDragonkai
Copy link
Member Author

I might put this on the backburner for now as it isn't entirely necessary and just release the latest version.

@CMCDragonkai
Copy link
Member Author

New version to be released with the new macos build too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Clean up superfluous compilation flags Integrate ccache for caching native compilation
1 participant