Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add linux-aarch64 and osx-arm64 support to rpsbproc #51929

Merged
merged 5 commits into from
Nov 5, 2024

Conversation

dslarm
Copy link
Contributor

@dslarm dslarm commented Nov 5, 2024

Fix compile flags in build.sh to enable linux-aarch64. Add linux-aarch64 and osx-arm64 to additional platforms.

Fix compile flags in build.sh to enable linux-aarch64.
Add linux-aarch64 and osx-arm64 to additional platforms.
Copy link
Contributor

coderabbitai bot commented Nov 5, 2024

📝 Walkthrough
📝 Walkthrough

Walkthrough

The pull request introduces modifications to the build.sh script and the meta.yaml file for the rpsbproc package. In build.sh, conditional logic has been added to improve platform-specific configurations, particularly ensuring that the --with-64 flag is applied only for x86_64 architectures on Linux systems. This change replaces the previous unconditional application of the flag. Additionally, the script's parallel jobs in the make command have been updated from -j1 to -j 4, allowing for increased concurrency during the build process. Comments in the script have been enhanced for clarity regarding configuration flags and their implications, while the overall structure and logic of the script remain unchanged.

In meta.yaml, the build section has been updated by incrementing the number from 0 to 1, and a new additional-platforms entry has been added under the extra section to specify support for the linux-aarch64 platform. A comment indicates that the osx-arm64 platform is encountering issues during builds, but it is not included in the list. The rest of the file, including package metadata, source URLs, build requirements, and descriptions, remains unchanged.

Possibly related PRs

Suggested labels

please review & merge, aarch64, osx-arm64

Suggested reviewers

  • martin-g

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between a44c11d and bc24b86.

📒 Files selected for processing (1)
  • recipes/rpsbproc/meta.yaml (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • recipes/rpsbproc/meta.yaml

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (2)
recipes/rpsbproc/meta.yaml (1)

Line range hint 39-67: Consider documenting architecture support in package description.

The description section provides detailed information about the package but doesn't mention platform support. Consider adding a note about the supported architectures.

 about:
   description: |
     The rpsbproc command line utility is an addition to the standalone version of
     Reverse Position-Specific BLAST (RPS-BLAST), also known as CD-Search (Conserved
     Domain Search).
+
+    Supported architectures: x86_64, aarch64 (ARM64) on Linux, and x86_64, arm64 on macOS.
recipes/rpsbproc/build.sh (1)

128-132: LGTM! Fix indentation for consistency.

The architecture-specific logic for the --with-64 flag is correct and essential for supporting ARM architectures. The flag should indeed only be applied for x86_64 platforms.

Fix the indentation to match the rest of the file:

        if [[ "$(arch)" = "x86_64" ]]; then
-	    CONFIGURE_FLAGS="$CONFIGURE_FLAGS --with-64"
+            CONFIGURE_FLAGS="$CONFIGURE_FLAGS --with-64"
        fi
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between b623d5b and f232f70.

📒 Files selected for processing (2)
  • recipes/rpsbproc/build.sh (1 hunks)
  • recipes/rpsbproc/meta.yaml (1 hunks)
🔇 Additional comments (4)
recipes/rpsbproc/meta.yaml (1)

84-86: LGTM! Verify patch compatibility across architectures.

The addition of linux-aarch64 and osx-arm64 platforms aligns with the PR objectives. However, let's ensure the patches are compatible with these architectures.

✅ Verification successful

Patches are architecture-aware and compatible with the new platforms

The analysis of the patches shows that update_configsub.patch already includes proper support for the new architectures:

  • Contains explicit handling for aarch64 and arm64 architectures
  • Includes Darwin (macOS) platform support
  • Updates the config.sub to recognize osx-arm64
  • Other patches don't contain any architecture-specific code

The patches are compatible with the newly added platforms and will work correctly for linux-aarch64 and osx-arm64.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check if patches contain architecture-specific code or paths
# that might affect the newly added platforms

# Test: Look for architecture-specific patterns in patches
echo "Checking patches for architecture-specific code..."
for patch in recipes/rpsbproc/*.patch; do
  echo "=== Analyzing $patch ==="
  # Look for architecture-specific patterns
  rg -i "x86|amd64|arm|aarch|darwin" "$patch" || true
done

Length of output: 3081

recipes/rpsbproc/build.sh (3)

Line range hint 8-16: Excellent debug information for platform verification.

The debug output will be valuable for verifying the build environment on new architectures. This will help track any platform-specific issues that arise.


Line range hint 71-106: Verify library availability for ARM architectures.

The script configures several library dependencies. Please ensure:

  1. All required libraries are available in bioconda for linux-aarch64 and osx-arm64
  2. Static linking (--without-dll --with-static-exe) works correctly on ARM platforms

Let's check the library availability:


Line range hint 17-24: Verify compiler optimizations for ARM architectures.

While the platform detection for the --with-64 flag is correct, please verify:

  1. If any ARM-specific compiler optimizations should be enabled
  2. That OpenMP support works correctly on ARM architectures (both linux-aarch64 and osx-arm64)

Let's check the configure script for available ARM-specific options:

Also applies to: 130-131

✅ Verification successful

Based on the search results, I can now provide a final response:

OpenMP support and ARM optimizations are properly handled in the build script.

The build script correctly:

  1. Detects the architecture and configures OpenMP appropriately (--with-openmp for x86 and --without-openmp for macOS)
  2. No ARM-specific compiler optimizations are needed as the NCBI toolkit handles architecture-specific optimizations internally
  3. The configure script uses standard autotools which ensures proper architecture support
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for ARM-related configuration options
cd "$SRC_DIR/ncbi_cxx/c++"
./configure.orig --help | grep -i "arm\|aarch"

# Check if OpenMP detection works on ARM
./configure.orig --help | grep -i "openmp"

Length of output: 387


Script:

#!/bin/bash
# Let's first check if we can find the configure script and its location
fd configure

# Check for any ARM or architecture specific compiler flags in build scripts
rg -i "arm|aarch" --type sh

# Look for OpenMP related configurations
rg -i "openmp" --type sh

Length of output: 19515

@dslarm
Copy link
Contributor Author

dslarm commented Nov 5, 2024

@bioconda/core - the CI is timing out whilst building this package on linux-aarch64. It does take a long time locally, given it is building some ncbi c++ package under the hood. Is there a way to bump the time up for this one?

There is also an error in osx-arm64 builds - yet this also builds locally - it's not clear but it looks to be complaining about the version of developer tools on the osx-arm64 CI.

@BiocondaBot
Copy link
Collaborator

Reposting for @dslarm to enable pings (courtesy of the BiocondaBot):

@bioconda/core - the CI is timing out whilst building this package on linux-aarch64. It does take a long time locally, given it is building some ncbi c++ package under the hood. Is there a way to bump the time up for this one?

There is also an error in osx-arm64 builds - yet this also builds locally - it's not clear but it looks to be complaining about the version of developer tools on the osx-arm64 CI.

@dslarm
Copy link
Contributor Author

dslarm commented Nov 5, 2024

arm64-osx error is:

10:53:06 BIOCONDA INFO (ERR) 2024-11-05 10:53:06.961 xcodebuild[18514:40154] [MT] DVTSDK: Skipped SDK /Applications/Xcode-15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk; its version (11.0) is below required minimum (13.0) for the macosx platform.
10:53:07 BIOCONDA INFO (ERR) 2024-11-05 10:53:07.997 xcodebuild[18514:40154] Writing error result bundle to /var/folders/dp/l1c55xzj5fv1frff4m7llvvr0000gn/T/ResultBundle_2024-05-11_10-53-0007.xcresult
10:53:09 BIOCONDA INFO (ERR) xcodebuild: error: SDK "/Applications/Xcode-15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk" cannot be located.
10:53:10 BIOCONDA INFO (ERR) make: error: sh -c '/Applications/Xcode-15.4.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode-15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -find make 2> /dev/null' failed with exit code 16384: (null) (errno=No such file or directory)
10:53:10 BIOCONDA INFO (ERR) xcode-select: Failed to locate 'make', requesting installation of command line developer tools.
10:53:10 BIOCONDA INFO (ERR) 
10:53:10 BIOCONDA INFO (ERR) ------------------------------------------------------
10:53:10 BIOCONDA INFO (ERR) Current dir:  /opt/mambaforge/envs/bioconda/conda-bld/rpsbproc_1730803880556/work/ncbi_cxx/c++/Release/build/corelib
10:53:10 BIOCONDA INFO (ERR) 
10:53:10 BIOCONDA INFO (ERR) [create_flat_makefile.sh] FAILED (status 72):
10:53:10 BIOCONDA INFO (ERR)    make

@martin-g
Copy link
Contributor

martin-g commented Nov 5, 2024

Is there a way to bump the time up for this one?

No. 60 mins is the maximum in the Free plan - https://support.circleci.com/hc/en-us/articles/4410707277083-Context-deadline-exceeded-after-1-hour-Build-timed-out-Free-tier-only
I am not sure whether the parallelism setting would help. You could try by editing .circleci/config.yml in this PR.

@martin-g
Copy link
Contributor

martin-g commented Nov 5, 2024

Another thing:
currently the build uses make -j1 - https://github.com/bioconda/bioconda-recipes/pull/51929/files#diff-93630cbcb3d69000383f789843095591654c79ce6dba03a84d402bb27cbe94e0R159
You could try with make -j${CPU_COUNT} ...

@martin-g
Copy link
Contributor

martin-g commented Nov 5, 2024

arm64-osx error is:

The following is also interesting:

10 �[32mBIOCONDA INFO�[0m (ERR) make: error: sh -c '/Applications/Xcode-15.4.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode-15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -find make 2> /dev/null' failed with exit code 16384: (null) (errno=No such file or directory)�[0m
10:53:10 �[32mBIOCONDA INFO�[0m (ERR) xcode-select: Failed to locate 'make', requesting installation of command line developer tools

make is listed as a build dependency but it fails to find it ...

@ebete
Copy link
Contributor

ebete commented Nov 5, 2024

Another thing: currently the build uses make -j1 - https://github.com/bioconda/bioconda-recipes/pull/51929/files#diff-93630cbcb3d69000383f789843095591654c79ce6dba03a84d402bb27cbe94e0R159 You could try with make -j${CPU_COUNT} ...

Be careful with too many jobs though. Last time I tried it while building BLAST, it ate up all the memory causing the CI runner to grind to a halt and getting axed by the host. Maybe rpsbproc is a bit lighter to build, but the NCBI C++ toolchain is still quite resource-hungry. -j4 would probably be fine.

@dslarm
Copy link
Contributor Author

dslarm commented Nov 5, 2024 via email

the correct make, and tripping over the system one(s).
@mencian mencian merged commit 3587571 into bioconda:master Nov 5, 2024
6 checks passed
@mencian mencian added the aarch64 Related to adding linux-aarch64 support label Nov 5, 2024
@dslarm dslarm deleted the rpsbproc branch November 11, 2024 14:19
@coderabbitai coderabbitai bot mentioned this pull request Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aarch64 Related to adding linux-aarch64 support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants