Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add libcufile recipe #21908

Merged
merged 72 commits into from
Apr 12, 2023
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
b5e65d6
Add libcufile recipe
mroeschke Jan 31, 2023
c9f1fc1
Add Licence metadata
mroeschke Jan 31, 2023
372f26e
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
488dbda
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
4279763
Add summary
mroeschke Jan 31, 2023
1a4969e
Change description to summary
mroeschke Jan 31, 2023
0c20a35
Add file tests
mroeschke Jan 31, 2023
fbac483
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
e41a0ac
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
5d18921
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
cdbcdba
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
9e6d832
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
ee146b6
Add build.sh instead of build.bat
mroeschke Feb 1, 2023
9ad326f
Update recipes/libcufile/Makefile.patch
mroeschke Feb 3, 2023
64b3393
some formatting
mroeschke Feb 3, 2023
a90df31
Try patchfile fix
mroeschke Feb 4, 2023
702a290
Add directory
mroeschke Feb 4, 2023
dbbd113
Create and move tools and samples to gds dir
mroeschke Feb 7, 2023
8e21b79
Merge branch 'main' into libcufile
mroeschke Feb 7, 2023
1970a53
typo -p
mroeschke Feb 7, 2023
6158a96
just -d
mroeschke Feb 7, 2023
7497aed
Move man
mroeschke Feb 7, 2023
c8e9e42
include new gds dir
mroeschke Feb 7, 2023
a17812a
Update recipes/libcufile/meta.yaml
mroeschke Feb 8, 2023
5317c44
Move to difference section
mroeschke Feb 8, 2023
f264627
Move to outputs build
mroeschke Feb 8, 2023
f93c0b2
New build section
mroeschke Feb 8, 2023
11ee0dc
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
79c750a
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
c8de857
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
54a340a
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
178099f
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
1ba2afe
Update recipes/libcufile/meta.yaml
mroeschke Feb 11, 2023
76cd37c
Merge branch 'main' into libcufile
mroeschke Feb 11, 2023
61ab85d
Update recipes/libcufile/meta.yaml
mroeschke Feb 13, 2023
93593ce
Update recipes/libcufile/meta.yaml
mroeschke Feb 13, 2023
498ac3a
Update recipes/libcufile/meta.yaml
mroeschke Feb 13, 2023
89875b4
wildcards missing?
mroeschke Feb 14, 2023
86b9763
Revert "wildcards missing?"
mroeschke Feb 16, 2023
7939393
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
9cd3345
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
2556d27
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
9d4601f
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
0452766
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
28723e1
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
f03b6c1
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
be9e78c
Merge branch 'main' into libcufile
mroeschke Mar 22, 2023
900a8aa
Merge branch 'main' into libcufile
jakirkham Mar 27, 2023
1ed22aa
Merge branch 'main' into libcufile
mroeschke Apr 5, 2023
25ee3e5
Make changes to meta and build files
mroeschke Apr 5, 2023
c671519
Keep top level about section
mroeschke Apr 5, 2023
afa2ad6
Merge branch 'main' into libcufile
mroeschke Apr 7, 2023
ba1e05d
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
ca53dd3
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
21f096d
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
694ac62
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
32b88d0
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
3cda9c8
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
23d673c
Refactor -dev tests
mroeschke Apr 7, 2023
ebfbf07
pin_compatible?
mroeschke Apr 7, 2023
f32c306
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
7456ba7
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
296b975
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
08a0182
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
4b402b1
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
9e35a54
Fix libcufile tests and packaging.
bdice Apr 12, 2023
c1f3c4b
Attempt to fix RDMA issue.
bdice Apr 12, 2023
5b28f1b
Make tests pass.
bdice Apr 12, 2023
08a70e1
Fix RDMA cdt dependency.
bdice Apr 12, 2023
e1d5f02
Don't ship etc/cufile.json.
bdice Apr 12, 2023
16cfcdd
Rerun CI.
bdice Apr 12, 2023
21053dd
Merge pull request #1 from bdice/fix-libcufile-tests
mroeschke Apr 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions recipes/libcufile/Makefile.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
--- samples/Makefile 2022-06-24 00:23:23.000000000 +0000
+++ samples/Makefile.new 2022-06-27 15:04:32.235125647 +0000
@@ -35,8 +35,8 @@

# Common includes and paths for CUDA
CUDA_PATH := /usr/local/cuda
-CUFILE_PATH ?= /usr/local/cuda/targets/x86_64-linux/lib/
-CUFILE_INCLUDE_PATH ?= /usr/local/cuda/targets/x86_64-linux/include/
+CUFILE_PATH ?= /usr/local/cuda/lib/
+CUFILE_INCLUDE_PATH ?= /usr/local/cuda/include/
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
CXXFLAGS := -Wall
CXXFLAGS += -I $(CUDA_PATH)/include/
CXXFLAGS += -I $(CUFILE_INCLUDE_PATH)

17 changes: 17 additions & 0 deletions recipes/libcufile/build.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
if not exist %PREFIX% mkdir %PREFIX%
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this Windows build script, but we do need a linux one to set up the targets directory:
https://github.com/conda-forge/staged-recipes/pull/21723/files#diff-e5bafa1dffa25db6c210c96887404cd7168b4c02772daf9fa5cbedb52b5fbd0e
The pkgconfig name will need to be updated to "cufile*.pc"
And the ppc64le line can be dropped


rem Directories...
for /D %%i in (.\*) do (
robocopy /E %%i %PREFIX%\%%i
)

rem Files...
for %%i in (.\*) do (
if not %%~ni==build_env_setup (
if not %%~ni==conda_build (
if not %%~ni==metadata_conda_debug (
xcopy %%i %PREFIX%
)
)
)
)
94 changes: 94 additions & 0 deletions recipes/libcufile/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
{% set name = "libcufile" %}
{% set version = "1.5.0.59" %}
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
{% set platform = "linux-x86_64" %} # [linux64]
{% set platform = "linux-ppc64le" %} # [ppc64le]
{% set platform = "linux-sbsa" %} # [aarch64]
{% set platform = "windows-x86_64" %} # [win]
{% set extension = "tar.xz" %} # [not win]
{% set extension = "zip" %} # [win]
mroeschke marked this conversation as resolved.
Show resolved Hide resolved

package:
name: {{ name|lower }}
version: {{ version }}

source:
url: https://developer.download.nvidia.com/compute/cuda/redist/{{ name }}/{{ platform }}/{{ name }}-{{ platform }}-{{ version }}-archive.{{ extension }}
sha256: 222c49b29afb544c69542c4e658b0a1bd01fd2306f2136ad6990ccfff482a097 # [linux64]
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
patches:
- Makefile.patch

build:
# uncomment the following line if the package is pure python and the recipe is exactly the same for all platforms.
# it is okay if the dependencies are not built for all platforms/versions, although selectors are still not allowed.
# see https://conda-forge.org/docs/maintainer/knowledge_base.html#noarch-python for more details.
# noarch: python
# if the installation is complex, or different between unix and windows, use separate bld.bat and build.sh files instead of this key.
# by default, the package will be built for the python versions supported by conda-forge and for all major oss.
# add the line "skip: true # [py<35]" (for example) to limit to python 3.5 and newer, or "skip: true # [not win]" to limit to windows.
# more info about selectors can be found in the conda-build docs:
# https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#preprocessing-selectors
#script: {{ python }} -m pip install . -vv
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
number: 0
skip: true # [osx]
skip: true # [ppc64le]
skip: true # [aarch64]
skip: true # [win]
mroeschke marked this conversation as resolved.
Show resolved Hide resolved

outputs:
- name: libcufile
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
files:
- lib/libcufile*.so.*
- etc/cufile.json
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's already been checked, but does cuFile use the file put here as expected?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good question. Will ask

It's possible we would need to set CUFILE_ENV_PATH_JSON for this to work. That could be done with activation script

Suppose a fair question would be whether we want to do this or whether users would rather maintain their own configuration file

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we don't need this file. So we dropped it to avoid confusion

We can always revisit this in the feedstock

test:
commands:
- test -f $PREFIX/lib/libcufile*.so.*
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
- test -f $PREFIX/etc/cufile.json

- name: libcufile-dev
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
requirements:
run:
- libcufile >={{ version }}
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
files:
- lib/libcufile*.so
- include/cufile*
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
- man
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adibbley, what do you think about including man in libcufile? Wonder if it is helpful for the end user to have access to the man files for configuration, etc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it should be there, it is in the 12.0.0 branch of the original recipe.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- man

test:
commands:
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
- test -f $PREFIX/lib/libcufile*.so
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
- test -f $PREFIX/include/cufile*
- test -f $PREFIX/man

- name: libcufile-static
requirements:
run:
- libcufile-dev >={{ version }}
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
files:
- lib/libcufile*_static.a
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
test:
commands:
- test -f $PREFIX/lib/libcufile*_static.a
mroeschke marked this conversation as resolved.
Show resolved Hide resolved

- name: gds-tools
requirements:
run:
- libcufile >={{ version }}
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
files:
- gds/samples
- gds/tools
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
test:
commands:
- test -f $PREFIX/gds/samples
- test -f $PREFIX/gds/tools
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which test here is failing?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah nvm. It's gds/samples (from CI)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test flags should also be -d once they are shuffled to the expected location.


about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: |
Library for GPU Direct Storage with CUDA
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

extra:
recipe-maintainers:
- adibbley