add mxnet #17580

ngam · 2022-01-19T02:41:19Z

Checklist

Title of this PR is meaningful: e.g. "Adding my_nifty_package", not "updated meta.yaml".
License file is packaged (see here for an example).
Source is from official source.
Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged).
If static libraries are linked in, the license of the static library is packaged.
Build number is 0.
A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details).
GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there.
When in trouble, please check our knowledge base documentation before pinging a team.

conda-forge-linter · 2022-01-19T02:41:23Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipes/mxnet:

License is not an SPDX identifier (or a custom LicenseRef) nor an SPDX license expression.

Documentation on acceptable licenses can be found here.

conda-forge-linter · 2022-01-19T02:42:58Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found it was in an excellent condition.

conda-forge-linter · 2022-01-20T14:43:54Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found some lint.

Here's what I've got...

For recipes/mxnet:

requirements: host: requests >= 2.20.0,<3 should not contain a space between relational operator and the version, i.e. requests >=2.20.0,<3
requirements: run: requests >= 2.20.0,<3 should not contain a space between relational operator and the version, i.e. requests >=2.20.0,<3

conda-forge-linter · 2022-01-20T14:45:51Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found it was in an excellent condition.

conda-forge-linter · 2022-01-20T20:15:26Z

Hi! This is the friendly automated conda-forge-linting service.

I was trying to look for recipes to lint for you, but couldn't find any.
Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

conda-forge-linter · 2022-01-20T20:23:34Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found it was in an excellent condition.

conda-forge-linter · 2022-01-20T21:18:25Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found some lint.

Here's what I've got...

For recipes/mxnet:

Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [64]

conda-forge-linter · 2022-01-20T21:29:44Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found it was in an excellent condition.

conda-forge-linter · 2022-01-22T07:39:27Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found some lint.

Here's what I've got...

For recipes/mxnet:

Failed to even lint the recipe, probably because of a conda-smithy bug 😢. This likely indicates a problem in your meta.yaml, though. To get a traceback to help figure out what's going on, install conda-smithy and run conda smithy recipe-lint . from the recipe directory.

conda-forge-linter · 2022-01-22T07:42:28Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found it was in an excellent condition.

ngam · 2022-01-22T07:44:42Z

@conda-forge/staged-recipes ready for review.

The CI may not finish the cuda builds in time due to the complexity of the build --- I could try to upload the logs at some point. Here's a file from this recipe https://anaconda.org/ngam/mxnet/v2.0.0.beta0.rc1/download/osx-arm64/mxnet-v2.0.0.beta0.rc1-py39h32763b8_0_cpu.tar.bz2 but I have since tightened the pins on opencv and libjpeg-turbo for further safety; would potentially like to add an additional modification on numpy (selecting the corresponding backend by default, but not sure if that's doable or desirable)

hmaarrfk · 2022-01-22T14:57:28Z

recipes/mxnet/meta.yaml

+    - onednn
+    - cudnn  # [cuda_compiler_version != "None"]
+    - nccl  # [cuda_compiler_version != "None"]
+    - numpy >=1.17


Numpy is quite forward compatible, but it isn't really backward compatible. Doing this kind of pinning uses

2022-01-22T08:20:42.7430135Z + numpy 1.21.5 py37hf2998dd_0 conda-forge/linux-64 6 MB

which makes it only compatible with numpy 1.21.5

Please follow
https://conda-forge.org/docs/maintainer/knowledge_base.html#building-against-numpy

So, should I just numpy ==1.17.* or should I just drop the pinning here?

I actually don't if numpy needs to be in this section... I will need to check that too!

just drop the pin. it will get a newer number from the config.

OK, will remove. I now think numpy should only be in run... testing this takes forever, that's why I am hesitating to keep going back and forth, but I will try tweaking things around and tidying up locally first.

Thanks a lot! 🥇

recipes/mxnet/meta.yaml

hmaarrfk · 2022-01-22T14:58:51Z

recipes/mxnet/meta.yaml

+    - requests >=2.20.0,<3
+    - python-graphviz >=0.8.1,<0.9.0
+    - python_version
+    - {{ pin_compatible('libjpeg-turbo') }}


This is going to cause inter-operability problems. You should simply build with jpeg. Do you need turbo? or is it just for performance?

I believe they explicitly require turbo, but I can look deeper into it. It is actually built against libjpeg-turbo and I am not really sure if it is a runtime dep. I added all this pinning because when I was testing locally, I would run into problems like "can't find library libopencv.xxx.405" so I thought ooops, I gotta pin them all to start and then relax them later. What's your recommendation here? We potentially will have all sorts of problematic pinning because these are used in the building, not merely as a flexible runtime deps and so this is quite challenging for me to understand as a beginner!

hmaarrfk · 2022-01-22T15:02:47Z

recipes/mxnet/meta.yaml

+    - {{ pin_compatible('llvm-openmp') }}  # [osx]
+    - openblas
+    - {{ pin_compatible('mkl') }}  # [(target_platform != "osx-arm64") and (cuda_compiler_version == "None")]
+    - {{ pin_compatible('mkl-include') }}  # [(target_platform != "osx-arm64") and (cuda_compiler_version == "None")]


Do you need development headers?

There is an explicit line in build.sh for the MKL include, but I haven't really checked if this passes without this and I am the whole onednn--mkl building is very confusing :(

I think mkl-include can be dropped, but I will want to check that first

ngam · 2022-01-22T15:32:00Z

@hmaarrfk, this is going to be a challenging and interesting feedstock to maintain --- while I am only running some of the "supported" distributions here (config files), the whole thing upstream is really flexible and we can potentially do a lot of interesting mixes for our purposes and help them out with distributing and optimizing their build. For example, currently, I believe at least some of onednn gets built --- even though it looks for it at the start to ensure compatibility, which is very confusing!

Anyway, would you like me to add you as a maintainer here? This way you'd be on the tensorflow, pytorch, and mxnet feedstocks --- the whole axis lol. And you will teach me all the tricks :) Copying @h-vetinari to gauge interest in joining the maintainership too

Edit: What I mean is: Please join me in maintaining this 🙇‍♂️

hmaarrfk · 2022-01-22T19:29:06Z

recipes/mxnet/meta.yaml

+
+extra:
+  recipe-maintainers:
+    - barry-jin  


This person should chime in before we can merge. I suggest you add them after in the feedstock itself. instead of now at the beginning.

@barry-jin indicated willingness (also a maintainer upstream). See above for the "like" and here #4447 (comment) but happy to remove for now and tidy up things later.

Btw, I couldn't figure out a clean way to get the library separated. The python package itself can be noarch and can be built separately. This is one main thing I wanna do once we get this into a feedstock. Another thing is being involved with niche (?) dependencies so that we can avoid "vendoring" them (to be clear, we are not vendoring much, but still I would like to figure out a way to get dlpack, dmlc, nnvm (libtvm) and mshadow from elsewhere --- though, it is not clear to me exactly how; I tried and I failed, so I am going to delay that until we get this up and running in a feedstock). I will also have one last swipe at the messy deps once this run concludes.

ngam · 2022-01-22T20:43:03Z

deps need fixing

   INFO: sysroot: '/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/' files: '['usr/share/man/mann/zero.n', 'usr/share/man/mann/yencode.n', 'usr/share/man/mann/yaml.n', 'usr/share/man/mann/xsxp.n']'
   INFO (mxnet,mxnet/libmxnet.dylib): Needed DSO lib/libmkl_intel_ilp64.1.dylib found in conda-forge::mkl-2021.4.0-h89fa619_689
   INFO (mxnet,mxnet/libmxnet.dylib): Needed DSO /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/lib/libSystem.B.dylib found in $SYSROOT
   INFO (mxnet,mxnet/libmxnet.dylib): Needed DSO lib/libturbojpeg.0.dylib found in conda-forge::libjpeg-turbo-2.1.1-h0d85af4_0
  ERROR (mxnet,mxnet/libmxnet.dylib): Needed DSO lib/libopencv_imgcodecs.405.dylib found in ['libopencv']
  ERROR (mxnet,mxnet/libmxnet.dylib): .. but ['libopencv'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
  ERROR (mxnet,mxnet/libmxnet.dylib): Needed DSO lib/libopencv_imgproc.405.dylib found in ['libopencv']
  ERROR (mxnet,mxnet/libmxnet.dylib): .. but ['libopencv'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
  ERROR (mxnet,mxnet/libmxnet.dylib): Needed DSO lib/libopencv_core.405.dylib found in ['libopencv']
  ERROR (mxnet,mxnet/libmxnet.dylib): .. but ['libopencv'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
  ERROR (mxnet,mxnet/libmxnet.dylib): Needed DSO lib/libprotobuf.30.dylib found in ['libprotobuf']
  ERROR (mxnet,mxnet/libmxnet.dylib): .. but ['libprotobuf'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
   INFO (mxnet,mxnet/libmxnet.dylib): Needed DSO lib/libc++.1.dylib found in conda-forge::libcxx-12.0.1-habf9029_1
WARNING (mxnet): interpreter (Python) package conda-forge::python-3.8.12-h17280f6_2_cpython in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (mxnet): interpreted library (Python) package conda-forge::python_version-0.0.2-pyhd8ed1ab_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (mxnet): interpreted library (Python) package conda-forge::python-graphviz-0.8.4-py_1003 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (mxnet): run-exports library package conda-forge::llvm-openmp-11.1.0-hda6cdc1_1 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (mxnet): interpreted library (Python) package conda-forge::pip-21.3.1-pyhd8ed1ab_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (mxnet): run-exports library package conda-forge::libgfortran5-9.3.0-h6c81a4c_23 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (mxnet): run-exports library package conda-forge::onednn-2.5.2-omp_h4900ede_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (mxnet): plugin library (Python) package conda-forge::numpy-1.22.0-py38h9d72dae_1 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
WARNING (mxnet): interpreted library (Python) package conda-forge::requests-2.27.1-pyhd8ed1ab_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)

ngam · 2022-01-22T23:08:29Z

While this is actually more or less ready, I'd rather spend a little more time trying to split it into a library and python pkg instead of this. Converting back to draft, will ping again soon.

conda-forge-linter · 2022-01-23T00:38:28Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/libmxnet, recipes/mxnet) and found some lint.

Here's what I've got...

For recipes/libmxnet:

If python is a host requirement, it should be a run requirement.

conda-forge-linter · 2022-01-23T00:39:29Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/libmxnet, recipes/mxnet) and found it was in an excellent condition.

conda-forge-linter · 2022-01-23T01:45:36Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found some lint.

Here's what I've got...

For recipes/mxnet:

Non noarch packages should have python requirement without any version constraints.
Non noarch packages should have python requirement without any version constraints.

conda-forge-linter · 2022-01-23T01:47:36Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/mxnet) and found it was in an excellent condition.

h-vetinari · 2024-05-28T04:24:48Z

@ngam, I missed this PR at the time, and I don't see much context for its closure (except that I remember you don't like having open PRs...?). What would you think about giving this one another shot?

ngam · 2024-05-28T14:47:53Z

I don't think we should... at the time when I was trying to push this, it appeared as though mxnet was going to move to a community model (apache) and there was a push to get a big release (2.0.0). However, things stalled to essentially a halt. It was clear to me at the time something was up, and unsurprisingly --- but sadly --- the project has been retired last time I checked about a year later...

This is kinda similar to other "disappointing" news wrt to tvm, etc :/

So, I think in conda-forge, we should focus on the most active of the frameworks in this space (tf, jax, pytorch --- and as you know, they're already very challenging)

ngam marked this pull request as draft January 19, 2022 03:03

This was referenced Jan 19, 2022

Package mxnet #4447

Open

Adding MXNet #14832

Closed

Error: output metadata set is empty. Please file an issue on the conda-build conda/conda-build#4362

Closed

ngam force-pushed the libmxnet branch from f992a01 to 2eaa1c6 Compare January 20, 2022 15:29

ngam marked this pull request as ready for review January 20, 2022 15:56

ngam force-pushed the libmxnet branch from 1c5eaf7 to ccf73d3 Compare January 20, 2022 20:31

ngam force-pushed the libmxnet branch 2 times, most recently from 20e3417 to 341ef1a Compare January 21, 2022 08:10

ngam mentioned this pull request Jan 21, 2022

add osxarm conda-forge/dmlc-feedstock#9

Merged

ngam force-pushed the libmxnet branch from 3d1e192 to 765c6b0 Compare January 22, 2022 06:24

ngam force-pushed the libmxnet branch from cfac3cb to b569b67 Compare January 22, 2022 07:43

hmaarrfk reviewed Jan 22, 2022

View reviewed changes

recipes/mxnet/meta.yaml Outdated Show resolved Hide resolved

hmaarrfk reviewed Jan 22, 2022

View reviewed changes

ngam marked this pull request as draft January 23, 2022 00:37

ngam force-pushed the libmxnet branch from b21c372 to bfbba35 Compare January 23, 2022 02:17

ngam closed this Jan 23, 2022

ngam force-pushed the libmxnet branch from bfbba35 to 0cab488 Compare January 23, 2022 02:20

ngam deleted the libmxnet branch January 23, 2022 02:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add mxnet #17580

add mxnet #17580

ngam commented Jan 19, 2022 •

edited

Loading

conda-forge-linter commented Jan 19, 2022

conda-forge-linter commented Jan 19, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 22, 2022

conda-forge-linter commented Jan 22, 2022

ngam commented Jan 22, 2022 •

edited

Loading

hmaarrfk Jan 22, 2022

ngam Jan 22, 2022

ngam Jan 22, 2022

hmaarrfk Jan 22, 2022

ngam Jan 22, 2022

ngam Jan 22, 2022

hmaarrfk Jan 22, 2022

ngam Jan 22, 2022

hmaarrfk Jan 22, 2022

ngam Jan 22, 2022

ngam commented Jan 22, 2022 •

edited

Loading

hmaarrfk Jan 22, 2022

ngam Jan 22, 2022 •

edited

Loading

ngam commented Jan 22, 2022

ngam commented Jan 22, 2022

conda-forge-linter commented Jan 23, 2022

conda-forge-linter commented Jan 23, 2022

conda-forge-linter commented Jan 23, 2022

conda-forge-linter commented Jan 23, 2022

h-vetinari commented May 28, 2024

ngam commented May 28, 2024

add mxnet #17580

add mxnet #17580

Conversation

ngam commented Jan 19, 2022 • edited Loading

conda-forge-linter commented Jan 19, 2022

conda-forge-linter commented Jan 19, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 20, 2022

conda-forge-linter commented Jan 22, 2022

conda-forge-linter commented Jan 22, 2022

ngam commented Jan 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ngam commented Jan 22, 2022 • edited Loading

Choose a reason for hiding this comment

ngam Jan 22, 2022 • edited Loading

Choose a reason for hiding this comment

ngam commented Jan 22, 2022

ngam commented Jan 22, 2022

conda-forge-linter commented Jan 23, 2022

conda-forge-linter commented Jan 23, 2022

conda-forge-linter commented Jan 23, 2022

conda-forge-linter commented Jan 23, 2022

h-vetinari commented May 28, 2024

ngam commented May 28, 2024

ngam commented Jan 19, 2022 •

edited

Loading

ngam commented Jan 22, 2022 •

edited

Loading

ngam commented Jan 22, 2022 •

edited

Loading

ngam Jan 22, 2022 •

edited

Loading