package tensorflow_addons #16888

h-vetinari · 2021-11-10T12:36:58Z

See #16646, conda-forge/tensorflow-feedstock#133, tensorflow/addons#201

Closes #16646
Closes conda-forge/tensorflow-feedstock#133

conda-forge-linter · 2021-11-10T12:37:04Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/tensorflow-addons) and found it was in an excellent condition.

h-vetinari · 2021-11-10T14:22:39Z

@conda-forge/tensorflow @jaimergp
Before , I could see that the upstream configure.py adds the following lines to .bazelrc:

build --action_env CUDA_TOOLKIT_PATH="/usr/local/cuda"
build --action_env CUDNN_INSTALL_PATH="/usr/lib/x86_64-linux-gnu"

Those variables seem to be specific to tf-addons (cannot find them elsewhere on GH), so will most likely need adapting. Any tips (much less joint maintainership) are appreciated. It seems that the INCLUDEDIR bit works in a different way here than in the tf-feedstock...

bazel-out/k8-opt/bin/external/local_config_tf/include/tensorflow/core/framework/full_type.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory
   10 | #include <google/protobuf/port_def.inc>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

PS. For OSX, we're missing more recent tf-builds (ever since the build-string got refactored) - or even better: add tensorflow-cpu for osx. Some discussion here.

jaimergp · 2021-11-10T14:32:55Z

CUDNN_INSTALL_PATH is set to $PREFIX. I don't know what the build system will do with CUDA_TOOLKIT_PATH, but in principle, that path is correct on our Docker images and, if anything, it can be set to CUDA_HOME. I don't know what will happen if it's set to (BUILD_)PREFIX, but it might be worth a try (so our nvcc wrapper is found).

h-vetinari · 2021-11-10T15:02:46Z

Thanks for the quick response!

CUDNN_INSTALL_PATH is set to $PREFIX.

Not sure if that's a comment that what I'm doing is questionable or that it should be like that (I guessed based on the expected header-path)...?

xhochy

What about also using the bazel-toolchain here (will probably be needed for osx, at least for osx-arm64`)? The latest changes in tensorflow in the bazel-toolchain should also fix some of the CUDA issues.

recipes/tensorflow-addons/meta.yaml

h-vetinari · 2021-11-10T21:54:18Z

What about also using the bazel-toolchain [...]?

Absolutely, I just hadn't had time to invest yet how to make use of that package. Happy to try.

h-vetinari · 2021-11-10T22:07:12Z

Ah, I did not expect that bazel-toolchain does not depend on bazel, but I can understand why.

h-vetinari · 2021-11-10T22:46:44Z

So tf-addons seems to process the cuda capabilities differently than tf proper. Also, I don't know what they're doing behind the scenes (sm_ vs. compute_).

Further point that jumped out from the log - since we're not dealing with manylinux etc., we probably don't need TF_CXX11_ABI_FLAG?

Co-authored-by: Uwe L. Korn <[email protected]>

also add libtensorflow as host dep

h-vetinari · 2021-11-11T05:04:14Z

recipes/tensorflow-addons/meta.yaml

+    # due to lack of ABI, we need the same version at runtime, see
+    # https://github.com/tensorflow/addons#c-custom-op-compatibility-matrix
+    - tensorflow {{ tf_version_maj_min }}.*
+    - libtensorflow_cc {{ tf_version_maj_min }}.*


TBH, I don't understand this - it's already there as a host req, so why does it also need to be in run? - but without this we get:

ERROR (tensorflow-addons,lib/[something].so): Needed DSO lib/libtensorflow_framework.so.2 found in ['libtensorflow_cc'] ERROR (tensorflow-addons,lib/[something].so): .. but ['libtensorflow_cc'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)

libtensorflow_cc doesn't have a run_exports, we should add one there.

This reverts commit 21874b8.

h-vetinari · 2021-11-11T05:20:33Z

OSX builds are blocked on conda-forge/tensorflow-feedstock#144...

h-vetinari · 2021-11-29T01:09:21Z

OK, trying to add a reference to $SP_DIR/tensorflow/include runs into:

ERROR: /home/conda/staged-recipes/build_artifacts/tensorflow-addons_1638147416024/work/tensorflow_addons/custom_ops/image/BUILD:35:18: Compiling tensorflow_addons/custom_ops/image/cc/ops/resampler_ops.cc failed: The include path '/home/conda/staged-recipes/build_artifacts/[...]/lib/python3.7/site-packages/tensorflow/include' references a path outside of the execution root.

This SO answer has a number of steps to work around this (not sure if outdated), but before I try, is this something that bazel-toolchain is taking care of, or something it could do?
@xhochy

h-vetinari · 2022-02-11T12:22:21Z

@xhochy, do you know a better way of handing an external dependency to bazel than the above?

thadd3us-abcellera · 2022-02-24T14:27:13Z

@xhochy, do you know a better way of handing an external dependency to bazel than the above?

@h-vetinari

Does it work to do something like this in .bazelrc:

# This syntax exposes the variable from the .envrc environment.
# https://bazel.build/designs/2016/06/21/environment.html#new-flag---action_env
build --action_env TF_HEADER_DIR
build --action_env TF_SHARED_LIBRARY_DIR
build --action_env TF_SHARED_LIBRARY_NAME
build --action_env TF_CXX11_ABI_FLAG

Are you able to get those environment variables set somehow, and then use this mechanism to expose them to bazel?

In my deployment, I was using direnv to get them into the environment.

stale · 2022-07-30T19:33:28Z

Hi friend!

We really, really, really appreciate that you have taken the time to make a PR on conda-forge/staged-recipes! conda-forge only exists because people like you donate their time to build and maintain conda recipes for use by the community.

In an effort to maintain this repository and increase the signal-to-noise for open PRs, the maintainers of staged-recipes close excessively old PRs after six months. This PR will remain open for another month, and then will be closed.

If you'd like to keep it open, please comment/push and we will be happy to oblige! Note that very old PRs will likely need to be rebased on main so that they can be rebuilt with the most recent CI scripts. If you have any trouble, or we missed reviewing this PR in the first place (sorry!), feel free to ping the team using a special command in a comment on the PR to get the attention of the staged-recipes team.

Cheers and thank you for contributing to this community effort!

h-vetinari · 2022-07-31T21:40:47Z

not stale

h-vetinari · 2022-07-31T21:41:52Z

@thadd3us-abcellera

Sorry I overlooked your tip! Would you mind raising a PR against https://github.com/h-vetinari/staged-recipes/tree/tf-addons?

h-vetinari · 2022-08-01T22:07:41Z

@conda-forge/help-python

For some reason the bot here has not removed the stale label - would someone be so kind to do that please?

h-vetinari · 2022-08-01T22:09:26Z

Interesting, now it worked all of a sudden. Sorry for the noise (and this is not yet a request for review, unless someone wants to help me tell bazel where to look for the tensorflow shared lib).

recipes/tensorflow-addons/build.sh

jonas-eschle · 2022-09-06T15:46:12Z

@h-vetinari does the above maybe do the trick?

Co-authored-by: jaimergp <[email protected]>

conda-forge-webservices · 2023-02-14T00:38:30Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/tensorflow-addons) and found it was in an excellent condition.

stale · 2023-07-22T16:54:48Z

Hi friend!

We really, really, really appreciate that you have taken the time to make a PR on conda-forge/staged-recipes! conda-forge only exists because people like you donate their time to build and maintain conda recipes for use by the community.

In an effort to maintain this repository and increase the signal-to-noise for open PRs, the maintainers of staged-recipes close excessively old PRs after six months. This PR will remain open for another month, and then will be closed.

If you'd like to keep it open, please comment/push and we will be happy to oblige! Note that very old PRs will likely need to be rebased on main so that they can be rebuilt with the most recent CI scripts. If you have any trouble, or we missed reviewing this PR in the first place (sorry!), feel free to ping the team using a special command in a comment on the PR to get the attention of the staged-recipes team.

Cheers and thank you for contributing to this community effort!

stale · 2024-01-01T03:56:20Z

Hi again! About a month ago, we commented on this PR saying it would be closed in another month if it was still inactive. It has been a month and so now it is being closed. Thank you so much for making it in the first place and contributing to the community project that is conda-forge. If you'd like to reopen this PR, please feel free to do so at any time!

Cheers and have a great day!

h-vetinari mentioned this pull request Nov 10, 2021

Migrate libprotobuf, grpcio, icu conda-forge/tensorflow-feedstock#144

Merged

6 tasks

xhochy reviewed Nov 10, 2021

View reviewed changes

recipes/tensorflow-addons/meta.yaml Outdated Show resolved Hide resolved

h-vetinari force-pushed the tf-addons branch 2 times, most recently from 4f8c9ea to 9477c9f Compare November 11, 2021 02:35

h-vetinari and others added 10 commits November 11, 2021 15:33

add tensorflow_addons

f74fe06

harmonize tf-versions due to lack of ABI

732f982

make up for missing tf-cpu on osx

21874b8

Direct maintainership to team of tensorflow-feedstock

fe7d8a9

Co-authored-by: Uwe L. Korn <[email protected]>

switch to bazel-toolchain

f01a5b8

change compute capabilities to format used by tf-addons

2e0b6dd

add GCC_HOST_COMPILER_PATH

84c72c5

adapt output of configure.py with sed

96a5d0e

also add libtensorflow as host dep

use libtf_cc

3a8b268

add rsync

382a68c

h-vetinari force-pushed the tf-addons branch from 61c7782 to 382a68c Compare November 11, 2021 04:34

h-vetinari added 2 commits November 11, 2021 16:00

add libtensorflow_cc as run-req

c7fecf9

centralize tf-version used in recipe

61393ef

h-vetinari commented Nov 11, 2021

View reviewed changes

h-vetinari mentioned this pull request Nov 11, 2021

Package Addons on conda-forge tensorflow/addons#201

Closed

h-vetinari added 2 commits November 11, 2021 16:16

Revert "make up for missing tf-cpu on osx"

57ac135

This reverts commit 21874b8.

add typeguard as run-req

dd1cc0a

pass through test folder

bda1641

h-vetinari mentioned this pull request Nov 26, 2021

Dropping CentOS 6 & Moving to CentOS 7 conda-forge/conda-forge.github.io#1436

Open

h-vetinari added 2 commits November 29, 2021 11:41

simplify dependencies again

5549d4a

pass extra header directory to bazel build

fa03456

h-vetinari force-pushed the tf-addons branch from ccc1669 to fa03456 Compare November 29, 2021 01:43

stale bot added the stale will be closed in 30 days label Jul 30, 2022

Merge remote-tracking branch 'upstream/main' into tf-addons

9ae7a50

stale bot removed the stale will be closed in 30 days label Aug 1, 2022

github-actions bot added python review-requested labels Aug 1, 2022

carterbox removed the review-requested label Aug 13, 2022

github-actions bot added the Awaiting author contribution label Aug 13, 2022

jaimergp reviewed Aug 18, 2022

View reviewed changes

recipes/tensorflow-addons/build.sh Show resolved Hide resolved

Update recipes/tensorflow-addons/build.sh

ddcd0fd

Co-authored-by: jaimergp <[email protected]>

stale bot added the stale will be closed in 30 days label Jul 22, 2023

stale bot closed this Jan 1, 2024

h-vetinari reopened this Jan 1, 2024

stale bot removed the stale will be closed in 30 days label Jan 1, 2024

h-vetinari mentioned this pull request Jan 19, 2024

ENH: equivalent of local_cc_dependency override in bzlmod bazelbuild/bazel#20947

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

package tensorflow_addons #16888

package tensorflow_addons #16888

h-vetinari commented Nov 10, 2021 •

edited

Loading

conda-forge-linter commented Nov 10, 2021

h-vetinari commented Nov 10, 2021

jaimergp commented Nov 10, 2021

h-vetinari commented Nov 10, 2021

xhochy left a comment •

edited

Loading

h-vetinari commented Nov 10, 2021

h-vetinari commented Nov 10, 2021

h-vetinari commented Nov 10, 2021

h-vetinari Nov 11, 2021

xhochy Nov 16, 2021

h-vetinari commented Nov 11, 2021

h-vetinari commented Nov 29, 2021

h-vetinari commented Feb 11, 2022

thadd3us-abcellera commented Feb 24, 2022 •

edited

Loading

stale bot commented Jul 30, 2022

h-vetinari commented Jul 31, 2022

h-vetinari commented Jul 31, 2022

h-vetinari commented Aug 1, 2022

h-vetinari commented Aug 1, 2022

jonas-eschle commented Sep 6, 2022

conda-forge-webservices bot commented Feb 14, 2023

stale bot commented Jul 22, 2023

stale bot commented Jan 1, 2024

package tensorflow_addons #16888

Are you sure you want to change the base?

package tensorflow_addons #16888

Conversation

h-vetinari commented Nov 10, 2021 • edited Loading

conda-forge-linter commented Nov 10, 2021

h-vetinari commented Nov 10, 2021

jaimergp commented Nov 10, 2021

h-vetinari commented Nov 10, 2021

xhochy left a comment • edited Loading

Choose a reason for hiding this comment

h-vetinari commented Nov 10, 2021

h-vetinari commented Nov 10, 2021

h-vetinari commented Nov 10, 2021

h-vetinari Nov 11, 2021

Choose a reason for hiding this comment

xhochy Nov 16, 2021

Choose a reason for hiding this comment

h-vetinari commented Nov 11, 2021

h-vetinari commented Nov 29, 2021

h-vetinari commented Feb 11, 2022

thadd3us-abcellera commented Feb 24, 2022 • edited Loading

stale bot commented Jul 30, 2022

h-vetinari commented Jul 31, 2022

h-vetinari commented Jul 31, 2022

h-vetinari commented Aug 1, 2022

h-vetinari commented Aug 1, 2022

jonas-eschle commented Sep 6, 2022

conda-forge-webservices bot commented Feb 14, 2023

stale bot commented Jul 22, 2023

stale bot commented Jan 1, 2024

h-vetinari commented Nov 10, 2021 •

edited

Loading

xhochy left a comment •

edited

Loading

thadd3us-abcellera commented Feb 24, 2022 •

edited

Loading