Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use dlopen for FFmpeg #3353

Closed
wants to merge 1 commit into from
Closed

Use dlopen for FFmpeg #3353

wants to merge 1 commit into from

Conversation

mthrok
Copy link
Collaborator

@mthrok mthrok commented May 21, 2023

This commit changes the way FFmpeg extension is built and used.
Instead of linking (LGPL) FFmpeg libraries to torchaudio at build time,
It uses dlopen to search and link them at run time.

For dlopen-ing, we use PyTorch's at::DynamicLibrary class, which provides
portable wrapper.

@pytorch-bot
Copy link

pytorch-bot bot commented May 21, 2023

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/audio/3353

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure

As of commit b7f3547:

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following job failed but were present on the merge base c7ac1af:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot
Copy link
Contributor

@mthrok has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@mthrok mthrok force-pushed the dlopen branch 2 times, most recently from 508f735 to 8248236 Compare May 23, 2023 22:36
@mthrok mthrok force-pushed the dlopen branch 2 times, most recently from 35ed61f to 0f69178 Compare June 1, 2023 05:06
@facebook-github-bot
Copy link
Contributor

@mthrok has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

1 similar comment
@facebook-github-bot
Copy link
Contributor

@mthrok has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@mthrok mthrok changed the title WIP: Use dlopen Use dlopen for FFmpeg Jun 1, 2023
@mthrok mthrok marked this pull request as ready for review June 1, 2023 12:11
mthrok added a commit to mthrok/audio that referenced this pull request Jun 1, 2023
Summary:
This commit changes the way FFmpeg extension is built and used.
Instead of linking (LGPL) FFmpeg libraries to torchaudio at build time,
It uses dlopen to search and link them at run time.

For dlopen-ing, we use PyTorch's `at::DynamicLibrary` class, which provides
portable wrapper.

Pull Request resolved: pytorch#3353

Differential Revision: D46059199

Pulled By: mthrok

fbshipit-source-id: 6ae94cd3f3ded717d0e0956e9d214e1df5c1185f
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D46059199

mthrok added a commit to mthrok/audio that referenced this pull request Jun 1, 2023
Summary:
This commit changes the way FFmpeg extension is built and used.
Instead of linking (LGPL) FFmpeg libraries to torchaudio at build time,
It uses dlopen to search and link them at run time.

For dlopen-ing, we use PyTorch's `at::DynamicLibrary` class, which provides
portable wrapper.

Pull Request resolved: pytorch#3353

Differential Revision: D46059199

Pulled By: mthrok

fbshipit-source-id: 8acf2b572a524278aa8d839828740d082a40b434
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D46059199

Summary:
This commit changes the way FFmpeg extension is built and used.
Instead of linking (LGPL) FFmpeg libraries to torchaudio at build time,
It uses dlopen to search and link them at run time.

For dlopen-ing, we use PyTorch's `at::DynamicLibrary` class, which provides
portable wrapper.

Pull Request resolved: pytorch#3353

Differential Revision: D46059199

Pulled By: mthrok

fbshipit-source-id: e030fdec7ee2dc99c2d1c29d56c1e66e2ca42ccc
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D46059199

@facebook-github-bot
Copy link
Contributor

@mthrok merged this pull request in b14ced1.

@github-actions
Copy link

github-actions bot commented Jun 1, 2023

Hey @mthrok.
You merged this PR, but labels were not properly added. Please add a primary and secondary label (See https://github.com/pytorch/audio/blob/main/.github/process_commit.py).


Some guidance:

Use 'module: ops' for operations under 'torchaudio/{transforms, functional}', and ML-related components under 'torchaudio/csrc' (e.g. RNN-T loss).

Things in "examples" directory:

  • 'recipe' is applicable to training recipes under the 'examples' folder,
  • 'tutorial' is applicable to tutorials under the “examples/tutorials” folder
  • 'example' is applicable to everything else (e.g. C++ examples)
  • 'module: docs' is applicable to code documentations (not to tutorials).
    Regarding examples in code documentations, please also use 'module: docs'.

Please use 'other' tag only when you’re sure the changes are not much relevant to users, or when all other tags are not applicable. Try not to use it often, in order to minimize efforts required when we prepare release notes.


When preparing release notes, please make sure 'documentation' and 'tutorials' occur as the last sub-categories under each primary category like 'new feature', 'improvements' or 'prototype'.

Things related to build are by default excluded from the release note, except when it impacts users. For example:
* Drop support of Python 3.7.
* Add support of Python 3.X.
* Change the way a third party library is bound (so that user needs to install it separately).

@facebook-github-bot
Copy link
Contributor

This pull request has been reverted by ab7a39f.

mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Differential Revision: D46403783

fbshipit-source-id: f39c9c639e9da1cbdd61c31008014b2b0b91af67
mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
Pull Request resolved: pytorch#3402

This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Differential Revision: D46403783

fbshipit-source-id: 097a4cc6ba7a0fbae317632ee81e4aa895f62fde
mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
Pull Request resolved: pytorch#3402

This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Differential Revision: D46403783

fbshipit-source-id: 10a193c7fa27f6075975609974890c9ab80ddc6b
mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
Pull Request resolved: pytorch#3402

This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Differential Revision: D46403783

fbshipit-source-id: 2e3f36de6962ebceb05dccf6f7763826c25960aa
mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
Pull Request resolved: pytorch#3402

This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Differential Revision: D46403783

fbshipit-source-id: 4b675e1c1239503f087a0fd50d078340c09d2282
mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
Pull Request resolved: pytorch#3402

This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Differential Revision: D46403783

fbshipit-source-id: 5cbf733b40e4499570336804af597c3d281321ba
mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
Pull Request resolved: pytorch#3402

This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Differential Revision: D46403783

fbshipit-source-id: 08975d1973a1f8c2c402c5e76fe0cbf469fa9156
mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
Pull Request resolved: pytorch#3402

This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Differential Revision: D46403783

fbshipit-source-id: 5b28c0b136552b2f5c70ee487ae7840efda4e18a
mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
Pull Request resolved: pytorch#3402

This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Please refer to stub.h for more technical detail.

Differential Revision: D46403783

fbshipit-source-id: 28758ed3b40e4de418c9f7014bd29e35cefbe3ec
mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
Pull Request resolved: pytorch#3402

This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Please refer to stub.h for more technical detail.

Differential Revision: D46403783

fbshipit-source-id: 3578de808d6c87358226ee5372365c71cc62d6d7
mthrok added a commit to mthrok/audio that referenced this pull request Jun 2, 2023
Summary:
Pull Request resolved: pytorch#3402

This is a second attempt of pytorch#3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Please refer to stub.h for more technical detail.

Differential Revision: D46403783

fbshipit-source-id: 8481ca9d465d46b08db8c24d86b471142054d5c6
facebook-github-bot pushed a commit that referenced this pull request Jun 3, 2023
Summary:
Pull Request resolved: #3402

This is a second attempt of #3353.

The basic logic to enable dlopen for FFmpeg libraries are same.
It uses `at::DynamicLibrary`, which allows to compile torchaudio without
linking FFmpeg libraries.

This time, the option to enable this feature DLOPEN_FFMPEG has been added,
so that users have a way to disable this feature and keep using build-time
linking.

Please refer to stub.h for more technical detail.

Differential Revision: D46403783

fbshipit-source-id: ca3db57ff6bdc50c8c225d22f12f3e76c6dc3f16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants