-
Notifications
You must be signed in to change notification settings - Fork 7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add RAFT model for optical flow #5022
Merged
Merged
Changes from 3 commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
d51172f
Add RAFT
NicolasHug 12375b8
Merge branch 'main' of github.com:pytorch/vision into raft_model_arch
NicolasHug 578373b
Minor fixes
NicolasHug 19629a3
add _init_weights() method
NicolasHug 44b4290
weights -> pretrained
NicolasHug 7ea67ea
Use ConvNormActivation in MaskPredictor
NicolasHug 6b3af06
Use nn.Identity instead of checking for None layers
NicolasHug 173ccde
Extract out _compute_corr_volume method
NicolasHug 8e444fd
Use F.relu instead of torch.relu
NicolasHug c05b872
Re-organize file structure and rename raft() into raft_large
NicolasHug 6678cff
Added support for torchscript, and added expect test
NicolasHug b1375ab
avoid import
NicolasHug 6f9bc85
ValueError instead of NotImplementedError
NicolasHug f655ec6
Allow higher tolerance for expectTest
NicolasHug a85b6b4
The docssssssss
NicolasHug c891a93
fix hooks
NicolasHug 9ae9e38
Fix hooks -- remastered
NicolasHug f077d7c
Merge branch 'main' into raft_model_arch
NicolasHug File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
from ._raft.raft import RAFT, raft, raft_small | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should expose the building blocks as well, stuff like
ResidualBlock
,FeatureEncoder
, etc.Should we expose these in a
models.optical_flow.raft
namespace?I remember @datumbox mentioning a few issues when we have both a module and a function with the same name. Ideally, I'd like to keep the
raft()
name for the function builder, but I'm happy to get your thoughts on thisThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Blocking:
See @pmeier's #4867 (comment) around this. We might need to rename this to
raft_large
similar tomobilenet_v3_large
andmobilenet_v3_small
.Question:
Can you talk about the choice to create a second private submodule
_raft
? Why is that?FYI:
If you check existing models, such as resnet and faster_rcnn you will see we don't expose these on the
__all__
or anywhere else. We had discussions about what this means but we didn't have a consensus (public API vs developer API discussion). I would be in favour of not exposing these publicly to be consistent with everywhere else. We should discuss and resolve this once we have time in the roadmap.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the feedback
I wrote it like this out of habit but I had no intention of keeping it this way
Instead of renaming
raft()
toraft_large()
, would it be OK instead to rename the.py
file to something likeraft_core.py
? Orraft_implem.py
?We would have
raft()
,raft_small()
andRAFT
available fromtorchvision.models.optical_flow
and then we would be able to access the other building blocks likeResidualBlock
intorchvision.models.optical_flow.raft_core
-- I would make sure to exclude them from__all__
.Would that be OK?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Though calling it
raft_large
would make it consistent with other small/large models, I can see that on the paper they don't actually call it large. They do useRaft-s
or "small" for the small version, so that's canonical at least. So I understand why you want to avoid naming it like that.Renaming the file should be OK but @pmeier should confirm. Though I'm not sure what the actual name should be. We typically name the files after their algorithm and we let the model builder have extra info on the variant.
@fmassa thoughts on this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I'll use
raft.py
andraft_large()
for now to move forward. We can revisit later if needed, thanks for the inputThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not commenting on the name, because I don't have an opinion on that. We just need to make sure that we don't have an attribute with the same name as a module in the same namespace. Otherwise we can no longer access the module. Thus having
raft.py
anddef raft()
is problematic, butraft.py
andraft_large()
is fine.