Dynamically load model code from the Hub #13467

sgugger · 2021-09-07T18:57:32Z

What does this PR do?

This PR allows a user to upload a modeling file along their model weights, and then lets the Auto classes load that model. The API is still a bit manual for now, it will be smoothed in follow-up PRs.

In terms of user of the model, the only thing to add is the flag trust_remote_code=True when using an auto class:

from transformers import AutoModel

model = AutoModel.from_pretrained("sgugger/test_dynamic_model", trust_remote_code=True)

This will load a FakeModel as defined in this file.

For the person uploading the model there is a tiny bit more work. First save the code of the model (with all necessary imports) in a single python file, for instance modeling.py. In the rest of this PR description, I'll use MODELING_FILE to represent the name of that file without the suffix (so in my example of modeling.py, MODELING_FILE is "modeling"). That file needs to be in the repository you upload.

Second, add a filed auto_map to the configuration of the model you want to upload. This needs to be a dictionary with the name of auto classes as keys, and the full name of the corresponding modeling class as values. By full name, I mean MODELING_FILE.CLASS_NAME. For instance modeling.FakeModel for a class FakeModel defined in the modeling.py module. Here is an example:

config.auto_map = {"AutoModel": "modeling.FakeModel"}

This needs to be done before the model is saved, so that when doing the model.save_pretrained call, the config is saved with that field. Once this is done, push everything to the Hub, and you should be good!

flozi00 · 2021-09-07T20:26:54Z

Forgive me for interjecting and asking, but does this allow custom modules to be uploaded to modelhub and loaded and run with the official Transformers library ? just to be sure.

If that's the case, do you plan to add some code scanning ? I think on some cases where this would be an security vulnerability for remote code execution

sgugger · 2021-09-08T16:21:27Z

The PR requires an extra argument in the call to from_pretrained: allow_custom_model=True to execute the code from the hub. A user should not pass along that flag without having scanned the code of the repo to make sure they trust it.

flozi00 · 2021-09-08T17:53:31Z

Great, thanks for clearing this

LysandreJik

I'm having a few failures so I'll wait until you have updated the description of the PR before finishing my review

src/transformers/models/auto/auto_factory.py

src/transformers/models/auto/dynamic.py

LysandreJik · 2021-09-16T20:58:28Z

src/transformers/models/auto/dynamic.py

+        init_path.touch()
+
+
+def check_imports(filename):


That's extremely useful!

LysandreJik · 2021-09-16T21:08:13Z

src/transformers/models/auto/auto_factory.py

-
-        if type(config) in cls._model_mapping.keys():
+        if hasattr(config, "auto_map") and cls.__name__ in config.auto_map:
+            if not trust_remote_code:


I wonder if it wouldn't be good practice to force a revision here too. Either through raising an error (best) or through logging a warning (ok): if a public repository becomes trendy, it wouldn't be surprising to see the user update the code and for it to, at best break systems running the code, at worst run arbitrary code through torch.load

That would make the API harder to use, no?

Then let's go with a warning?

After discussing with @LysandreJik, we will need to remember to parse this auto_map attribute in the hub so we know which models are custom code models

LysandreJik

This is in very good shape! I tried to break it as hard as I could but couldn't manage to do so!

Following our IRL discussions, the following point stood out: if defining a model that has a significantly different architecture than the original model (for example using a BertConfig as the model configuration, and then defining a totally different model to BERT), then the model can still be loaded in traditional BERT architectures; this is quite an edge case here, but we should keep that in mind as new configurations are allowed -> we should then make sure that models error out when loaded in architectures that don't support them.

Fantastic that it works out of the box for private models too. This should enable a myriad of use-cases.

Looks good to me, eagerly awaiting the next update enabling configuration/tokenizer creation!

LysandreJik · 2021-09-17T20:58:52Z

src/transformers/models/auto/auto_factory.py

-
-        if type(config) in cls._model_mapping.keys():
+        if hasattr(config, "auto_map") and cls.__name__ in config.auto_map:
+            if not trust_remote_code:


Then let's go with a warning?

Co-authored-by: Lysandre Debut <[email protected]>

LysandreJik

LGTM, feel free to merge!

src/transformers/models/auto/auto_factory.py

Co-authored-by: Lysandre Debut <[email protected]>

p-christ · 2022-01-24T12:36:27Z

hi, quick question about this PR:

It lets people upload a custom model to the hub great, but it seems that in order to call that model using the API it still requires the model to be using one of the provided Pipelines? Is there any way of also allowing a custom pipeline when calling the custom models?

sgugger · 2022-01-24T12:41:46Z

Custom pipelines is something we will add support for, it's on the roadmap, but for now you can't use the custom model in the API indeed.

sgugger requested a review from LysandreJik September 7, 2021 18:57

sgugger force-pushed the dynamic_model branch from c195c2a to de17667 Compare September 7, 2021 18:58

sgugger mentioned this pull request Sep 13, 2021

The new impl for CONFIG_MAPPING prevents users from adding any custom models #13522

Closed

4 tasks

LysandreJik reviewed Sep 16, 2021

View reviewed changes

sgugger changed the title ~~[WiP] Dynamically load model code from the Hub~~ Dynamically load model code from the Hub Sep 16, 2021

LysandreJik approved these changes Sep 17, 2021

View reviewed changes

sgugger force-pushed the dynamic_model branch from 753fca4 to ffa125f Compare September 20, 2021 15:28

sgugger and others added 9 commits September 20, 2021 11:30

Dynamic model

2d85635

Use defensive flag

1bdb6ce

Style

8b7d17a

Doc and arg rename

16316b3

Arg rename

36cd741

Add tests

9c44152

Apply suggestions from code review

07c7374

Co-authored-by: Lysandre Debut <[email protected]>

Apply suggestions from code review

c3e78c9

Co-authored-by: Lysandre Debut <[email protected]>

Address review comments

ffa125f

LysandreJik approved these changes Sep 20, 2021

View reviewed changes

src/transformers/models/auto/auto_factory.py Outdated Show resolved Hide resolved

Apply suggestions from code review

321f457

Co-authored-by: Lysandre Debut <[email protected]>

sgugger merged commit 002a078 into master Sep 20, 2021

sgugger deleted the dynamic_model branch September 20, 2021 17:59

sgugger mentioned this pull request Sep 22, 2021

[RFC] adding Tensor and Pipeline Parallelism to transformers #13690

Closed

LysandreJik mentioned this pull request Oct 6, 2021

[Feature request] Introduce GenericTransformer to ease deployment of custom models to the Hub #13311

Closed

NielsRogge mentioned this pull request Oct 11, 2021

Add model to HuggingFace rmokady/CLIP_prefix_caption#4

Open

benderama3 mentioned this pull request Jan 19, 2022

Copy of the custom modeling file when saving a model #15224

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamically load model code from the Hub #13467

Dynamically load model code from the Hub #13467

sgugger commented Sep 7, 2021 •

edited

Loading

flozi00 commented Sep 7, 2021

sgugger commented Sep 8, 2021 •

edited

Loading

flozi00 commented Sep 8, 2021

LysandreJik left a comment

LysandreJik Sep 16, 2021

LysandreJik Sep 16, 2021

sgugger Sep 16, 2021

LysandreJik Sep 17, 2021

julien-c Sep 20, 2021

LysandreJik left a comment

LysandreJik Sep 17, 2021

LysandreJik left a comment

p-christ commented Jan 24, 2022

sgugger commented Jan 24, 2022

Dynamically load model code from the Hub #13467

Dynamically load model code from the Hub #13467

Conversation

sgugger commented Sep 7, 2021 • edited Loading

What does this PR do?

flozi00 commented Sep 7, 2021

sgugger commented Sep 8, 2021 • edited Loading

flozi00 commented Sep 8, 2021

LysandreJik left a comment

Choose a reason for hiding this comment

LysandreJik Sep 16, 2021

Choose a reason for hiding this comment

LysandreJik Sep 16, 2021

Choose a reason for hiding this comment

sgugger Sep 16, 2021

Choose a reason for hiding this comment

LysandreJik Sep 17, 2021

Choose a reason for hiding this comment

julien-c Sep 20, 2021

Choose a reason for hiding this comment

LysandreJik left a comment

Choose a reason for hiding this comment

LysandreJik Sep 17, 2021

Choose a reason for hiding this comment

LysandreJik left a comment

Choose a reason for hiding this comment

p-christ commented Jan 24, 2022

sgugger commented Jan 24, 2022

sgugger commented Sep 7, 2021 •

edited

Loading

sgugger commented Sep 8, 2021 •

edited

Loading