Skip to content

Commit

Permalink
Remove duplicate values from pytorch model json (microsoft#600)
Browse files Browse the repository at this point in the history
## Describe your changes
Pytorch model adds values from the hf model config into model attributes
to make them easily accessible. However, this makes the model json
messy/confusing with values in model_attributes that are already in
`model_config`. This PR removes information from model_attributes that
is already in the hf_config.

## Checklist before requesting a review
- [ ] Add unit tests for this change.
- [ ] Make sure all tests can pass.
- [ ] Update documents if necessary.
- [ ] Format your code by running `pre-commit run --all-files`
- [ ] Is this a user-facing change? If yes, give a description of this
change to be included in the release notes.

## (Optional) Issue link
  • Loading branch information
jambayk authored Sep 27, 2023
1 parent 51271f8 commit ae0091c
Showing 1 changed file with 10 additions and 5 deletions.
15 changes: 10 additions & 5 deletions olive/model/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -724,11 +724,16 @@ def to_json(self, check_object: bool = False):
"hf_config": self.hf_config,
}
)
# convert script_dir and model_script to string
# the original config has them as serialized ResourcePath
for resource_name in ["script_dir", "model_script"]:
if self.resource_paths[resource_name]:
config["config"][resource_name] = self.get_resource(resource_name)
# clean up redundant information in model_attributes
config["config"].pop("model_attributes", None)
# using a copy of self.model_attributes since config["config"]["model_attributes"] is already
# serialized and might not match self.model_attributes
model_attributes = deepcopy(self.model_attributes)
if model_attributes and self.hf_config:
for key, value in self.get_hf_model_config().to_dict().items():
if key in model_attributes and model_attributes[key] == value:
del model_attributes[key]
config["config"]["model_attributes"] = model_attributes or None
return serialize_to_json(config, check_object)


Expand Down

0 comments on commit ae0091c

Please sign in to comment.