Reduce by 2 the memory requirement in `generate()` 🔥🔥🔥#30536

Merged

ArthurZucker merged 34 commits intohuggingface:mainfrom Cyrilvallez:cache-efficiency

Jun 5, 2024

+327-206

Commits on May 23, 2024

Fix contrastive_search for new cache structure, and improve performance by removing inneficient torch.stack(torch.split(x, top_k, dim=0))
Cyrilvallez
committed
Fix _contrastive_search for non-standard cache using ellipsis slicing
Cyrilvallez
committed
Fix all outputs.logits memory leaks for all decoding strategies!
Cyrilvallez
committed
Fix small error in _contrastive_search()
Cyrilvallez
committed
Make all necessary change and revert for the new class
Cyrilvallez
committed
Apply coding style
Cyrilvallez
committed
Remove pipes in type hints for compatibility
Cyrilvallez
committed
correct type hint
Cyrilvallez
committed
apply style
Cyrilvallez
committed
Use DynamicCache by default and solve conflicts
Cyrilvallez
committed
Fix rebase issues
Cyrilvallez
committed
Add _supports_dynamic_cache_class in models for models that support DynamicCache but not other caches to make DynamicCache the default for more models
Cyrilvallez
committed
Create generation config to return legacy format by default, or to choose not to
Cyrilvallez
committed
style
Cyrilvallez
committed
Fix case when use_cache is False
Cyrilvallez
committed
Remove default DynamicCache in assiste_decoding if assistant_model does not support it + fix _seen_tokens when cropping cache
Cyrilvallez
committed
Update prepare_inputs_for_generation() for case with empty DynamicCache
Cyrilvallez
committed
Correct return of args in _assisted_decoding
Cyrilvallez
committed
Remove EfficientDynamicCache as it is no longer needed
Cyrilvallez
committed
Correct mistake in generation config
Cyrilvallez
committed
Move cache logic of assisted decoding to AssistedCandidateGenerator.__init__
Cyrilvallez
committed
change DynamicCache function names from "split" to "batch_split" for readability + apply coding style
Cyrilvallez
committed
Remove _supports_dynamic_cache_class attribute after rebase
Cyrilvallez
committed
Correct missing line lost in conflict resolution during rebasing
Cyrilvallez
committed
Add special case for Jamba
Cyrilvallez
committed
Fix jamba test
Cyrilvallez
committed
Coding style
Cyrilvallez
committed
coding style
Cyrilvallez
committed
Correct missing import in rebasing
Cyrilvallez
committed
Simplify _validate_model_kwargs based on removal of _supports_dynamic_cache attribute
Cyrilvallez
committed
Simplify code paths in _contrastive_search
Cyrilvallez
committed
coding style
Cyrilvallez
committed

Commits on Jun 3, 2024

Update docstrings of cache methods
Cyrilvallez
committed
Update prepare_inputs_for_generation() -> past_key_values are always Cache objects
Cyrilvallez
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce by 2 the memory requirement in `generate()` 🔥🔥🔥#30536

Commits on May 23, 2024

Fix contrastive_search for new cache structure, and improve performance by removing inneficient torch.stack(torch.split(x, top_k, dim=0))

Fix _contrastive_search for non-standard cache using ellipsis slicing

Fix all outputs.logits memory leaks for all decoding strategies!

Fix small error in _contrastive_search()

Make all necessary change and revert for the new class

Apply coding style

Remove pipes in type hints for compatibility

correct type hint

apply style

Use DynamicCache by default and solve conflicts

Fix rebase issues

Add `_supports_dynamic_cache_class` in models for models that support DynamicCache but not other caches to make DynamicCache the default for more models

Create generation config to return legacy format by default, or to choose not to

style

Fix case when use_cache is False

Remove default DynamicCache in assiste_decoding if assistant_model does not support it + fix _seen_tokens when cropping cache

Update prepare_inputs_for_generation() for case with empty DynamicCache

Correct return of args in _assisted_decoding

Remove EfficientDynamicCache as it is no longer needed

Correct mistake in generation config

Move cache logic of assisted decoding to AssistedCandidateGenerator.init

change DynamicCache function names from "split" to "batch_split" for readability + apply coding style

Remove `_supports_dynamic_cache_class` attribute after rebase

Correct missing line lost in conflict resolution during rebasing

Add special case for Jamba

Fix jamba test

Coding style

coding style

Correct missing import in rebasing

Simplify _validate_model_kwargs based on removal of _supports_dynamic_cache attribute

Simplify code paths in _contrastive_search

coding style

Commits on Jun 3, 2024

Update docstrings of cache methods

Update prepare_inputs_for_generation() -> past_key_values are always Cache objects