-
Notifications
You must be signed in to change notification settings - Fork 170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Batch input activations #33
Commits on Jan 4, 2021
-
- Created a __call__ method in lm to allow running the model without …
…generation. - Activation collection and processing now supports batched inputs [still work in progress] - Updated existing tests to support the new shape of the activations tensor with a batch dimension - Creating a mockGPT to more properly test lm functionality [work in progress]
Configuration menu - View commit details
-
Copy full SHA for 65c6516 - Browse repository at this point
Copy the full SHA 65c6516View commit details
Commits on Jan 19, 2021
-
- Initial BERT/DistilBERT support now works in __call__ for activatio…
…n collection. No support for saliency or other features yet. - Activation collection and processing now supports batched inputs [still work in progress] - Creating a mockGPT to more properly test lm functionality [work in progress]
Configuration menu - View commit details
-
Copy full SHA for 43a9450 - Browse repository at this point
Copy the full SHA 43a9450View commit details
Commits on Jan 20, 2021
-
- BERT/Distilbert now work for activation collection, NMF reduction, …
…and NMF visualization. Tested with a batch of one input. - Activation collection and processing now supports batched inputs [still work in progress] - Creating a mockGPT to more properly test lm functionality [work in progress]
Configuration menu - View commit details
-
Copy full SHA for 02d20d6 - Browse repository at this point
Copy the full SHA 02d20d6View commit details
Commits on Feb 11, 2021
-
- BERT/Distilbert now work for activation collection, NMF reduction, …
…and NMF visualization. Tested with a batch of one input. - Activation collection and processing now supports batched inputs [still work in progress] - Creating a mockGPT to more properly test lm functionality [work in progress]
Configuration menu - View commit details
-
Copy full SHA for fb47b7b - Browse repository at this point
Copy the full SHA fb47b7bView commit details -
Merge remote-tracking branch 'origin/main' into batch-input-activations
# Conflicts: # setup.py # src/ecco/__init__.py # src/ecco/lm.py # tests/lm_test.py # tests/output_test.py
Configuration menu - View commit details
-
Copy full SHA for 583e0af - Browse repository at this point
Copy the full SHA 583e0afView commit details -
Configuration menu - View commit details
-
Copy full SHA for eb88b7d - Browse repository at this point
Copy the full SHA eb88b7dView commit details
Commits on Feb 13, 2021
-
- carved out model configurations to model-config. To enable more mod…
…els to be supported and define their key layers (embeddings for saliency, and FFNN for activations) in YAML. - Defined a batch of initial models in model-config.yaml. The top pytorch models and two dummy models for testing purposes. - Started writing tests in lm_test.py that act as integration tests with HF Transformers. These use tiny GPT/BERT models to ensure functionality works between ecco and the models. This is to automated tests that were previously done manually in jupyter notebooks before release. - Switched docs from sphinx to mkdoc - Wrote a couple of docs pages, set up a skeleton for navigation. Expanding on docstrings - Added a 'verbose' parameter to LM to suppress printing tokens during generation. - Removed unused MockGPT code.
Configuration menu - View commit details
-
Copy full SHA for 0bace9d - Browse repository at this point
Copy the full SHA 0bace9dView commit details
Commits on Feb 14, 2021
-
- When specifying the layer name to collect activations for, we switc…
…hed from layer name, to using a regex pattern. This was we can be more specific and not collect other layers by mistake (example: both attention and ffnn layers of bert contain "output.dense" layers. - Starting documenting lm as a module. Looks messy for now. Will see how to clean it up later. - Can now force LM to use CPU even if GPU is available. That is done by setting the "gpu" parameter to False in ecco.from_pretrained(). - __call__ now automatically moves input token to GPU if the model is on GPU. - In _get_activations_hook, extracting the layer number is now done more precisely using regex.
Configuration menu - View commit details
-
Copy full SHA for dab2111 - Browse repository at this point
Copy the full SHA dab2111View commit details -
- To better support new models, AutoModel is now used on everything that doesn't have "GPT2" in its model name. Stopgap for now.
Configuration menu - View commit details
-
Copy full SHA for 181feb6 - Browse repository at this point
Copy the full SHA 181feb6View commit details
Commits on Feb 18, 2021
-
- Fixed NMF.explore() issue where the resulting 'factors' parameter w…
…as not consistent between outputs of "generate()" and "__call__". - generate() now produces the same shape of 'token_ids' and 'tokens' produced by __call__. dims: (batch, position). OutputSeq should likely verify the dims of inputs. - __call__ now returns 'token_ids' without the 'input_ids' dict key. Consumers shouldn't know about the distinction. - Added tests for nmf pipeline for both dummy bert and GPT. - More docs
Configuration menu - View commit details
-
Copy full SHA for 7901faa - Browse repository at this point
Copy the full SHA 7901faaView commit details
Commits on Feb 20, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 5b37c34 - Browse repository at this point
Copy the full SHA 5b37c34View commit details -
Configuration menu - View commit details
-
Copy full SHA for c11b079 - Browse repository at this point
Copy the full SHA c11b079View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9f72b64 - Browse repository at this point
Copy the full SHA 9f72b64View commit details -
Configuration menu - View commit details
-
Copy full SHA for 218c5e4 - Browse repository at this point
Copy the full SHA 218c5e4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 73f7d77 - Browse repository at this point
Copy the full SHA 73f7d77View commit details -
Configuration menu - View commit details
-
Copy full SHA for 10ebce7 - Browse repository at this point
Copy the full SHA 10ebce7View commit details -
Configuration menu - View commit details
-
Copy full SHA for c79ff19 - Browse repository at this point
Copy the full SHA c79ff19View commit details -
Configuration menu - View commit details
-
Copy full SHA for fc0c78d - Browse repository at this point
Copy the full SHA fc0c78dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 50f1bc2 - Browse repository at this point
Copy the full SHA 50f1bc2View commit details
Commits on Feb 21, 2021
-
Configuration menu - View commit details
-
Copy full SHA for c8d6686 - Browse repository at this point
Copy the full SHA c8d6686View commit details -
Configuration menu - View commit details
-
Copy full SHA for 124c7f4 - Browse repository at this point
Copy the full SHA 124c7f4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2abe152 - Browse repository at this point
Copy the full SHA 2abe152View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4e68c64 - Browse repository at this point
Copy the full SHA 4e68c64View commit details
Commits on Feb 22, 2021
-
- Fixed rankings() and saliency() to adapt to the new (batch, positio…
…n_ dimensions of token_ids and tokens - created index of docs - Added simple css to the docs template
Configuration menu - View commit details
-
Copy full SHA for 7ce6e47 - Browse repository at this point
Copy the full SHA 7ce6e47View commit details -
Configuration menu - View commit details
-
Copy full SHA for dcff20f - Browse repository at this point
Copy the full SHA dcff20fView commit details
Commits on Feb 24, 2021
-
- Updating readme. Switching to markdown and adding the content from …
…the docs homepage
Configuration menu - View commit details
-
Copy full SHA for a8d20c8 - Browse repository at this point
Copy the full SHA a8d20c8View commit details -
Configuration menu - View commit details
-
Copy full SHA for c78a435 - Browse repository at this point
Copy the full SHA c78a435View commit details -
Configuration menu - View commit details
-
Copy full SHA for 86f048b - Browse repository at this point
Copy the full SHA 86f048bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 86cdb9e - Browse repository at this point
Copy the full SHA 86cdb9eView commit details