Consistency improvements and tests #121
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Changes taking place in this PR:
A new method
inseq.list_feature_attribution_methods()
is introduced to provide a list of all names of available FA methods.Fixed the problematic setting in which layer attribution methods are combined with target-side attribution, now we raise an error to mark this setting as unsupported. This could be supported in the future.
Removed the
GradientShapAttribution
method, since it leads to problematic behavior during tests, producing widely inconsistent results when evaluating attribution on a single input or a batch containing the same input (similar to DIG). The method can be reintroduced with an ad-hoc pull request addressing this issue.Huggingface models don't need the
InterpretableEmbeddings
helper class from Captum to perform attributions since they already support input embeddings for both the encoder and the decoder. In light of this,configure_interpretable_embeddings
andremove_interpretable_embeddings
were moved from theHuggingfaceModel
class to the baseAttributionModel
class, with precisions on how to use them when extending the latter to new frameworks.Added a suite of test to verify consistent behavior for the
model.attribute
core method in the following settings:Type of Change