Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consistency improvements and tests #121

Merged
merged 2 commits into from
Mar 2, 2022
Merged

Consistency improvements and tests #121

merged 2 commits into from
Mar 2, 2022

Conversation

gsarti
Copy link
Member

@gsarti gsarti commented Mar 2, 2022

Description

Changes taking place in this PR:

  • A new method inseq.list_feature_attribution_methods() is introduced to provide a list of all names of available FA methods.

  • Fixed the problematic setting in which layer attribution methods are combined with target-side attribution, now we raise an error to mark this setting as unsupported. This could be supported in the future.

  • Removed the GradientShapAttribution method, since it leads to problematic behavior during tests, producing widely inconsistent results when evaluating attribution on a single input or a batch containing the same input (similar to DIG). The method can be reintroduced with an ad-hoc pull request addressing this issue.

  • Huggingface models don't need the InterpretableEmbeddings helper class from Captum to perform attributions since they already support input embeddings for both the encoder and the decoder. In light of this, configure_interpretable_embeddings and remove_interpretable_embeddings were moved from the HuggingfaceModel class to the base AttributionModel class, with precisions on how to use them when extending the latter to new frameworks.

  • Added a suite of test to verify consistent behavior for the model.attribute core method in the following settings:

    • Testing with and without GPU should produce the same results
    • Testing the same example inside a batch or as a stand-alone input should produce the same results.
    • Test that the attribution process completes successfully for all available methods and options (adding probabilities, target-side attribution, computing convergence deltas, force decoding with custom references)

Type of Change

  • 🔧 Bug fix (non-breaking change which fixes an issue)

@gsarti gsarti merged commit ac07669 into main Mar 2, 2022
@gsarti gsarti deleted the layer-attr-fixes branch March 2, 2022 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant