This is the official repository of the paper An Evaluation Framework for Legal Document Summarization accepted in the Poster and Demo Session in the 13th Edition of its Language Resources and Evaluation Conference (LREC 2022)
Authors - Ankan Mullick†, Abhilash Nandy†, Manav Nitin Kapadnis†, Sohan Patnaik, R Raghav, Roshni Kar
A law practitioner has to go through numerous lengthy legal case proceedings for their practices of various categories, such as land dispute, corruption, etc. Hence, it is important to summarize these documents, and ensure that summaries contain phrases with intent matching the category of the case. To the best of our knowledge, there is no evaluation metric that evaluates a summary based on its intent. We propose an automated intent-based summarization metric, which shows a better agreement with human evaluation as compared to other automated metrics like BLEU, ROUGE-L etc. in terms of human satisfaction. We also curate a dataset by annotating intent phrases in legal documents, and show a proof of concept as to how this system can be automated.
- Paper
- Repository Organisation
- Files Descriptions
- Demonstration
- Poster
- Citation
- Author Contact Details
Our paper can be found here.
├── JointBert_input_dataset_indian/
| ├── train/
| ├── label
| ├── seq.in
| └── seq.out
| ├── dev/
| ├── label
| ├── seq.in
| └── seq.out
| ├── test/
| ├── label
| ├── seq.in
| └── seq.out
| ├── intent_label.txt
| └── slot_label.txt
├── dataset/
| ├── australian_data/
| ├── aus_phrases/
| ├── aus_text/
| ├── aus_data_statistics.csv
| └── aus_labels.csv
| ├── indian_data/
| ├── ind_phrases_2/
| ├── ind_text/
| ├── ind_dataset_statistics.csv
| └── aus_labels.csv
├── demo_app_code/
├── summarization_models/
| ├── Abstractive summarization/
| ├── Bert/
| ├── Graphical_Model/
| ├── LetSum Model/
| ├── Case_Summarizer.ipynb
| ├── Graphical_Model.ipynb
| ├── legal-led.ipynb
| ├── requirements.txt
├── README.md- JointBert_input_dataset_indian - This folder contains the indian dataset formatted to JointBERT input dataset format. Please refer to JointBERT repository for more details on how to convert data to their format.
- dataset - Contains the main indian and australian data with NER labels as well as sentence classification labels. The folder named as "ind_phrases_2" and "aus_phrases" contains the extracted intent phrases (indian were manually annotated, australian were automatically extracted using JointBERT). The files "ind_labels.csv" and "aus_labels.csv" contains the different types of labels used for our experiments.
- demo_app_code - This folder contains the code for the demonstration hosted on the website
- summarization_models - This folder contains the implementation of different types of summarization models used for our experimentations.
The code for the demostration can be found in the "demo_app_code/" folder in this repository.
Additionally, the live demo is deployed on this Demo Link
Note: Please wait for 2 minutes, if the website isn't showing the demo website (since the app is dynamically being hosted in livetime when you click on the link).
If you want to use this code in your work, please add the following citation -
@misc{https://doi.org/10.48550/arxiv.2205.08478,
doi = {10.48550/ARXIV.2205.08478},
url = {https://arxiv.org/abs/2205.08478},
author = {Mullick, Ankan and Nandy, Abhilash and Kapadnis, Manav Nitin and Patnaik, Sohan and Raghav, R and Kar, Roshni},
keywords = {Computation and Language (cs.CL), Information Retrieval (cs.IR), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {An Evaluation Framework for Legal Document Summarization},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
Manav Nitin Kapadnis
Abhilash Nandy
Note: † - equal contribution