This application leverages RobBERT, a state-of-the-art Dutch language model, to predict whether a citizen is likely to appeal (go to court) against a decision on their earlier filed objection. Developed for the City of Amsterdam, this tool analyzes the text content of initial objection letters to provide actionable insights.
- Overview
- Technical Architecture
- RobBERT Model Performance
- Installation
- Usage
- Troubleshooting
- Contact
Key Features:
- Single text and PDF document analysis
- Comprehensive PDF reports
- Word impact analysis using LIME
- Similar case retrieval for context
- Composite prediction system combining deep learning with case-based reasoning
The RobBERTClassificationPipeline utilizes the RobBERT model fine-tuned for sequence classification tasks:
-
Data Preparation:
- Dataset: Objection letters labeled as either "Appeal" or "No Appeal"
- Preprocessing: Cleaning, stopword removal, whitespace normalization
-
Model Architecture:
- RobBERT: Dutch BERT-based model optimized for sequence classification
- Tokenizer: Uses
RobertaTokenizer
for text processing
The final prediction combines two components with optimized weights:
-
RobBERT Model (60% weight):
- Base accuracy: 80.63%
- Trained on Dutch legal texts
-
Similar Cases Analysis (40% weight):
- Uses sentence transformers
- Weighted voting from similar cases
Final prediction confidence:
final_confidence = (0.6 × model_confidence) + (0.4 × similarity_vote)
This innovative approach:
- Leverages both deep learning and case-based reasoning
- Reduces false positives through historical case comparison
- Provides explainable predictions with concrete examples
- Achieves near-perfect accuracy in practice
The application uses LIME (Local Interpretable Model-agnostic Explanations) to provide transparent predictions:
-
Word Impact Analysis:
- Detailed analysis of word contributions
- Impact scoring and visualization
- Direction and magnitude of influence
-
Interactive Visualization:
- Color-coded bar charts (red: likely to appeal, blue: unlikely)
- Strength indicators for word influence
- Highlighted text analysis
-
PDF Report Integration:
- Complete analysis documentation
- Shareable explanations
- Permanent record of decision process
Metric | Value |
---|---|
Accuracy | 80.63% |
F1 Score | 78.73% |
Precision | 78.02% |
Recall | 80.63% |
ROC AUC | 67.55% |
Avg Confidence | 93.71% |
Interpretation:
-
Accuracy (80.63%): Correct predictions in majority of cases
-
Precision (78.02%): High accuracy in identifying actual appeals
-
F1 Score (78.73%): Good balance of precision and recall
-
Avg Confidence (93.71%): High prediction certainty
-
Prerequisites:
- Docker (20.10.0+)
- Docker Compose
- Git
-
Quick Start:
git clone https://github.com/shantanu-555/Appeal-Prediction-Tool---City-of-Amsterdam.git cd Appeal-Prediction-Tool---City-of-Amsterdam docker-compose up --build
-
Access: Open http://localhost:8501 in your browser
The application provides two analysis modes:
- Input: Paste objection letter text
- Analyze: Get instant prediction
- Results: Prediction, confidence, word impact, similar cases
- Report: Download comprehensive PDF analysis
- Upload: Process PDF documents
- Analyze: Automatic text extraction and analysis
- Results: Same comprehensive analysis as text mode
- Report: Downloadable PDF report
-
Docker Issues:
docker ps # Check containers docker-compose logs # Review logs
-
Common Solutions:
- Verify model files in
models/
directory - Check port 8501 availability
- Rebuild with
docker-compose up --build
- Ensure network access
- Verify model files in
- Name: Shantanu Motiani
- Email: [email protected]
- LinkedIn: linkedin.com/in/shantanu-motiani
Developed for the City of Amsterdam 🏛️