GlobalCausalAnalysis.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Global Causal Analysis (GCA)\n",
    "We perform experiments, where we use GCA with a dataset of high-level features $Z$ (constructed using `gca.data.prepare_data()`) and predicted labels $\\hat{Y}$ of a finetuned `DistilRoBERTa-base` model (trained and applied with `gca.model.train_and_apply()`).\n",
    "\n",
    "## 0. Installing requirements\n",
    "We first install the supplied `gca` package used for reproducing the experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install -e ."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If inferring features for yourself, do not forget to download spaCy model `en_core_web_sm`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!python3 -m spacy download en_core_web_sm"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment this line if you are on Google Colab or Linux to download the `NRC Sentiment Emotion Lexicon (EmoLex)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget -nc http://saifmohammad.com/WebDocs/Lexicons/NRC-Suite-of-Sentiment-Emotion-Lexicons.zip"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load the data and define labels\n",
    "To avoid retraining of the model (which takes approximately 1.5 hours on a GPU), we have provided the dataset with high-level features $Z$ and the predicted label $\\hat{Y}$ in `'data/go_emotions_xai-distilroberta-base.csv'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from gca.data import get_data\n",
    "\n",
    "# Download data with PRED_label column\n",
    "dataset = get_data('data/go_emotions_xai-distilroberta-base.csv')\n",
    "\n",
    "# Take subset of\n",
    "columns_to_select = ['all_lower', 'flesch_grade', 'is_active', 'subreddit',\n",
    "                     'len_chr', 'len_tok', 'len_snt',\n",
    "                     'has_name', 'has_emoji', 'has_religion',\n",
    "                     'NRC_anger', 'NRC_anticipation', 'NRC_disgust', 'NRC_fear',\n",
    "                     'NRC_joy', 'NRC_sadness', 'NRC_surprise', 'NRC_trust',\n",
    "                     'NRC_valence', 'NRC_arousal', 'NRC_dominance',\n",
    "                     'male_words', 'female_words', 'non-binary_words',\n",
    "                     'PRED_label']\n",
    "df = dataset.with_format('pandas')[:][columns_to_select]\n",
    "\n",
    "# One-hot encode labels (for class-wise contrastive explanation)\n",
    "df = pd.concat([df, pd.get_dummies(df['PRED_label'], prefix='PRED')], axis=1)\n",
    "\n",
    "# Names of labels\n",
    "LABELS = ['label', 'positive', 'neutral', 'negative', 'ambiguous']\n",
    "\n",
    "# Place to hold all results\n",
    "results = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Experiments\n",
    "### 2a. Task-related features\n",
    "Use GCA to estimate a global explanatory graph $\\mathcal{P}$ over $V=(Z_{task}, y)$ for $y \\in \\{\\hat{Y}, \\hat{Y}_{positive}, \\hat{Y}_{negative}, \\hat{Y}_{ambiguous}, \\hat{Y}_{neutral}\\}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gca.data.tasks import TASK_FEATURES\n",
    "TASK_FEATURES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from gca import generate_and_evaluate\n",
    "\n",
    "results['task'] = [generate_and_evaluate(df[TASK_FEATURES + [f'PRED_{label}']],\n",
    "                                         continuous=['NRC_valence', 'NRC_arousal', 'NRC_dominance'],\n",
    "                                         n_trials=0)\n",
    "                   for label in LABELS]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2b. Robustness-related features\n",
    "Use GCA to estimate a global explanatory graph $\\mathcal{P}$ over $V=(Z_{robust}, y)$ for $y \\in \\{\\hat{Y}, \\hat{Y}_{positive}, \\hat{Y}_{negative}, \\hat{Y}_{ambiguous}, \\hat{Y}_{neutral}\\}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gca.data.tasks import ROBUSTNESS_FEATURES\n",
    "ROBUSTNESS_FEATURES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from gca import generate_and_evaluate\n",
    "\n",
    "results['robustness'] = [generate_and_evaluate(df[ROBUSTNESS_FEATURES + [f'PRED_{label}']], n_trials=0)\n",
    "                         for label in LABELS]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2c. Fairness-related features\n",
    "Use GCA to estimate a global explanatory graph $\\mathcal{P}$ over $V=(Z_{fair}, y)$ for $y \\in \\{\\hat{Y}, \\hat{Y}_{positive}, \\hat{Y}_{negative}, \\hat{Y}_{ambiguous}, \\hat{Y}_{neutral}\\}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gca.data.tasks import FAIRNESS_FEATURES\n",
    "FAIRNESS_FEATURES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gca import generate_and_evaluate\n",
    "\n",
    "results['fairness'] = [generate_and_evaluate(df[FAIRNESS_FEATURES + [f'PRED_{label}']], n_trials=0)\n",
    "                       for label in LABELS]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2d. Combined (all aspects)\n",
    "Use GCA to estimate a global explanatory graph $\\mathcal{P}$ over $V=(Z, y)$ for $y \\in \\{\\hat{Y}, \\hat{Y}_{positive}, \\hat{Y}_{negative}, \\hat{Y}_{ambiguous}, \\hat{Y}_{neutral}\\}$, where $Z = Z_{task} \\cup Z_{robust} \\cup Z_{fair}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FEATURES = TASK_FEATURES + ROBUSTNESS_FEATURES + FAIRNESS_FEATURES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from gca import generate_and_evaluate\n",
    "\n",
    "results['all'] = [generate_and_evaluate(df[FEATURES + [f'PRED_{label}']],\n",
    "                                        continuous=['NRC_valence', 'NRC_arousal', 'NRC_dominance'],\n",
    "                                        color=True,\n",
    "                                        depth=2,\n",
    "                                        n_trials=0)\n",
    "                  for label in LABELS]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.+ Save results locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "from datetime import datetime\n",
    "\n",
    "with open(f'results-{datetime.utcnow().strftime(\"%Y-%m-%d-%H%M\")}.pickle', 'wb') as f:\n",
    "   pickle.dump(results, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('all_all.svg', 'wb') as f:\n",
    "    f.write(results['all'][0].svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('task_positive.svg', 'wb') as f:\n",
    "    f.write(results['task'][1].svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('robustness_neutral.svg', 'wb') as f:\n",
    "    f.write(results['robustness'][2].svg)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Results\n",
    "Tabulate the results of the experiments conducted above."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the label distributions of $\\hat{Y}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['PRED_label'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def latexify_df(df, **kwargs):\n",
    "    return df.reset_index().rename(columns={'index': 'Aspect'}).to_latex(index=False, **kwargs).replace('Aspect', '\\\\textit{Aspect}')\n",
    "\n",
    "def results_to_table(to_select,\n",
    "                     results=results,\n",
    "                     round_to=None,\n",
    "                     percentage=False,\n",
    "                     columns=LABELS,\n",
    "                     to_latex=True):\n",
    "    df = pd.DataFrame.from_dict(\n",
    "        {aspect: [to_select(res) for res in aspect_results]\n",
    "         for aspect, aspect_results in results.items()},\n",
    "        orient='index',\n",
    "        columns=columns\n",
    "    )\n",
    "    df = df.sort_index()\n",
    "\n",
    "    if percentage:\n",
    "        df *= 100\n",
    "    if round_to:\n",
    "        df = df.round(round_to)\n",
    "\n",
    "    return latexify_df(df) if to_latex else df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 $Z$-fidelity\n",
    "$Z$-fidelity estimates how predictive the selected high-level features $Z$ are of behavior $\\hat{Y}$. We report the $F_1$-score (because of non-equal label distributions), but other metrics can be used instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Z-Fidelity (f1-score)\n",
    "print(results_to_table(lambda res: res.z_f1, percentage=True, round_to=2))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Structural fit & stability\n",
    "The relative MVEE indicates the structural fit and stability of the generated explanatory graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Relative MVEE (PHD with strategy MVEE)\n",
    "print(results_to_table(lambda res: res.mvee_relative, round_to=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mean relative MVEE, SD\n",
    "df_mvee_relative = results_to_table(lambda res: res.mvee_relative, to_latex=False).stack()\n",
    "\n",
    "print(f'mean: {df_mvee_relative.mean().round(3)} | SD: {df_mvee_relative.std().round(3)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Absolute MVEE (PHD with strategy MVEE)\n",
    "df_abs = results_to_table(lambda res: res.mvee, to_latex=False)\n",
    "df_abs.insert(0, 'nodes', results_to_table(lambda res: res.n_features + 1, to_latex=False)['label'])\n",
    "print(latexify_df(df_abs))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.+ Time\n",
    "We also report the time taken to generate the explanatory graphs, for each aspect-label combination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Time (seconds)\n",
    "print(results_to_table(lambda res: res.elapsed_time, round_to=5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mean time\n",
    "results_to_table(lambda res: res.elapsed_time, to_latex=False).mean(axis=1).round(2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}