run_probing_experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e86cf59f",
   "metadata": {},
   "source": [
    "# Probe the internal representation of diffusion model\n",
    "\n",
    "Before running this notebook, please run [create_the_synthetic_dataset.ipynb](create_the_synthetic_dataset.ipynb) to first create the dataset for probing.\n",
    "\n",
    "Also make sure you have switch to the right conda environment (kernel). If you create the conda environment using the provided environment.yml file, the name of that environment should be `diffusion-viz-gpu`. Switch the notebook kernel to `diffusion-viz-gpu` before running the cells below.\n",
    "\n",
    "In addition to running the bash commands below in the jupyter notebook, you may paste them into a terminal for execution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba9a1716",
   "metadata": {},
   "source": [
    "## 1. Probe the Representation of Saliency (Foreground)\n",
    "\n",
    "Once you create the dataset, you could run the code below to probe the representation of salient regions (foreground) in the LDM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "baedc029",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "for step in {0..14}\n",
    "do\n",
    "    python probing_binary_depth.py --layer_name \"transformer_blocks.0.attn1.to_out.0\" --step $step --block_type \"attentions\" --output_dir \"attn1_out\" --postfix \"self_attn_out\"\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "076c9b2c",
   "metadata": {},
   "source": [
    "## 2. Probe the Representation of Depth\n",
    "\n",
    "Run the code below to probe the LDM's representation of inverse depth.\n",
    "\n",
    "`--probe_type \"Linear-no-bias\"` means we are using a linear regressor without the bias term when probing the depth representation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dcd2658",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "\n",
    "for step in {0..14}\n",
    "do\n",
    "    python probing_continuous_depth.py --layer_name \"transformer_blocks.0.attn1.to_out.0\" --step $step --block_type \"attentions\" --output_dir \"attn1_out\" --postfix \"self_attn_out\" --probe_type \"Linear-no-bias\"\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14208438",
   "metadata": {},
   "source": [
    "## 3. Controlled Experiment\n",
    "\n",
    "The code below train the linear classifiers and regressors on a randomized latent diffusion model. The probing performance on the randomized model served as the baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "425016e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "\n",
    "# Baseline experiment for salient object segmentation task (saliency representation)\n",
    "for step in {0..14}\n",
    "do\n",
    "    python probing_binary_depth_controlled.py --layer_name \"transformer_blocks.0.attn1.to_out.0\" --step $step --block_type \"attentions\" --output_dir \"attn1_out\" --postfix \"self_attn_out\"\n",
    "done\n",
    "\n",
    "# Baseline experiment for monocular depth estimation task (depth representation)\n",
    "for step in {0..14}\n",
    "do\n",
    "    python probing_continuous_depth_controlled.py --layer_name \"transformer_blocks.0.attn1.to_out.0\" --step $step --block_type \"attentions\" --output_dir \"attn1_out\" --postfix \"self_attn_out\" --probe_type \"Linear-no-bias\"\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce945773",
   "metadata": {},
   "source": [
    "## 4. Probe Representation of Saliency and Depth in VAE\n",
    "\n",
    "VAE also has one self-attention layer in the bottleneck of its Decoder. The code below explore the saliency and depth representations in VAE's self-attention layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09cd555c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Probe the representation of salient regions in VAE self-attention\n",
    "for step in {0..14}\n",
    "do\n",
    "    python probing_binary_depth_vae.py --layer_name \"proj_attn\" --step $step --block_type \"attentions\" --output_dir \"attn1_out\" --postfix \"self_attn_out\"\n",
    "done\n",
    "\n",
    "# Probe the representation of depth in VAE self-attention\n",
    "for step in {0..14}\n",
    "do\n",
    "    python probing_continuous_depth_vae.py --layer_name \"proj_attn\" --step $step --block_type \"attentions\" --output_dir \"attn1_out\" --postfix \"self_attn_out\" --probe_type \"Linear-no-bias\"\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b4c3043",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}