metadata_file_handling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Metadata file handling for PlacesAudio and Places205\n",
    "\n",
    "The audio-visual embeddings models of Harwath et al. that use PlacesAudio:\n",
    "\n",
    "- [NIPS 2016 model](https://papers.nips.cc/paper/6186-unsupervised-learning-of-spoken-language-with-visual-context.pdf)\n",
    "- [ACL 2017 model](https://arxiv.org/pdf/1701.07481.pdf)\n",
    "- [DAVEnet model](https://github.com/dharwath/DAVEnet-pytorch)\n",
    "\n",
    "In this notebook, we process the metadata files of PlacesAudio in different ways. The processes are:\n",
    " 1. Create small samples of our own for making local test runs on desktop.\n",
    " 2. Create .json files from the lists provided with PlacesAudio (under `metadata/lists`) . The lists define subsets of the full PlacesAudio dataset that were used in the NIPS and ACL papers \n",
    "\n",
    "First we make necessary imports and define some utility functions. The utility functions assume the following json file structure:\n",
    "\n",
    "    {\n",
    "        \"image_base_path\": \"/path/to/images/\",\n",
    "        \"audio_base_path\": \"/path/to/audio/\",\n",
    "        \"data\": [\n",
    "            {\n",
    "                \"uttid\": \"A1A6D2RDPGVX5F-GSUN_C4E9B966E3F4AF2A83AF01C8ACFB47BB\",\n",
    "                \"speaker\": \"A1A6D2RDPGVX5F\",\n",
    "                \"asr_text\": \"a wooden table with a lobster in the center and plates are around the tape\",\n",
    "                \"wav\": \"wavs/13/utterance_374111.wav\",\n",
    "                \"image\": \"r/restaurant/gsun_c4e9b966e3f4af2a83af01c8acfb47bb.jpg\"\n",
    "            },\n",
    "            ...\n",
    "            {\n",
    "                \"uttid\": \"A13G469LJFEIYZ-GSUN_48C633C668C194469102D3B8E0BDE81C\",\n",
    "                \"speaker\": \"A13G469LJFEIYZ\",\n",
    "                \"asr_text\": \"a woman sitting in a small cluttered office\",\n",
    "                \"wav\": \"wavs/375/utterance_286274.wav\",\n",
    "                \"image\": \"h/home_office/gsun_48c633c668c194469102d3b8e0bde81c.jpg\"\n",
    "            }\n",
    "        ]\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# Copy base paths from a given json\n",
    "def copy_base_paths(input_dict):\n",
    "    \n",
    "    all_keys = input_dict.keys()\n",
    "    keys = [key for key in (k for k in all_keys if k not in 'data')]\n",
    "    return { key: input_dict[key] for key in keys }\n",
    "\n",
    "# Copies a sample of data from given json file to an output json file\n",
    "def copy_sample(input_path, output_path, sample_size=1000):\n",
    "    \n",
    "    print(\"Copying a sample of size {:d} from {}\".format(sample_size, input_path))\n",
    "    with open(input_path) as f:\n",
    "        inputs = json.load(f)\n",
    "\n",
    "    # Copy the image and audio paths at the top of the json first...\n",
    "    outputs = copy_base_paths(inputs)\n",
    "    # ... and then copy a sample of the data.\n",
    "    outputs['data'] = inputs['data'][0:sample_size]\n",
    "    \n",
    "    print(\"Writing output to {}\".format(output_path))\n",
    "    with open(output_path, 'w') as f:\n",
    "        json.dump(outputs, f, indent=4)\n",
    "    print(\"Finished.\\n\")\n",
    "\n",
    "# Copies image paths from input and writes them to the output one path per line\n",
    "def copy_image_paths(input_path, output_path):\n",
    "    \n",
    "    with open(input_path) as f:\n",
    "        inputs = json.load(f)\n",
    "    \n",
    "    print(\"Writing image paths from {}\\n to {}\".format(input_path, output_path))\n",
    "    base_path = inputs['image_base_path']\n",
    "    with open(output_path, 'w') as f:\n",
    "        for item in inputs['data']:\n",
    "            f.write(base_path + item['image'] + \"\\n\")\n",
    "    print(\"Finished.\\n\")\n",
    "            \n",
    "# Create an output json file from an input json file using a list of utterance ids\n",
    "def json_from_uttid_list(uttid_path, input_path, output_path):\n",
    "    \n",
    "    print(\"Using utterance ids from {}\".format(uttid_path))\n",
    "    with open(uttid_path) as f:\n",
    "        uttids = [line.rstrip() for line in f]\n",
    "    print(\"Number of uttids: {:d}\".format(len(uttids)))\n",
    "    \n",
    "    print(\"Input json is {}\".format(input_path))\n",
    "    with open(input_path) as f:\n",
    "        inputs = json.load(f)\n",
    "    \n",
    "    print(\"Copying data that matches utterance ids...\")\n",
    "    outputs = copy_base_paths(inputs)\n",
    "    outputs['data'] = [x for x in inputs['data'] if x['uttid'] in uttids]\n",
    "    \n",
    "    print(\"Writing output to {}\".format(output_path))\n",
    "    with open(output_path, 'w') as f:\n",
    "        json.dump(outputs, f, indent=4)\n",
    "    print(\"Done, number of elements in output data is: {}\\n\".format(len(outputs['data'])))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Small sample\n",
    "First, take a subset of image/caption pairs from `train.json` and write it to a separate file. Then, write the image paths to a text file. This is for collecting the images from Triton, where the images are stored in the computer vision groups database folders. Audio is stored in `teamwork` corpus folder, so the same is not necessary for the wavs. After that, we do the same for the validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying a sample of size 1000 from /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/train.json\n",
      "Writing output to /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/train_1k.json\n",
      "Finished.\n",
      "Writing image paths from /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/train_1k.json\n",
      " to /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/train_image_paths.txt\n",
      "Finished.\n",
      "Writing image paths from /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/val.json\n",
      " to /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/val_image_paths.txt\n",
      "Finished.\n"
     ]
    }
   ],
   "source": [
    "data_path = \"/teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/\"\n",
    "train_json = \"metadata/train.json\"\n",
    "val_json = \"metadata/val.json\"\n",
    "val_image_path = \"metadata/val_image_paths.txt\"\n",
    "sample_size = 1000\n",
    "\n",
    "train_1k = \"metadata/train_1k.json\"\n",
    "train_1k_image_path = \"metadata/train_image_paths.txt\"\n",
    "\n",
    "copy_sample(data_path + train_json, data_path + train_1k, sample_size=1000)\n",
    "copy_image_paths(data_path + train_1k, data_path + train_1k_image_path)\n",
    "copy_image_paths(data_path + val_json, data_path + val_image_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON from an utterance ID list\n",
    "\n",
    "We recreate the NIPS and ACL datasets using the utterance id lists provided and the full data json file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using utterance ids from /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/lists/nips_2016_train_uttids\n",
      "Number of uttids: 116111\n",
      "Input json is /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/train.json\n",
      "Copying data that matches utterance ids...\n",
      "Writing output to /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/nips_train.json\n",
      "Done, number of elements in output data is: 115162\n",
      "Using utterance ids from /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/lists/nips_2016_val_uttids\n",
      "Number of uttids: 1000\n",
      "Input json is /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/train.json\n",
      "Copying data that matches utterance ids...\n",
      "Writing output to /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/nips_val.json\n",
      "Done, number of elements in output data is: 993\n",
      "Using utterance ids from /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/lists/acl_2017_train_uttids\n",
      "Number of uttids: 214585\n",
      "Input json is /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/train.json\n",
      "Copying data that matches utterance ids...\n",
      "Writing output to /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/acl_train.json\n",
      "Done, number of elements in output data is: 213018\n",
      "Using utterance ids from /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/lists/acl_2017_val_uttids\n",
      "Number of uttids: 1000\n",
      "Input json is /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/train.json\n",
      "Copying data that matches utterance ids...\n",
      "Writing output to /teamwork/t40511_asr/c/PlacesAudio400k/PlacesAudio_400k_distro/metadata/acl_val.json\n",
      "Done, number of elements in output data is: 990\n"
     ]
    }
   ],
   "source": [
    "train_json = \"metadata/train.json\"\n",
    "uttid_lists_and_output = [(\"metadata/lists/nips_2016_train_uttids\", \"metadata/nips_train.json\"), \n",
    "                          (\"metadata/lists/nips_2016_val_uttids\", \"metadata/nips_val.json\"),\n",
    "                          (\"metadata/lists/acl_2017_train_uttids\", \"metadata/acl_train.json\"),\n",
    "                          (\"metadata/lists/acl_2017_val_uttids\", \"metadata/acl_val.json\")]\n",
    "\n",
    "for (uttid_list, output_file) in uttid_lists_and_output:\n",
    "    json_from_uttid_list(data_path + uttid_list, \n",
    "                         data_path + train_json,\n",
    "                         data_path + output_file)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}