AccoMontage_inference.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Piano Accompaniment Arrangement with AccoMontage\n",
    "This notebook is a showcase and tutorial for\n",
    " [AccoMontage](https://arxiv.org/abs/2108.11213), a state-of-the-art piano accompanient arrangement system. \n",
    "* The input to the system is a lead sheet of a complete song (i.e., a MIDI file with a melody track and a chord track) with phrase labels. In this notebook, the input can also be a general MIDI with a melody track and any number of other arbiturary tracks.\n",
    "* In this notebook, we assume the phrase labels of the input song to be manually provided.\n",
    "* The output of the system is a piano accompaniment for the whole piece.\n",
    "\n",
    "### Instructions for running\n",
    "* A minimum of 8GB RAM and 4GB GPU memory is requested.\n",
    "* Press ▶️ on the left of each code cell to execute the cell, or hit `SHIFT+ENTER`.\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load dependencies and set up the system\n",
    "### 1.1 Premisies\n",
    "* In the first place, please download `checkpoints.zip` [via this link](https://drive.google.com/file/d/1zQ5xds8oeeAlnn_PK5e0PWNKyM7unUFO/view?usp=sharing) (300MB in total, including model checkpoints and pre-processed data).\n",
    "* After download, please extract the zip file at the working directory. You should now have a `./checkpoints/` folder with relevant pt and npz files inside. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading reference data (texture donors) from POP909. This may takes several seconds of time ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 16/16 [00:06<00:00,  2.53it/s]\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ[\"KMP_DUPLICATE_LIB_OK\"]=\"TRUE\"\n",
    "import datetime\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "from AccoMontage import set_premises, load_lead_sheet, phrase_selection, re_harmonization, matrix2leadsheet\n",
    "\n",
    "phrase_data_dir = 'checkpoints/phrase_data.npz'\n",
    "edge_weights_dir = 'checkpoints/edge_weights.npz'\n",
    "checkpoint_dir = 'checkpoints/model_master_final.pt'\n",
    "pop909_meta_dir = 'checkpoints/pop909_quadraple_meters_index.xlsx'\n",
    "\n",
    "print('Loading reference data (texture donors) from POP909. This may takes several seconds of time ...')\n",
    "model, acc_pool, reference_check, params = pre_liminary = set_premises(phrase_data_dir, edge_weights_dir, checkpoint_dir, pop909_meta_dir)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Put it easy\n",
    "* Let's pack up the whole working pipeline of AccoMontage into a concise function under the name `piano_arrangement`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def piano_arrangement(SONG_NAME, SEGMENTATION, DEMO_ROOT='demo', PREFILTER=None, SPOTLIGHT=None, RANDOMENESS=0, PICK_UP_BEAT=0.0, TEMPO=100, MELODY_TRACK_ID=0):\n",
    "    \"\"\"\n",
    "    Piano accompaniment arrangement using AccoMontage.\n",
    "    :param:`SONG_NAME` (str, requested): the name (i.e., MIDI directory) of the input lead sheet.\n",
    "    :param:`SEGMENTATION` (str, requested): the phrase annotation of the lead sheet. For example, `A8A8B8B8` for a 32-bar lead sheet.\n",
    "    :param:`DEMO_ROOT` (str, optional): the demo root. Default `./demo`.\n",
    "    :param:`PREFILTER` (tuple, optional): pre-filter of accompaniment w.r.t. rhythmic density and voice number. Valid range is [0..4] corresponding to [low .. high]. Default None.\n",
    "    :param:`SPOTLIGHT` (list, optional): spotlight certain reference pieces as the donor of texture. Valid elements can be POP909 song index (int), song name (str, in Simplified Chinese), or artist name (str, also in Simplified Chinese). For detail, see the checktable `./checkpoints/pop909_quadraple_meters_index.xlsx`. Default None.\n",
    "    :param:`RANDOMENESS` (float, optional): the degree of randomness to be introduced to the phrase selection process. Valid range is [0, 1]. Default 0 (i.e., no randomness).\n",
    "    :param:`PICK_UP_BEAT` (float, optional): the number of beats in the pick-up measure, requested if there is one. Half beat (0.5) and quater beats (0.25, 0.75) are also supported. Default 0.\n",
    "    :param:`TEMPO` (int, optional): the BPM to render the accompaniment. Default 100.\n",
    "    :param:`MELODY_TRACK_ID` (int, optional): The index of the melody track in the input MIDI file. Default 0 (i.e., the first track).\n",
    "    \"\"\"\n",
    "\n",
    "    \"\"\"step 0: load lead sheet\"\"\"\n",
    "    lead_sheet, chord_roll, phrase_label = load_lead_sheet(DEMO_ROOT, SONG_NAME, SEGMENTATION, PICK_UP_BEAT, MELODY_TRACK_ID)\n",
    "    #midi_recon = matrix2leadsheet(lead_sheet).write(os.path.join(DEMO_ROOT, SONG_NAME, 'lead_sheet_recon.mid'))\n",
    "\n",
    "    \"\"\"step 1: phrase selection (in search for donors of piano textures)\"\"\"\n",
    "    selection = phrase_selection(lead_sheet, phrase_label, reference_check, acc_pool, *params, PREFILTER, SPOTLIGHT, RANDOMENESS)\n",
    "\n",
    "    \"\"\"step 2: re-harmonization (chord-texture style transfer)\"\"\"\n",
    "    midi = re_harmonization(lead_sheet, chord_roll, phrase_label, *selection, model, acc_pool, TEMPO)\n",
    "\n",
    "    \"\"\"finally, save arrangement result\"\"\"\n",
    "    time = datetime.datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
    "    save_path = os.path.join(DEMO_ROOT, SONG_NAME, f'piano_arrangement_{time}.mid')\n",
    "    midi.write(save_path)\n",
    "    print('Result saved at', save_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Arrangement showcase\n",
    "In the following, we showcase the AccoMontage system with 5 demo pieces. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Auld Lang Syne\n",
    "* This is a well-known folk Scottish music with an `A8B8A8B8` phrase structure.\n",
    "* We set `PREFILTER` for rhythmic density at degree 4 and voice number at degree 2.\n",
    "* This piece has a pick-up measure of 1 beat hence we set `PICK_UP_BEAT`=1.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phrase selection begins: 4 phrases in total. \n",
      "\t Set note density filter: (4, 2).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:13<00:00,  4.43s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reference pieces: ['0: 284_天在下雨我在想你', '1: 272_多想留在你身边', '2: 269_外面的世界', '3: 272_多想留在你身边']\n",
      "Result saved at ./demo\\Auld Lang Syne\\piano_arrangement_20230529_161802.mid\n"
     ]
    }
   ],
   "source": [
    "kwargs = dict({\n",
    "            #compulsory entries\n",
    "            'SONG_NAME': 'Auld Lang Syne',  # song name / directory\n",
    "            'SEGMENTATION': 'A8B8A8B8',     # phrase annotation\n",
    "            #optional pre-filter\n",
    "            'PREFILTER': (4, 2),            # set rhythmic density degree 4 and voice number degree 2\n",
    "            'RANDOMENESS': 0.1,             # introduce a little randomness\n",
    "            #song meta\n",
    "            'PICK_UP_BEAT': 1.0,            # there is a one-beat pick-up measure in this song\n",
    "            'TEMPO': 75,                    # render the arrangement at 75 BPM\n",
    "            })\n",
    "\n",
    "piano_arrangement(**kwargs)     # call piano arrangement process"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Castles in the Air\n",
    "* This is the demo piece in the [AccoMontage paper](https://arxiv.org/abs/2108.11213) with an `A8A8B8B8` phrase structure.\n",
    "* We set `SPOTLIGHT` on `song_322` and `song_346` of POP909 dataset during phrase selection.\n",
    "* We set `PREFILTER` for rhythmic density at degree 4 and voice number at degree 1.\n",
    "* This piece has a pick-up measure of 1 beat hence we set `PICK_UP_BEAT`=1.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phrase selection begins: 4 phrases in total. \n",
      "\t Set note density filter: (4, 1).\n",
      "\t Refer to [322, 346] as much as possible\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:20<00:00,  6.80s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reference pieces: ['0: 322_寂寞沙洲冷', '1: 322_寂寞沙洲冷', '2: 346_平凡之路', '3: 346_平凡之路']\n",
      "Result saved at ./demo\\Castles in the Air\\piano_arrangement_20230529_145318.mid\n"
     ]
    }
   ],
   "source": [
    "kwargs = dict({\n",
    "            #compulsory entries\n",
    "            'SONG_NAME': 'Castles in the Air',  # song name / directory\n",
    "            'SEGMENTATION': 'A8A8B8B8',         # phrase annotation\n",
    "            #optional pre-filter\n",
    "            'SPOTLIGHT': [322, 346],            # spotlight on song_322 and song_346 (of POP909) during phrase selection \n",
    "            'PREFILTER': (4, 1),                # set rhythmic density degree 4 and voice number degree 1\n",
    "            #song meta\n",
    "            'PICK_UP_BEAT': 1.0,                # there is a one-beat pick-up measure in this song\n",
    "            'TEMPO': 120,                       # render the arrangement at 120 BPM\n",
    "            })\n",
    "\n",
    "piano_arrangement(**kwargs)         # call piano arrangement process"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Cuillin Reel\n",
    "* This is the full demo piece in the [deep music analogy (EC2-VAE) paper](https://arxiv.org/abs/1906.03626) with an `A8B8B8`.\n",
    "* We set spotlight on the artist `邓丽君` during phrase selection.\n",
    "* This piece has a pick-up measure of 1 beat hence we set `PICK_UP_BEAT`=1.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phrase selection begins: 3 phrases in total. \n",
      "\t Set note density filter: None.\n",
      "\t Refer to ['邓丽君'] as much as possible\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2/2 [00:12<00:00,  6.43s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reference pieces: ['0: 707_粉红色的回忆', '1: 707_粉红色的回忆', '2: 707_粉红色的回忆']\n",
      "Result saved at ./demo\\Cuillin Reel\\piano_arrangement_20230529_145953.mid\n"
     ]
    }
   ],
   "source": [
    "kwargs = dict({\n",
    "            #compulsory entries\n",
    "            'SONG_NAME': 'Cuillin Reel',  # song name / directory\n",
    "            'SEGMENTATION': 'A8B8B8',     # phrase annotation\n",
    "            #optional pre-filter\n",
    "            'SPOTLIGHT': ['邓丽君'],       # spotlight on the artist 邓丽君 (Teresa Teng) during phrase selection \n",
    "            'RANDOMENESS': 0.2,           # introduce a little randomness\n",
    "            #song meta\n",
    "            'PICK_UP_BEAT': 1.0,          # there is a one-beat pick-up measure in this song\n",
    "            })\n",
    "\n",
    "piano_arrangement(**kwargs)   # call piano arrangement process"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 ECNU Alma Mater Music\n",
    "* This is the Alma Mater Music of East China Normal University. AccoMontage is capable of handling long-term pieces with multiple phrases (16 in this case).\n",
    "* We set `SPOTLIGHT` on the specific piece `小龙人` during phrase selection.\n",
    "* See [this video](https://zhaojw1998.github.io/accomontage_demo) for our performance demo for the 70th-anniversary celebration of ECNU! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phrase selection begins: 16 phrases in total. \n",
      "\t Set note density filter: None.\n",
      "\t Refer to ['小龙人'] as much as possible\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 15/15 [01:10<00:00,  4.69s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reference pieces: ['0: 907_小龙人', '1: 907_小龙人', '2: 907_小龙人', '3: 907_小龙人', '4: 907_小龙人', '5: 907_小龙人', '6: 247_和你一样', '7: 265_在深秋', '8: 907_小龙人', '9: 907_小龙人', '10: 907_小龙人', '11: 907_小龙人', '12: 907_小龙人', '13: 907_小龙人', '14: 830_酒醉的探戈2001', '15: 265_在深秋']\n",
      "Result saved at ./demo\\ECNU Alma Mater Music\\piano_arrangement_20230529_150642.mid\n"
     ]
    }
   ],
   "source": [
    "kwargs = dict({\n",
    "            #compulsory entries\n",
    "            'SONG_NAME': 'ECNU Alma Mater Music',                   # song name / directory\n",
    "            'SEGMENTATION': 'A8A8B8B8C8D8E4F6A8A8B8B8C8D8E4F6',     # phrase annotation\n",
    "            #optional pre-filter\n",
    "            'SPOTLIGHT': ['小龙人'],        # spotlight on the piece 小龙人 (Little Dragon Man) during phrase selection \n",
    "            'RANDOMENESS': 0.1,             # introduce a little randomness\n",
    "            #song meta\n",
    "            'PICK_UP_BEAT': 0,              # there is no pick-up measure\n",
    "            'TEMPO': 106,                   # render the arrangement at 106 BPM\n",
    "            })\n",
    "\n",
    "piano_arrangement(**kwargs)     # call piano arrangement process"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Let It Be\n",
    "* This is a well-known pop song by John Lennon and Paul McCartney. AccoMontage is capable of handling whole pieces with intro, interlude, and outro.\n",
    "* In this case, our input is a general multi-track MIDI and the melody is on the 10-th track. Hence we set `MELODY_TRACK_ID`=9 (counting from 0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phrase selection begins: 11 phrases in total. \n",
      "\t Set note density filter: (3, 4).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:23<00:00,  2.30s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reference pieces: ['0: 027_一曲红尘', '1: 413_我', '2: 645_电台情歌', '3: 510_春泥', '4: 425_我们都是好孩子', '5: 908_小乌龟', '6: 684_突然好想你', '7: 014_一个人的精彩', '8: 683_空城', '9: 747_芊芊', '10: 378_忽然好想念']\n",
      "Result saved at ./demo\\Let It Be\\piano_arrangement_20230529_151022.mid\n"
     ]
    }
   ],
   "source": [
    "kwargs = dict({\n",
    "            #compulsory entries\n",
    "            'SONG_NAME': 'Let It Be',                   # song name / directory\n",
    "            'SEGMENTATION': 'i4A8B4A8B8x12B4A8B8B4x2',  # phrase annotation\n",
    "            #optional pre-filter\n",
    "            'PREFILTER': (3, 4),        # set rhythmic density degree 3 and voice number degree 4\n",
    "            'RANDOMENESS': 0.1,         # introduce a little randomness\n",
    "            #song meta\n",
    "            'MELODY_TRACK_ID': 9,       # melody is on the 10-th track of the input MIDI file\n",
    "            'TEMPO': 75,                # render the arrangement at 75 BPM\n",
    "            })\n",
    "\n",
    "piano_arrangement(**kwargs)     # call piano arrangement process"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Test with your own music\n",
    "If you wish to use AccoMontage to arrange piano accompaniments for your own piece, please go for the following steps:\n",
    "1. Rename your MIDI file as `lead sheet.mid`.\n",
    "2. Put the renamed file to `./demo/SONG_NAME`, where `SONG_NAME` is the name of your piece.\n",
    "3. Specify the phrase `SEGMENTATION` for your piece, together with `PICK_UP_BEAT`, `MELODY_TRACK_ID` and `TEMPO` if necessory.\n",
    "4. Set up `SPOTLIGHT`, `PREFILTER`, and `RANDOMENESS` accoridng to you preference.\n",
    "5. Put these argumants to the `piano_arrangement` function as shown above, and your arrangement is ready at `./demo/SONG_NAME/piano_arrangement.mid`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torch1.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}