PartOfSpeechTagging.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Problem Set 2: Sequence Labeling\n",
    "=======================\n",
    "\n",
    "- This problem set focuses on sequence labeling with Hidden Markov models and Deep Learning Models.\n",
    "- The target domain is part-of-speech tagging on English and Norwegian from the Universal Dependencies dataset\n",
    "\n",
    "You will:\n",
    "- Do some basic preprocessing of the data\n",
    "- Build a naive classifier that tags each word with its most common tag\n",
    "- Implement a `Viterbi` Tagger using `Hidden Markov Model` in PyTorch\n",
    "- Build a `Bi-LSTM` deep learning model using PyTorch\n",
    "- Build a `Bi-LSTM_CRF` model using the above components (`Viterbi` and `Bi-LSTM`) \n",
    "- then implement techniques to improve your classifier and compete on Kaggle.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0. Setup\n",
    "\n",
    "In order to develop this assignment, you will need [python 3.6](https://www.python.org/downloads/) and the following libraries. Most if not all of these are part of [anaconda](https://www.continuum.io/downloads), so a good starting point would be to install that.\n",
    "\n",
    "- [jupyter](http://jupyter.readthedocs.org/en/latest/install.html)\n",
    "- [numpy](https://docs.scipy.org/doc/numpy/user/install.html)\n",
    "- [matplotlib](http://matplotlib.org/users/installing.html)\n",
    "- [nosetests](https://nose.readthedocs.org/en/latest/)\n",
    "- [pandas](http://pandas.pydata.org/) Dataframes\n",
    "\n",
    "Here is some help on installing packages in python: https://packaging.python.org/installing/. You can use ```pip --user``` to install locally without sudo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About this assignment\n",
    "\n",
    "- This is a Jupyter notebook. You can execute cell blocks by pressing control-enter.\n",
    "- Most of your coding will be in the python source files in the directory ```gtnlplib```.\n",
    "- The directory ```tests``` contains unit tests that will be used to grade your assignment, using ```nosetests```. You should run them as you work on the assignment to see that you're on the right track. You are free to look at their source code, if that helps -- though most of the relevant code is also here in this notebook. Learn more about running unit tests at http://pythontesting.net/framework/nose/nose-introduction/\n",
    "- You may want to add more tests, but that is completely optional. \n",
    "- **To submit this assignment, run the script ```make-submission.sh```, and submit the tarball ```pset2-submission.tgz``` on Canvas.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "from importlib import reload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My Python version\n",
      "python: 3.6.3 |Anaconda, Inc.| (default, Dec  5 2017, 17:30:25) \n",
      "[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]\n"
     ]
    }
   ],
   "source": [
    "print('My Python version')\n",
    "\n",
    "print('python: {}'.format(sys.version))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nose\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "from torch import optim\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My library versions\n",
      "pandas: 0.22.0\n",
      "numpy: 1.14.0\n",
      "matplotlib: 2.1.1\n",
      "nose: 1.3.7\n",
      "torch: 0.3.0.post4\n"
     ]
    }
   ],
   "source": [
    "print('My library versions')\n",
    "\n",
    "print('pandas: {}'.format(pd.__version__))\n",
    "print('numpy: {}'.format(np.__version__))\n",
    "print('matplotlib: {}'.format(matplotlib.__version__))\n",
    "print('nose: {}'.format(nose.__version__))\n",
    "print('torch: {}'.format(torch.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test whether your libraries are the right version, run:\n",
    "\n",
    "`nosetests tests/test_environment.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.000s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "# use ! to run shell commands in notebook\n",
    "! nosetests tests/test_environment.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict, Counter\n",
    "\n",
    "# importing all the necessary files from gtnlplib\n",
    "from gtnlplib import constants, preproc, most_common, clf_base, evaluation\n",
    "from gtnlplib import scorer, tagger_base, naive_bayes, hmm, viterbi, bilstm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Data Processing\n",
    "===================\n",
    "The part-of-speech tags are defined on the [universal dependencies website](http://universaldependencies.org/en/pos/index.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(constants);\n",
    "## Define the file names\n",
    "TRAIN_FILE = constants.TRAIN_FILE\n",
    "DEV_FILE = constants.DEV_FILE\n",
    "TEST_FILE = constants.TEST_FILE \n",
    "TEST_FILE_HIDDEN = constants.TEST_FILE_UNLABELED\n",
    "NR_TRAIN_FILE = constants.NR_TRAIN_FILE\n",
    "NR_DEV_FILE = constants.NR_DEV_FILE\n",
    "NR_TEST_FILE = constants.NR_TEST_FILE\n",
    "NR_TEST_FILE_HIDDEN = constants.NR_TEST_FILE_UNLABELED\n",
    "# change the constant to test_file_unlabeled for release"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Here is demo code for using the function `conll_seq_generator(...)`. \n",
    "- The default value for max_insts is `1000000` indicating the num. of instances: and this should be enough for our dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ADJ', 'ADP', 'ADV', 'AUX', 'CONJ', 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']\n"
     ]
    }
   ],
   "source": [
    "## Demo\n",
    "all_tags = set([])\n",
    "for (words, tags) in preproc.conll_seq_generator(TRAIN_FILE):\n",
    "    for tag in tags:\n",
    "        all_tags.add(tag)\n",
    "all_tags = sorted(all_tags)\n",
    "print (all_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 1.1**: Counting words per tag. (* 1 point *)\n",
    "\n",
    "Implement the function `get_tag_word_counts` in `most_common.py`: The function should calculate the number of occurrences of all words for each tag.\n",
    "\n",
    "- **Input**: filename for data file, to be passed as argument to `preproc.conll_seq_generation`\n",
    "- **Output**: dict of counters, where keys are tags\n",
    "- **Tests**: ```test_most_common.py: test_get_top_noun_tags(),test_get_top_verb_tags()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.178s\n",
      "\n",
      "OK\n",
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.173s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "reload(most_common);\n",
    "! nosetests tests/test_most_common.py:test_get_top_noun_tags\n",
    "! nosetests tests/test_most_common.py:test_get_top_verb_tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROPN [('Bush', 100), ('US', 81), ('Al', 78)]\n",
      "PUNCT [(',', 1616), ('.', 1588), ('\"', 383)]\n",
      "ADJ [('other', 46), ('many', 41), ('Indian', 35)]\n",
      "NOUN [('people', 53), ('time', 48), ('world', 46)]\n",
      "VERB [('is', 335), ('was', 128), ('have', 110)]\n",
      "DET [('the', 1926), ('a', 650), ('The', 216)]\n",
      "ADP [('of', 887), ('in', 738), ('to', 380)]\n",
      "AUX [('have', 139), ('was', 126), ('has', 124)]\n",
      "PRON [('I', 251), ('it', 208), ('he', 131)]\n",
      "PART [('to', 542), (\"'s\", 218), ('not', 172)]\n",
      "SCONJ [('that', 304), ('if', 56), ('as', 47)]\n",
      "NUM [('one', 52), ('two', 28), ('2001', 17)]\n",
      "ADV [('also', 63), ('now', 54), ('when', 53)]\n",
      "CONJ [('and', 932), ('or', 127), ('but', 88)]\n",
      "X [('1', 3), ('2', 3), ('3', 3)]\n",
      "INTJ [('Please', 15), ('please', 3), ('Well', 3)]\n",
      "SYM [('$', 20), ('-', 13), ('/', 7)]\n"
     ]
    }
   ],
   "source": [
    "# this block uses your code to find the three most common words per tag\n",
    "counters = most_common.get_tag_word_counts(TRAIN_FILE)\n",
    "for tag,tag_ctr in counters.items():\n",
    "    print (tag,tag_ctr.most_common(3))\n",
    "# PROPN [('Bush', 100), ('US', 81), ('Al', 78)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 2. Tagging as classification \n",
    "\n",
    "Now you will implement part-of-speech tagging via classification.\n",
    "\n",
    "Tagging quality is evaluated using evalTagger, which takes three arguments:\n",
    "- a tagger, which is a **function** taking a list of words and a tagset as arguments and returns the predicted tags for the words\n",
    "- an output filename\n",
    "- a test file\n",
    "\n",
    "You will want to use lambda expressions to create the first argument for the `eval_tagger(..)` function, as shown below.\n",
    "Here's how it works. I provide a tagger that labels everything as a noun."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.257s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "reload(tagger_base);\n",
    "! nosetests tests/test_classifier_tagger.py:test_classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1527613022274944\n"
     ]
    }
   ],
   "source": [
    "# here is a tagger that just tags everything as a noun\n",
    "noun_tagger = lambda words, alltags : ['NOUN' for word in words]\n",
    "\n",
    "confusion = tagger_base.eval_tagger(noun_tagger,'nouns.preds',all_tags=all_tags)\n",
    "print (scorer.accuracy(confusion))\n",
    "# most_common.get_noun_weights()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "** Deliverable 2.1 ** Classification-based tagging. (* 0.5 points *)\n",
    "\n",
    "Now do the same thing as above, but building your tagger *as a classifier.* To do this, implement `make_classifier_tagger()` in `tagger_base.py`. \n",
    "\n",
    "- **Input**: defaultdict of weights\n",
    "- **Output**: return a function that takes in (list of word tokens, list of all possible tags) $\\rightarrow$ tags for each word\n",
    "\n",
    "The function that you output should create the base-features for each token (**use the OFFSET and the TOKEN itself as base-features**)  and then use your `clf_base.predict()` function from pset 1. You are free to edit the `clf_base.predict()` function if you don't think you got it right in pset 1.\n",
    "- **Tests**: ```test_classifier_tagger.py:test_classifier()```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(float, {('NOUN', '**OFFSET**'): 1.0})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reload(tagger_base);\n",
    "reload(clf_base);\n",
    "from gtnlplib.constants import OFFSET #OFFSET token is for each tag\n",
    "most_common.get_noun_weights()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now create a tagger with weights that predict every token to be a NOUN. \n",
    "The function `get_noun_weights` is already implemented for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier_noun_tagger = tagger_base.make_classifier_tagger(most_common.get_noun_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1527613022274944\n"
     ]
    }
   ],
   "source": [
    "confusion = tagger_base.eval_tagger(classifier_noun_tagger,'all-nouns.preds',all_tags=all_tags)\n",
    "print (scorer.accuracy(confusion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Deliverable 2.2** Tagging words by their most common tag. (* 0.5 points *)\n",
    "\n",
    "Now build a classifier tagger that tags each word with its most common tag in the training set. To do this, implement `get_most_common_word_weights` in `most_common.py`.  \n",
    "\n",
    "- **Input**: training file\n",
    "\n",
    "- **Output**: defaultdict of weights\n",
    "\n",
    "This function should return a set weights such that each word should get the tag that is most frequently associated with it in the training data. If the word does not appear in the training data, the weights should be set so that the tagger outputs the **most common tag** in the training data. For the out of vocabulary words, you need to think on how to set the weights so that you tag them by the most common tag.\n",
    "\n",
    "- **Tests**: ```test_classifier.py:test_mcc_tagger_output(), test_mcc_tagger_accuracy()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.354s\n",
      "\n",
      "OK\n",
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 4.342s\n",
      "\n",
      "OK\n",
      "defaultdict(<class 'float'>, {('PROPN', 'Al'): 78.0, ('PUNCT', '-'): 312.0, ('PROPN', 'Zaman'): 3.0, ('PUNCT', ':'): 111.0, ('ADJ', 'American'): 16.0, ('NOUN', 'forces'): 17.0, ('VERB', 'killed'): 10.0, ('PROPN', 'Shaikh'): 4.0, ('PROPN', 'Abdullah'): 1.0, ('PROPN', 'al'): 32.0, ('PROPN', 'Ani'): 1.0, ('PUNCT', ','): 1616.0, ('DET', 'the'): 1926.0, ('NOUN', 'preacher'): 2.0, ('ADP', 'at'): 97.0, ('NOUN', 'mosque'): 2.0, ('ADP', 'in'): 738.0, ('NOUN', 'town'): 7.0, ('ADP', 'of'): 887.0, ('PROPN', 'Qaim'): 2.0, ('ADP', 'near'): 6.0, ('ADJ', 'Syrian'): 4.0, ('NOUN', 'border'): 10.0, ('PUNCT', '.'): 1588.0, ('PUNCT', '['): 6.0, ('DET', 'This'): 18.0, ('NOUN', 'killing'): 2.0, ('DET', 'a'): 650.0, ('ADJ', 'respected'): 1.0, ('NOUN', 'cleric'): 1.0, ('AUX', 'will'): 98.0, ('AUX', 'be'): 89.0, ('VERB', 'causing'): 1.0, ('PRON', 'us'): 19.0, ('NOUN', 'trouble'): 3.0, ('ADP', 'for'): 233.0, ('NOUN', 'years'): 32.0, ('PART', 'to'): 542.0, ('VERB', 'come'): 17.0, ('PUNCT', ']'): 6.0, ('PROPN', 'DPA'): 1.0, ('ADJ', 'Iraqi'): 26.0, ('NOUN', 'authorities'): 9.0, ('VERB', 'announced'): 4.0, ('SCONJ', 'that'): 304.0, ('PRON', 'they'): 124.0, ('AUX', 'had'): 60.0, ('VERB', 'busted'): 1.0, ('ADP', 'up'): 49.0, ('NUM', '3'): 8.0, ('ADJ', 'terrorist'): 11.0, ('NOUN', 'cells'): 5.0, ('VERB', 'operating'): 4.0, ('PROPN', 'Baghdad'): 11.0, ('NUM', 'Two'): 9.0, ('PRON', 'them'): 60.0, ('AUX', 'were'): 48.0, ('AUX', 'being'): 19.0, ('VERB', 'run'): 7.0, ('ADP', 'by'): 188.0, ('NUM', '2'): 8.0, ('NOUN', 'officials'): 20.0, ('PROPN', 'Ministry'): 3.0, ('PROPN', 'Interior'): 1.0, ('PUNCT', '!'): 14.0, ('DET', 'The'): 216.0, ('PROPN', 'MoI'): 1.0, ('PROPN', 'Iraq'): 66.0, ('VERB', 'is'): 335.0, ('ADJ', 'equivalent'): 2.0, ('ADP', 'to'): 380.0, ('PROPN', 'US'): 81.0, ('PROPN', 'FBI'): 19.0, ('ADV', 'so'): 37.0, ('PRON', 'this'): 44.0, ('AUX', 'would'): 96.0, ('VERB', 'be'): 104.0, ('SCONJ', 'like'): 6.0, ('VERB', 'having'): 6.0, ('PROPN', 'J.'): 1.0, ('PROPN', 'Edgar'): 1.0, ('PROPN', 'Hoover'): 1.0, ('ADV', 'unwittingly'): 1.0, ('VERB', 'employ'): 1.0, ('ADJ', 'high'): 10.0, ('NOUN', 'level'): 8.0, ('NOUN', 'members'): 11.0, ('PROPN', 'Weathermen'): 1.0, ('NOUN', 'bombers'): 2.0, ('ADV', 'back'): 15.0, ('NOUN', '1960s'): 1.0, ('ADJ', 'third'): 6.0, ('AUX', 'was'): 126.0, ('NOUN', 'head'): 15.0, ('DET', 'an'): 127.0, ('NOUN', 'investment'): 5.0, ('NOUN', 'firm'): 2.0, ('PRON', 'You'): 21.0, ('VERB', 'wonder'): 4.0, ('SCONJ', 'if'): 56.0, ('PRON', 'he'): 131.0, ('VERB', 'manipulating'): 1.0, ('NOUN', 'market'): 5.0, ('ADP', 'with'): 187.0, ('PRON', 'his'): 114.0, ('NOUN', 'bombing'): 5.0, ('NOUN', 'targets'): 10.0, ('PROPN', 'Ghazaliyah'): 1.0, ('CONJ', 'and'): 932.0, ('PROPN', 'Jihad'): 15.0, ('NOUN', 'districts'): 1.0, ('NOUN', 'capital'): 5.0, ('SCONJ', 'Although'): 8.0, ('NOUN', 'announcement'): 2.0, ('ADV', 'probably'): 12.0, ('VERB', 'made'): 29.0, ('VERB', 'show'): 7.0, ('NOUN', 'progress'): 6.0, ('SCONJ', 'in'): 30.0, ('VERB', 'identifying'): 1.0, ('VERB', 'breaking'): 1.0, ('NOUN', 'terror'): 17.0, ('PRON', 'I'): 251.0, ('AUX', 'do'): 56.0, ('PART', \"n't\"): 78.0, ('VERB', 'find'): 13.0, ('NOUN', 'news'): 12.0, ('PROPN', 'Baathists'): 2.0, ('VERB', 'continue'): 4.0, ('VERB', 'penetrate'): 1.0, ('NOUN', 'government'): 30.0, ('ADV', 'very'): 31.0, ('ADJ', 'hopeful'): 1.0, ('PRON', 'It'): 82.0, ('VERB', 'reminds'): 1.0, ('PRON', 'me'): 44.0, ('ADV', 'too'): 19.0, ('ADV', 'much'): 16.0, ('PROPN', 'ARVN'): 1.0, ('NOUN', 'officers'): 9.0, ('PRON', 'who'): 103.0, ('ADV', 'secretly'): 2.0, ('VERB', 'working'): 12.0, ('ADJ', 'other'): 46.0, ('NOUN', 'side'): 3.0, ('PROPN', 'Vietnam'): 7.0, ('NOUN', 'Guerrillas'): 3.0, ('NOUN', 'member'): 10.0, ('PROPN', 'Kurdistan'): 1.0, ('PROPN', 'Democratic'): 2.0, ('PROPN', 'Party'): 5.0, ('SCONJ', 'after'): 16.0, ('VERB', 'kidnapping'): 1.0, ('PRON', 'him'): 46.0, ('PROPN', 'Mosul'): 3.0, ('NOUN', 'police'): 8.0, ('NOUN', 'commander'): 4.0, ('PROPN', 'Ninevah'): 1.0, ('PROPN', 'Province'): 4.0, ('NOUN', 'bombings'): 7.0, ('VERB', 'declined'): 3.0, ('NUM', '80'): 5.0, ('NOUN', 'percent'): 2.0, ('SCONJ', 'whereas'): 1.0, ('PRON', 'there'): 44.0, ('VERB', 'been'): 32.0, ('ADJ', 'big'): 3.0, ('NOUN', 'jump'): 1.0, ('NOUN', 'number'): 18.0, ('NOUN', 'kidnappings'): 1.0, ('ADP', 'On'): 11.0, ('PROPN', 'Wednesday'): 7.0, ('NOUN', 'guerrillas'): 4.0, ('VERB', 'kidnapped'): 1.0, ('ADJ', 'cosmetic'): 1.0, ('NOUN', 'surgeon'): 1.0, ('NOUN', 'wife'): 2.0, ('SCONJ', 'while'): 21.0, ('VERB', 'were'): 47.0, ('ADP', 'on'): 193.0, ('PRON', 'their'): 88.0, ('NOUN', 'way'): 33.0, ('ADV', 'home'): 4.0, ('ADP', 'In'): 60.0, ('PROPN', 'Suwayrah'): 1.0, ('PROPN', 'Kut'): 2.0, ('NUM', 'two'): 28.0, ('NOUN', 'car'): 2.0, ('NOUN', 'bombs'): 10.0, ('VERB', 'discovered'): 3.0, ('SCONJ', 'before'): 14.0, ('AUX', 'could'): 31.0, ('VERB', 'detonated'): 1.0, ('PUNCT', '('): 135.0, ('ADJ', 'southeastern'): 1.0, ('VERB', 'has'): 35.0, ('ADV', 'overwhelmingly'): 2.0, ('ADJ', 'Shiite'): 9.0, ('NOUN', 'population'): 5.0, ('VERB', 'are'): 107.0, ('NOUN', 'lookout'): 1.0, ('PROPN', 'Baathist'): 1.0, ('NOUN', 'saboteurs'): 1.0, ('ADV', 'willingly'): 1.0, ('VERB', 'turn'): 6.0, ('NOUN', 'willingness'): 2.0, ('ADJ', 'main'): 7.0, ('NOUN', 'difference'): 2.0, ('NOUN', 'south'): 3.0, ('SCONJ', 'as'): 47.0, ('VERB', 'opposed'): 2.0, ('NOUN', 'center'): 2.0, ('NOUN', 'north'): 3.0, ('NOUN', 'country'): 23.0, ('PUNCT', ')'): 134.0, ('PROPN', 'Kadhim'): 2.0, ('PROPN', 'Talal'): 1.0, ('PROPN', 'Husain'): 1.0, ('ADJ', 'assistant'): 1.0, ('NOUN', 'dean'): 1.0, ('PROPN', 'School'): 4.0, ('PROPN', 'Education'): 2.0, ('PROPN', 'Mustansiriyah'): 1.0, ('PROPN', 'University'): 3.0, ('VERB', 'assassinated'): 3.0, ('NOUN', 'driver'): 5.0, ('PROPN', 'Salikh'): 1.0, ('NOUN', 'district'): 1.0, ('NOUN', 'engineer'): 1.0, ('PROPN', 'Asi'): 1.0, ('PROPN', 'Ali'): 3.0, ('ADP', 'from'): 146.0, ('PROPN', 'Tikrit'): 2.0, ('PRON', 'They'): 35.0, ('ADV', 'also'): 63.0, ('PROPN', 'Hamid'): 2.0, ('PROPN', \"'Akkab\"): 1.0, ('NOUN', 'clan'): 5.0, ('NOUN', 'elder'): 2.0, ('NOUN', 'branch'): 1.0, ('PROPN', 'Dulaim'): 3.0, ('NOUN', 'tribe'): 6.0, ('PRON', 'His'): 10.0, ('NOUN', 'mother'): 2.0, ('NOUN', 'attack'): 9.0, ('NOUN', 'leaders'): 27.0, ('AUX', 'have'): 139.0, ('AUX', 'been'): 68.0, ('ADJ', 'past'): 6.0, ('NOUN', 'week'): 14.0, ('NOUN', 'half'): 4.0, ('PROPN', 'Hawijah'): 2.0, ('VERB', 'launched'): 2.0, ('PRON', 'that'): 123.0, ('VERB', 'left'): 8.0, ('NUM', '6'): 5.0, ('ADJ', 'dead'): 11.0, ('VERB', 'including'): 13.0, ('NUM', '4'): 14.0, ('NOUN', 'soldiers'): 9.0, ('NUM', 'One'): 10.0, ('VERB', 'was'): 128.0, ('PROPN', 'Jubur'): 1.0, ('NOUN', 'deputy'): 2.0, ('NOUN', 'garrison'): 1.0, ('NUM', 'hundred'): 2.0, ('PROPN', 'Batawi'): 2.0, ('VERB', 'demonstrated'): 3.0, ('PROPN', 'Friday'): 10.0, ('VERB', 'protesting'): 1.0, ('PROPN', 'Sarhid'): 1.0, ('NOUN', 'sons'): 4.0, ('NOUN', 'gunmen'): 1.0, ('VERB', 'wearing'): 2.0, ('NOUN', 'army'): 16.0, ('NOUN', 'uniforms'): 2.0, ('PRON', 'This'): 20.0, ('ADV', 'largely'): 3.0, ('ADJ', 'Sunni'): 11.0, ('ADJ', 'Arab'): 21.0, ('DET', 'some'): 59.0, ('NOUN', 'observers'): 3.0, ('VERB', 'accused'): 3.0, ('NOUN', 'elements'): 5.0, ('SCONJ', 'of'): 32.0, ('VERB', 'being'): 15.0, ('ADP', 'behind'): 8.0, ('NOUN', 'assassination'): 8.0, ('PUNCT', ';'): 19.0, ('PRON', 'it'): 208.0, ('ADV', 'more'): 27.0, ('ADV', 'likely'): 7.0, ('NOUN', 'work'): 12.0, ('VERB', 'punishing'): 1.0, ('SCONJ', 'for'): 25.0, ('VERB', 'cooperating'): 2.0, ('PROPN', 'Dec.'): 2.0, ('NUM', '15'): 4.0, ('NOUN', 'elections'): 29.0, ('PROPN', 'High'): 1.0, ('PROPN', 'Electoral'): 2.0, ('PROPN', 'Commission'): 6.0, ('VERB', 'denied'): 1.0, ('NOUN', 'request'): 5.0, ('PROPN', 'Debaathification'): 2.0, ('VERB', 'exclude'): 1.0, ('NUM', '51'): 1.0, ('NOUN', 'individuals'): 2.0, ('SCONJ', 'from'): 13.0, ('VERB', 'running'): 5.0, ('NOUN', 'party'): 7.0, ('NOUN', 'lists'): 2.0, ('NOUN', 'grounds'): 3.0, ('AUX', 'having'): 2.0, ('ADV', 'sufficiently'): 1.0, ('ADJ', 'involved'): 7.0, ('PROPN', 'Baath'): 3.0, ('NOUN', 'activities'): 8.0, ('VERB', 'warrant'): 3.0, ('VERB', 'excluded'): 1.0, ('ADJ', 'civil'): 7.0, ('NOUN', 'office'): 10.0, ('NOUN', 'Commission'): 1.0, ('VERB', 'said'): 52.0, ('VERB', 'had'): 43.0, ('DET', 'no'): 48.0, ('ADJ', 'legal'): 1.0, ('DET', 'such'): 10.0, ('NOUN', 'exclusion'): 2.0, ('NOUN', 'item'): 1.0, ('ADJ', 'small'): 8.0, ('NOUN', 'one'): 20.0, ('ADV', 'easily'): 4.0, ('VERB', 'missed'): 3.0, ('CONJ', 'But'): 44.0, ('PRON', 'my'): 42.0, ('NOUN', 'view'): 3.0, ('ADV', 'highly'): 8.0, ('ADJ', 'significant'): 7.0, ('VERB', 'pushed'): 4.0, ('PROPN', 'Ahmad'): 1.0, ('PROPN', 'Chalabi'): 9.0, ('PROPN', 'Iraqi'): 6.0, ('PROPN', 'National'): 27.0, ('PROPN', 'Congress'): 2.0, ('ADV', 'hard'): 4.0, ('ADJ', 'many'): 41.0, ('PROPN', 'Arabs'): 2.0, ('ADP', 'into'): 49.0, ('NOUN', 'arms'): 18.0, ('AUX', 'has'): 124.0, ('ADV', 'increasingly'): 3.0, ('VERB', 'marginalized'): 1.0, ('ADP', 'within'): 9.0, ('ADV', 'however'): 10.0, ('ADP', 'despite'): 3.0, ('NOUN', 'ties'): 2.0, ('NOUN', 'clientelage'): 1.0, ('PROPN', 'Washington'): 12.0, ('PROPN', 'Tehran'): 3.0, ('PRON', 'He'): 44.0, ('ADV', 'no'): 7.0, ('ADV', 'longer'): 6.0, ('ADJ', 'dominant'): 4.0, ('NOUN', 'list'): 15.0, ('PROPN', 'United'): 24.0, ('PROPN', 'Alliance'): 7.0, ('AUX', 'wo'): 4.0, ('VERB', 'have'): 110.0, ('NOUN', 'seats'): 2.0, ('ADJ', 'new'): 33.0, ('NOUN', 'parliament'): 3.0, ('DET', 'Some'): 7.0, ('NUM', '2,000'): 1.0, ('ADJ', 'junior'): 2.0, ('ADJ', 'old'): 20.0, ('VERB', 'recalled'): 1.0, ('NOUN', 'duty'): 3.0, ('ADJ', 'recent'): 14.0, ('NOUN', 'months'): 10.0, ('PRON', 'something'): 15.0, ('VERB', 'blocked'): 1.0, ('ADV', 'Now'): 9.0, ('AUX', 'is'): 95.0, ('VERB', 'refusing'): 1.0, ('VERB', 'punish'): 1.0, ('NOUN', 'people'): 53.0, ('ADJ', 'mere'): 2.0, ('NOUN', 'membership'): 1.0, ('NOUN', 'situation'): 14.0, ('ADV', 'only'): 44.0, ('VERB', 'going'): 26.0, ('VERB', 'get'): 27.0, ('ADJ', 'better'): 4.0, ('DET', 'this'): 88.0, ('SCONJ', 'If'): 25.0, ('PRON', 'someone'): 10.0, ('VERB', 'committed'): 1.0, ('NOUN', 'crime'): 9.0, ('ADP', 'against'): 46.0, ('NOUN', 'humanity'): 1.0, ('VERB', 'prosecute'): 1.0, ('NOUN', 'person'): 4.0, ('CONJ', 'or'): 127.0, ('PRON', 'she'): 11.0, ('VERB', 'did'): 10.0, ('PART', 'not'): 172.0, ('ADV', 'then'): 25.0, ('AUX', 'should'): 42.0, ('DET', 'all'): 65.0, ('ADJ', 'same'): 24.0, ('NOUN', 'rights'): 8.0, ('ADP', 'as'): 118.0, ('PROPN', 'Iraqis'): 10.0, ('PROPN', 'Sharq'): 1.0, ('PROPN', 'Awsat'): 1.0, ('VERB', 'reports'): 3.0, ('ADJ', 'key'): 10.0, ('NOUN', 'eyewitness'): 1.0, ('NOUN', 'trial'): 7.0, ('PROPN', 'Saddam'): 13.0, ('PROPN', 'Hussein'): 5.0, ('NUM', '1982'): 2.0, ('NOUN', 'massacre'): 1.0, ('PROPN', 'Dujail'): 1.0, ('VERB', 'died'): 6.0, ('DET', 'A'): 36.0, ('NOUN', 'team'): 1.0, ('NOUN', 'court'): 2.0, ('VERB', 'managed'): 3.0, ('VERB', 'take'): 19.0, ('NOUN', 'deposition'): 1.0, ('VERB', 'begins'): 1.0, ('ADV', 'again'): 16.0, ('PROPN', 'Nov.'): 1.0, ('NUM', '28'): 3.0, ('NOUN', 'fighting'): 1.0, ('ADV', 'still'): 25.0, ('VERB', 'continues'): 3.0, ('ADJ', 'several'): 14.0, ('NOUN', 'areas'): 8.0, ('ADV', 'mostly'): 7.0, ('PROPN', 'Sadr'): 2.0, ('NOUN', 'city'): 2.0, ('PROPN', 'Adhamiya'): 1.0, ('PROPN', 'Baghdadis'): 1.0, ('VERB', 'venture'): 1.0, ('ADP', 'out'): 46.0, ('NOUN', 'neighbourhoods'): 1.0, ('ADV', 'any'): 2.0, ('PRON', 'you'): 130.0, ('ADV', 'never'): 18.0, ('VERB', 'know'): 48.0, ('ADV', 'where'): 17.0, ('AUX', 'might'): 10.0, ('ADJ', 'stuck'): 1.0, ('PRON', 'There'): 29.0, ('NOUN', 'talk'): 3.0, ('NOUN', 'night'): 11.0, ('NOUN', 'curfew'): 1.0, ('VERB', 'implemented'): 1.0, ('PRON', 'My'): 8.0, ('NOUN', 'neighbourhood'): 3.0, ('VERB', 'surrounded'): 5.0, ('NOUN', 'troops'): 5.0, ('NUM', 'three'): 13.0, ('NOUN', 'days'): 15.0, ('ADV', 'now'): 54.0, ('NOUN', 'helicopters'): 5.0, ('VERB', 'circling'): 1.0, ('ADP', 'over'): 31.0, ('PRON', 'our'): 28.0, ('NOUN', 'heads'): 4.0, ('ADV', 'non-stop'): 1.0, ('NOUN', 'Fedayeen'): 2.0, ('ADJ', 'visible'): 3.0, ('NOUN', 'street'): 4.0, ('VERB', 'become'): 14.0, ('ADJ', 'bolder'): 1.0, ('ADP', 'than'): 32.0, ('ADV', 'ever'): 7.0, ('NOUN', 'Yesterday'): 2.0, ('NOUN', 'tens'): 5.0, ('VERB', 'putting'): 3.0, ('NOUN', 'road'): 4.0, ('NOUN', 'blocks'): 2.0, ('VERB', 'setting'): 2.0, ('NOUN', 'mortars'): 2.0, ('ADV', 'out'): 17.0, ('ADJ', 'open'): 3.0, ('ADV', 'when'): 53.0, ('PROPN', 'Americans'): 9.0, ('VERB', 'leave'): 8.0, ('NOUN', 'area'): 6.0, ('VERB', 'start'): 2.0, ('VERB', 'firing'): 2.0, ('ADV', 'indiscriminately'): 1.0, ('VERB', 'shooting'): 4.0, ('NOUN', 'AK'): 1.0, ('NOUN', \"47's\"): 1.0, ('NOUN', 'air'): 2.0, ('AUX', 'are'): 69.0, ('ADV', 'exact'): 1.0, ('NOUN', 'positions'): 4.0, ('ADP', 'during'): 11.0, ('NOUN', 'war'): 32.0, ('ADJ', 'last'): 30.0, ('NOUN', 'year'): 25.0, ('PRON', 'which'): 86.0, ('VERB', 'indicates'): 1.0, ('CONJ', 'And'): 31.0, ('PRON', 'nothing'): 17.0, ('PRON', 'we'): 76.0, ('AUX', 'can'): 73.0, ('VERB', 'do'): 38.0, ('ADP', 'about'): 86.0, ('ADV', 'really'): 14.0, ('VERB', 'suggesting'): 2.0, ('VERB', 'go'): 17.0, ('VERB', 'fight'): 6.0, ('VERB', 'living'): 7.0, ('NOUN', 'dream'): 1.0, ('NOUN', 'land'): 4.0, ('ADV', 'Even'): 7.0, ('PROPN', 'IP'): 1.0, ('PROPN', 'ICDC'): 2.0, ('VERB', 'abandoned'): 1.0, ('PRON', 'those'): 15.0, ('ADJ', 'trained'): 1.0, ('ADJ', 'armed'): 5.0, ('VERB', 'expect'): 8.0, ('ADJ', 'scared'): 1.0, ('NOUN', 'civilians'): 4.0, ('PRON', 'anything'): 10.0, ('SCONJ', 'except'): 1.0, ('VERB', 'hide'): 3.0, ('ADV', 'inside'): 2.0, ('VERB', 'pray'): 1.0, ('NOUN', 'helicopter'): 1.0, ('NOUN', 'tank'): 1.0, ('AUX', 'does'): 24.0, ('VERB', 'bomb'): 2.0, ('ADV', 'how'): 30.0, ('VERB', 'distinguish'): 1.0, ('ADJ', 'brave'): 1.0, ('ADJ', 'valiant'): 1.0, ('PUNCT', '?'): 101.0, ('PRON', 'Everyone'): 1.0, ('ADJ', 'apprehensive'): 1.0, ('PROPN', 'April'): 11.0, ('NOUN', '9th'): 1.0, ('NOUN', '10th'): 1.0, ('ADJ', 'bloody'): 2.0, ('ADJ', 'Most'): 6.0, ('VERB', 'gone'): 5.0, ('ADJ', 'few'): 18.0, ('SCONJ', 'although'): 4.0, ('VERB', 'seems'): 10.0, ('NOUN', 'rest'): 2.0, ('PUNCT', \"'\"): 85.0, ('ADJ', 'normal'): 1.0, ('VERB', 'define'): 2.0, ('PRON', 'what'): 38.0, ('NOUN', 'normal'): 1.0, ('NOUN', 'rumours'): 1.0, ('NOUN', 'preparations'): 2.0, ('NOUN', 'slum'): 1.0, ('NOUN', 'dwellers'): 1.0, ('DET', 'another'): 29.0, ('NOUN', 'looting'): 1.0, ('NOUN', 'spree'): 1.0, ('NOUN', 'banks'): 2.0, ('ADJ', 'governmental'): 2.0, ('ADJ', 'public'): 10.0, ('NOUN', 'property'): 1.0, ('ADJ', 'similar'): 6.0, ('VERB', 'took'): 8.0, ('NOUN', 'place'): 8.0, ('ADV', 'already'): 20.0, ('VERB', 'overheard'): 1.0, ('NOUN', 'youngsters'): 1.0, ('VERB', 'joking'): 1.0, ('VERB', 'saying'): 7.0, ('NOUN', 'things'): 10.0, ('PUNCT', '\"'): 383.0, ('NOUN', 'time'): 48.0, ('ADJ', 'first'): 8.0, ('VERB', 'loot'): 1.0, ('AUX', 'did'): 42.0, ('NOUN', 'Mosques'): 1.0, ('VERB', 'calling'): 2.0, ('VERB', 'donating'): 1.0, ('NOUN', 'blood'): 3.0, ('NOUN', 'food'): 3.0, ('NOUN', 'medicine'): 1.0, ('PROPN', 'Fallujah'): 7.0, ('NOUN', 'convoys'): 1.0, ('VERB', 'headed'): 2.0, ('ADJ', 'most'): 13.0, ('VERB', 'returned'): 4.0, ('ADV', 'later'): 8.0, ('ADV', 'though'): 5.0, ('PRON', 'What'): 16.0, ('VERB', 'irritates'): 1.0, ('ADJ', 'sudden'): 2.0, ('ADJ', 'false'): 7.0, ('NOUN', 'solidarity'): 1.0, ('ADP', 'between'): 29.0, ('ADJ', \"Shi'ite\"): 1.0, ('NOUN', 'clerics'): 3.0, ('ADJ', 'glad'): 1.0, ('DET', 'each'): 9.0, ('PART', 's'): 1.0, ('NOUN', 'throats'): 1.0, ('NOUN', 'chance'): 2.0, ('ADJ', 'Shia'): 2.0, ('VERB', 'describing'): 3.0, ('ADJ', 'Fallujan'): 3.0, ('NOUN', 'insurgents'): 5.0, ('PROPN', \"Ba'athists\"): 4.0, ('PROPN', 'Saddamites'): 1.0, ('PROPN', 'Wahhabis'): 1.0, ('NOUN', 'terrorists'): 10.0, ('ADV', 'just'): 40.0, ('ADV', 'ago'): 10.0, ('ADV', 'So'): 13.0, ('VERB', 'happened'): 7.0, ('VERB', 'guess'): 5.0, ('VERB', \"'s\"): 45.0, ('PRON', 'Me'): 1.0, ('NOUN', 'brother'): 6.0, ('NOUN', 'cousin'): 2.0, ('NOUN', 'enemy'): 6.0, ('NOUN', 'friend'): 5.0, ('NOUN', 'thing'): 11.0, ('VERB', 'Speaking'): 1.0, ('PROPN', 'Jazeera'): 4.0, ('VERB', 'rely'): 1.0, ('ADV', 'there'): 25.0, ('VERB', 'sent'): 24.0, ('ADV', 'over'): 9.0, ('ADJ', 'top'): 6.0, ('NOUN', 'reporter'): 6.0, ('PROPN', 'Ahmed'): 8.0, ('PROPN', 'Mansour'): 1.0, ('VERB', 'spouting'): 1.0, ('NOUN', 'kinds'): 1.0, ('NOUN', 'propaganda'): 1.0, ('ADV', 'hourly'): 1.0, ('VERB', 'reminding'): 1.0, ('PROPN', 'Sahhaf'): 1.0, ('VERB', 'targetting'): 1.0, ('NOUN', 'ambulances'): 4.0, ('NOUN', 'snipers'): 1.0, ('NOUN', 'children'): 26.0, ('ADJ', 'pregnant'): 2.0, ('NOUN', 'women'): 10.0, ('VERB', 'using'): 16.0, ('NOUN', 'cluster'): 1.0, ('VERB', 'hear'): 6.0, ('ADV', 'once'): 9.0, ('VERB', 'make'): 13.0, ('ADJ', 'unforgivable'): 1.0, ('NOUN', 'error'): 1.0, ('VERB', 'mentioned'): 6.0, ('NOUN', 'militants'): 3.0, ('PROPN', 'Marines'): 2.0, ('NOUN', 'roofs'): 1.0, ('NOUN', 'mosques'): 5.0, ('NOUN', 'houses'): 1.0, ('PROPN', 'Hay'): 1.0, ('PROPN', 'Golan'): 1.0, ('CONJ', 'but'): 88.0, ('ADV', 'of'): 6.0, ('ADV', 'course'): 7.0, ('ADJ', 'okay'): 1.0, ('PRON', 'Someone'): 1.0, ('VERB', 'called'): 19.0, ('PRON', 'himself'): 10.0, ('PROPN', 'Abu'): 2.0, ('PROPN', 'Hafs'): 1.0, ('PROPN', 'Ibn'): 1.0, ('PROPN', 'Khattab'): 1.0, ('PROPN', 'Brigades'): 1.0, ('NOUN', 'group'): 25.0, ('ADJ', 'enormous'): 3.0, ('NOUN', 'casualties'): 3.0, ('ADP', 'among'): 13.0, ('VERB', 'sweared'): 1.0, ('VERB', 'mutilating'): 2.0, ('NOUN', 'bodies'): 6.0, ('ADV', 'Over'): 2.0, ('NUM', '300'): 5.0, ('VERB', 'reported'): 12.0, ('NUM', '500'): 4.0, ('ADJ', 'wounded'): 2.0, ('ADV', 'alone'): 4.0, ('PROPN', 'Iraqiyah'): 1.0, ('PROPN', 'tv'): 1.0, ('VERB', 'controlling'): 2.0, ('PROPN', 'Ramadi'): 3.0, ('PROPN', 'Azzaman'): 1.0, ('NOUN', 'newspaper'): 5.0, ('VERB', 'signed'): 7.0, ('PROPN', 'Abdul'): 6.0, ('PROPN', 'Aziz'): 1.0, ('PROPN', 'bin'): 2.0, ('PROPN', 'Muqrin'): 1.0, ('PROPN', 'Qaeda'): 61.0, ('NOUN', 'operative'): 6.0, ('PROPN', 'Saudi'): 5.0, ('PROPN', 'Arabia'): 5.0, ('ADJ', 'Islamic'): 3.0, ('NOUN', 'website'): 2.0, ('PROPN', 'voice'): 1.0, ('VERB', 'stated'): 5.0, ('ADV', 'originally'): 3.0, ('VERB', 'permitted'): 1.0, ('PROPN', 'Islam'): 3.0, ('NOUN', 'case'): 21.0, ('VERB', 'allowed'): 4.0, ('PROPN', 'Muslims'): 5.0, ('VERB', 'use'): 15.0, ('NOUN', 'infidels'): 1.0, ('VERB', 'deter'): 1.0, ('VERB', 'committing'): 1.0, ('ADJ', 'criminal'): 1.0, ('NOUN', 'actions'): 7.0, ('VERB', 'added'): 7.0, ('PROPN', 'America'): 7.0, ('VERB', 'understand'): 12.0, ('ADP', 'except'): 5.0, ('NOUN', 'language'): 2.0, ('NOUN', 'force'): 9.0, ('NOUN', 'retaliation'): 2.0, ('VERB', 'kicked'): 1.0, ('PROPN', 'Somalia'): 1.0, ('NOUN', 'humiliation'): 1.0, ('DET', 'that'): 39.0, ('NOUN', 'soldier'): 1.0, ('VERB', 'dragged'): 3.0, ('PROPN', 'Mogadishu'): 1.0, ('ADJ', 'whole'): 4.0, ('NOUN', 'world'): 46.0, ('VERB', 'see'): 23.0, ('NOUN', 'day'): 18.0, ('PROPN', 'Jews'): 1.0, ('VERB', 'defiled'): 1.0, ('VERB', 'stepped'): 4.0, ('PROPN', 'Arabian'): 1.0, ('PROPN', 'peninsula'): 1.0, ('ADV', 'together'): 7.0, ('NOUN', 'agents'): 10.0, ('NOUN', 'supporters'): 7.0, ('PROPN', 'Elena'): 7.0, ('PART', \"'s\"): 218.0, ('NOUN', 'motorcycle'): 1.0, ('NOUN', 'tour'): 3.0, ('ADP', 'through'): 14.0, ('NOUN', 'region'): 22.0, ('ADP', 'around'): 4.0, ('PROPN', 'Chernobyl'): 14.0, ('VERB', 'revived'): 1.0, ('NOUN', 'interest'): 4.0, ('NUM', 'one'): 52.0, ('ADV', 'most'): 20.0, ('ADJ', 'serious'): 7.0, ('ADJ', 'nuclear'): 12.0, ('NOUN', 'disasters'): 1.0, ('NOUN', 'history'): 9.0, ('PRON', 'We'): 24.0, ('ADV', 'even'): 44.0, ('ADJ', 'different'): 6.0, ('NOUN', 'versions'): 1.0, ('NOUN', 'opinions'): 1.0, ('DET', 'what'): 3.0, ('NOUN', 'effect'): 4.0, ('NOUN', 'health'): 8.0, ('VERB', 'affected'): 3.0, ('NOUN', 'fallout'): 3.0, ('NOUN', 'UPDATE'): 1.0, ('VERB', 'tour'): 1.0, ('VERB', 'write'): 1.0, ('PRON', 'your'): 28.0, ('ADJ', 'own'): 26.0, ('NOUN', 'story'): 10.0, ('NOUN', 'post'): 5.0, ('NOUN', 'fault'): 1.0, ('NOUN', 'finding'): 2.0, ('VERB', 'assigning'): 1.0, ('NOUN', 'blame'): 2.0, ('VERB', 'learn'): 4.0, ('AUX', 'may'): 48.0, ('VERB', 'affect'): 2.0, ('NOUN', 'future'): 2.0, ('ADJ', 'soothing'): 1.0, ('ADJ', 'authoritative'): 1.0, ('PROPN', 'UNSCEAR'): 1.0, ('PROPN', 'Nations'): 3.0, ('PROPN', 'Scientific'): 1.0, ('PROPN', 'Committee'): 3.0, ('PROPN', 'Effects'): 1.0, ('PROPN', 'Atomic'): 2.0, ('PROPN', 'Radiation'): 2.0, ('NOUN', 'report'): 12.0, ('NUM', '2000'): 15.0, ('NOUN', 'effects'): 5.0, ('VERB', 'confirming'): 2.0, ('ADJ', 'scientific'): 3.0, ('NOUN', 'evidence'): 10.0, ('DET', 'any'): 50.0, ('NOUN', 'radiation'): 9.0, ('ADJ', 'related'): 6.0, ('VERB', 'exposed'): 2.0, ('ADV', 'heavily'): 3.0, ('VERB', 'promoted'): 2.0, ('PROPN', 'Australasian'): 1.0, ('PROPN', 'Protection'): 1.0, ('PROPN', 'Society'): 1.0, ('NOUN', 'press'): 14.0, ('NOUN', 'release'): 1.0, ('VERB', 'titled'): 3.0, ('DET', 'THE'): 2.0, ('PROPN', 'MYTHS'): 1.0, ('ADP', 'OF'): 1.0, ('PROPN', 'CHERNOBYL'): 2.0, ('VERB', 'contained'): 2.0, ('VERB', 'following'): 13.0, ('ADJ', 'widespread'): 1.0, ('NOUN', 'myths'): 1.0, ('NOUN', 'times'): 1.0, ('NOUN', 'reactor'): 1.0, ('NOUN', 'accident'): 3.0, ('NUM', '1986'): 1.0, ('VERB', 'caused'): 5.0, ('NOUN', 'thousands'): 6.0, ('ADJ', 'extra'): 1.0, ('NOUN', 'cancer'): 1.0, ('NOUN', 'deaths'): 2.0, ('VERB', 'neighbouring'): 2.0, ('NOUN', 'regions'): 2.0, ('ADV', 'severely'): 1.0, ('NOUN', 'exposure'): 5.0, ('ADJ', 'Many'): 9.0, ('VERB', 'believe'): 10.0, ('ADJ', 'true'): 6.0, ('SCONJ', 'though'): 8.0, ('PROPN', 'Russian'): 1.0, ('NOUN', 'Federation'): 1.0, ('PROPN', 'Civil'): 1.0, ('PROPN', 'Defence'): 2.0, ('PROPN', 'Emergencies'): 1.0, ('PROPN', 'Elimination'): 1.0, ('PROPN', 'Conseguences'): 1.0, ('PROPN', 'Natural'): 1.0, ('PROPN', 'Disasters'): 1.0, ('PROPN', 'EMERCOM'): 1.0, ('PROPN', 'Russia'): 9.0, ('NUM', '1996'): 1.0, ('NOUN', 'ACCIDENT'): 1.0, ('NUM', 'TEN'): 1.0, ('NOUN', 'YEARS'): 1.0, ('ADV', 'ON'): 1.0, ('NOUN', 'decade'): 5.0, ('ADJ', 'real'): 13.0, ('NOUN', 'increase'): 4.0, ('NOUN', 'childhood'): 2.0, ('ADJ', 'certain'): 12.0, ('NOUN', 'extent'): 3.0, ('NOUN', 'adult'): 1.0, ('NOUN', 'carcinoma'): 1.0, ('NOUN', 'thyroid'): 2.0, ('VERB', 'contaminated'): 1.0, ('ADJ', 'former'): 25.0, ('PROPN', 'Soviet'): 3.0, ('PROPN', 'Union'): 2.0, ('PROPN', 'Wi940'): 1.0, ('VERB', 'attributed'): 1.0, ('SCONJ', 'until'): 3.0, ('VERB', 'proven'): 2.0, ('ADV', 'otherwise'): 2.0, ('ADJ', 'prestigious'): 1.0, ('PROPN', 'IAEA'): 3.0, ('PROPN', 'International'): 4.0, ('PROPN', 'Energy'): 2.0, ('PROPN', 'Agency'): 2.0, ('VERB', 'published'): 5.0, ('ADJ', 'early'): 15.0, ('VERB', 'based'): 19.0, ('NOUN', 'information'): 13.0, ('ADJ', 'Russian'): 4.0, ('NOUN', 'sources'): 4.0, ('ADV', 'However'): 11.0, ('NUM', '2001'): 17.0, ('PROPN', 'Fifteen'): 1.0, ('PROPN', 'Years'): 1.0, ('ADP', 'after'): 27.0, ('PROPN', 'Accident'): 1.0, ('PROPN', 'Lessons'): 2.0, ('VERB', 'learned'): 2.0, ('VERB', 'contradict'): 3.0, ('ADJ', 'earlier'): 6.0, ('NOUN', 'reports'): 13.0, ('ADV', 'Here'): 6.0, ('NOUN', 'excerpts'): 1.0, ('ADJ', 'dramatic'): 3.0, ('VERB', 'induced'): 1.0, ('NOUN', 'cancers'): 1.0, ('NOUN', 'adolescents'): 1.0, ('PROPN', 'Belarus'): 2.0, ('PROPN', 'Ukraine'): 2.0, ('VERB', 'observed'): 1.0, ('ADP', 'since'): 10.0, ('NUM', '1991'): 2.0, ('PUNCT', '...'): 27.0, ('NOUN', 'drop'): 1.0, ('NOUN', 'birth'): 3.0, ('NOUN', 'rate'): 3.0, ('NOUN', 'deterioration'): 3.0, ('ADJ', 'reproductive'): 1.0, ('NOUN', 'complications'): 1.0, ('NOUN', 'pregnancy'): 1.0, ('ADJ', 'neonatal'): 1.0, ('PUNCT', '....'): 10.0, ('NOUN', 'dynamics'): 1.0, ('NOUN', 'change'): 3.0, ('NOUN', 'state'): 18.0, ('NOUN', 'countries'): 14.0, ('ADJ', 'post-accident'): 1.0, ('NOUN', 'period'): 5.0, ('VERB', 'characterized'): 1.0, ('ADJ', 'persistent'): 1.0, ('ADJ', 'negative'): 1.0, ('NOUN', 'tendencies'): 1.0, ('NOUN', 'morbidity'): 1.0, ('ADV', 'up'): 9.0, ('ADJ', 'healthy'): 1.0, ('VERB', 'dropping'): 1.0, ('NOUN', 'disability'): 1.0, ('VERB', 'increasing'): 2.0, ('ADP', 'As'): 7.0, ('NOUN', 'parent'): 1.0, ('ADV', 'well'): 32.0, ('VERB', 'imagine'): 3.0, ('ADJ', 'painful'): 3.0, ('AUX', 'must'): 13.0, ('DET', 'those'): 16.0, ('NOUN', 'families'): 7.0, ('PRON', 'whose'): 4.0, ('VERB', 'succumbing'): 1.0, ('NOUN', 'poisoning'): 2.0, ('NOUN', 'lot'): 17.0, ('VERB', 'Being'): 1.0, ('VERB', 'informed'): 1.0, ('VERB', 'give'): 10.0, ('NOUN', 'certainty'): 1.0, ('ADJ', 'desirable'): 1.0, ('VERB', 'conflicting'): 1.0, ('NOUN', 'wealth'): 3.0, ('NOUN', 'references'): 1.0, ('VERB', 'Read'): 2.0, ('NOUN', 'links'): 4.0, ('VERB', 'draw'): 1.0, ('NOUN', 'conclusions'): 2.0, ('DET', 'These'): 8.0, ('VERB', 'present'): 4.0, ('NOUN', 'viewpoints'): 1.0, ('VERB', 'existed'): 5.0, ('VERB', 'exist'): 4.0, ('NOUN', 'disaster'): 2.0, ('NOUN', 'Report'): 2.0, ('X', 'http://www.ibiblio.org/expo/soviet.exhibit/chernobyl.html'): 1.0, ('X', 'http://www.ibrae.ac.ru/IBRAE/eng/chernobyl/nat_rep/nat_repe.htm#24'): 1.0, ('X', 'http://www.nsrl.ttu.edu/chernobyl/wildlifepreserve.htm'): 1.0, ('X', 'http://www.environmentalchemistry.com/yogi/hazmat/articles/chernobyl1.html'): 1.0, ('X', 'http://digon_va.tripod.com/Chernobyl.htm'): 1.0, ('X', 'http://www.oneworld.org/index_oc/issue196/byckau.html'): 1.0, ('X', 'http://www.collectinghistory.net/chernobyl/'): 1.0, ('X', 'http://www.ukrainianweb.com/chernobyl_ukraine.htm'): 1.0, ('X', 'http://www.bullatomsci.org/issues/1993/s93/s93Marples.html'): 1.0, ('X', 'http://www.calguard.ca.gov/ia/Chernobyl-15%20years.htm'): 1.0, ('X', 'http://www.infoukes.com/history/chornobyl/gregorovich/index.html'): 1.0, ('X', 'http://www.un.org/ha/chernobyl/'): 1.0, ('X', 'http://www.tecsoc.org/pubs/history/2002/apr26.htm'): 1.0, ('X', 'http://www.chernobyl.org.uk/page2.htm'): 1.0, ('X', 'http://www.time.com/time/daily/chernobyl/860901.accident.html'): 1.0, ('X', 'http://www.infoukes.com/history/chornobyl/elg/'): 1.0, ('X', 'http://www.world-nuclear.org/info/chernobyl/inf07.htm'): 1.0, ('X', 'http://www.nea.fr/html/rp/chernobyl/conclusions5.html'): 1.0, ('X', 'http://www.nea.fr/html/rp/chernobyl/c01.html'): 1.0, ('X', 'http://www.nea.fr/html/rp/chernobyl/c05.html'): 1.0, ('X', 'http://www.physics.isu.edu/radinf/chern.htm'): 1.0, ('X', 'http://www.chernobyl.info/en'): 1.0, ('X', 'http://www.arps.org.au/Chernobyl.htm'): 1.0, ('X', 'http://www-formal.stanford.edu/jmc/progress/chernobyl.html'): 1.0, ('X', 'http://www.21stcenturysciencetech.com/articles/chernobyl.html'): 1.0, ('ADJ', 'interested'): 9.0, ('VERB', 'hearing'): 1.0, ('VERB', 'reached'): 4.0, ('VERB', 'found'): 12.0, ('ADJ', 'convincing'): 1.0, ('NOUN', 'child'): 7.0, ('NOUN', \"50's\"): 1.0, ('ADJ', 'glandular'): 1.0, ('NOUN', 'problems'): 5.0, ('VERB', 'treated'): 2.0, ('NOUN', 'therapy'): 1.0, ('ADJ', 'primitive'): 1.0, ('ADJ', 'best'): 6.0, ('VERB', 'stopped'): 5.0, ('NOUN', 'lesion'): 3.0, ('NOUN', 'neck'): 1.0, ('VERB', 'started'): 7.0, ('VERB', 'enlarging'): 1.0, ('PUNCT', '--'): 40.0, ('NOUN', 'CA'): 1.0, ('NOUN', 'treatments'): 1.0, ('VERB', 'Remember'): 2.0, ('NOUN', 'shoe'): 1.0, ('VERB', 'sizing'): 1.0, ('NOUN', 'machines'): 1.0, ('NOUN', 'form'): 4.0, ('NOUN', 'xray'): 1.0, ('PRON', 'That'): 16.0, ('AUX', 'Do'): 20.0, ('VERB', 'mention'): 2.0, ('VERB', 'done'): 12.0, ('VERB', 'doing'): 11.0, ('PROPN', 'Hirsohima'): 1.0, ('CONJ', '&'): 6.0, ('PROPN', 'Nagaski'): 1.0, ('NOUN', 'folks'): 1.0, ('NOUN', 'doubts'): 1.0, ('PUNCT', '.......'): 1.0, ('PROPN', 'Children'): 1.0, ('PROPN', 'Project'): 1.0, ('X', 'http://www.adiccp.org/home/default.asp'): 1.0, ('VERB', 'offers'): 1.0, ('NOUN', 'ways'): 6.0, ('VERB', 'help'): 11.0, ('PROPN', 'Rest'): 1.0, ('PROPN', 'Recuperation'): 1.0, ('PROPN', 'Program'): 1.0, ('ADV', 'wherein'): 1.0, ('NOUN', 'weeks'): 8.0, ('NOUN', 'summer'): 1.0, ('ADJ', 'little'): 13.0, ('VERB', 'spent'): 5.0, ('VERB', 'receiving'): 2.0, ('ADJ', 'wholesome'): 1.0, ('ADJ', 'uncontaminated'): 1.0, ('ADJ', 'good'): 25.0, ('ADJ', 'medical'): 1.0, ('NOUN', 'care'): 2.0, ('X', 'etc.'): 2.0, ('VERB', 'add'): 2.0, ('NOUN', 'lives'): 6.0, ('PROPN', 'S.'): 1.0, ('NOUN', 'acquaintance'): 1.0, ('VERB', 'hosted'): 3.0, ('DET', 'these'): 33.0, ('NOUN', 'ones'): 3.0, ('VERB', 'breaks'): 2.0, ('NOUN', 'heart'): 4.0, ('NOUN', 'urge'): 1.0, ('VERB', 'protect'): 3.0, ('VERB', 'gather'): 1.0, ('ADV', 'in'): 13.0, ('ADV', 'almost'): 10.0, ('ADJ', 'overwhelming'): 4.0, ('VERB', 'makes'): 6.0, ('ADJ', 'grateful'): 1.0, ('NOUN', 'blessings'): 1.0, ('VERB', 'Take'): 1.0, ('PROPN', 'Linda'): 1.0, ('VERB', \"'m\"): 9.0, ('ADJ', 'sorry'): 1.0, ('VERB', 'say'): 17.0, ('VERB', 'revealed'): 2.0, ('NOUN', 'fake'): 1.0, ('NOUN', 'videotape'): 1.0, ('NOUN', 'audio'): 1.0, ('NOUN', 'speeches'): 1.0, ('PROPN', 'Bin'): 13.0, ('PROPN', 'Laden'): 14.0, ('PROPN', 'Ayman'): 6.0, ('PROPN', 'Zawahiri'): 41.0, ('VERB', 'tell'): 8.0, ('NOUN', 'hopes'): 1.0, ('ADJ', 'remaining'): 1.0, ('NOUN', 'leadership'): 3.0, ('NOUN', 'organization'): 4.0, ('SCONJ', 'Because'): 2.0, ('PROPN', 'Pakistan'): 51.0, ('VERB', 'capture'): 1.0, ('VERB', 'kill'): 9.0, ('ADV', 'about'): 5.0, ('PUNCT', '/'): 23.0, ('NOUN', '3s'): 1.0, ('NUM', '25'): 5.0, ('NOUN', 'commanders'): 3.0, ('ADJ', 'middle'): 3.0, ('NOUN', 'managers'): 1.0, ('ADJ', 'close'): 7.0, ('NOUN', 'contact'): 2.0, ('NOUN', 'tape'): 1.0, ('VERB', 'signal'): 1.0, ('NOUN', 'priorities'): 2.0, ('PRON', 'These'): 3.0, ('X', '1'): 3.0, ('VERB', 'Assassinate'): 1.0, ('VERB', 'overthrow'): 1.0, ('PROPN', 'Gen.'): 2.0, ('PROPN', 'Pervez'): 5.0, ('PROPN', 'Musharraf'): 28.0, ('ADJ', 'Pakistani'): 21.0, ('ADJ', 'military'): 24.0, ('NOUN', 'president'): 5.0, ('NOUN', 'coup'): 8.0, ('NUM', '1999'): 7.0, ('VERB', 'thrown'): 2.0, ('PROPN', 'States'): 21.0, ('PROPN', 'Taliban'): 32.0, ('VERB', 'trying'): 12.0, ('VERB', 'purge'): 1.0, ('NOUN', 'officer'): 6.0, ('NOUN', 'corps'): 2.0, ('ADJ', 'substantial'): 3.0, ('NOUN', 'sympathizers'): 3.0, ('NOUN', 'intelligence'): 20.0, ('VERB', 'captured'): 5.0, ('ADJ', 'major'): 7.0, ('NOUN', 'figures'): 5.0, ('ADP', 'like'): 20.0, ('PROPN', 'Zubayda'): 1.0, ('PROPN', 'Khalid'): 10.0, ('PROPN', 'Shaykh'): 1.0, ('PROPN', 'Muhammad'): 1.0, ('ADV', 'as'): 32.0, ('ADV', 'nearly'): 7.0, ('NOUN', 'operatives'): 2.0, ('NUM', '400'): 2.0, ('PRON', 'whom'): 4.0, ('PROPN', 'Pakistanis'): 2.0, ('VERB', 'turned'): 2.0, ('VERB', 'held'): 11.0, ('PROPN', 'October'): 9.0, ('NUM', '2002'): 11.0, ('ADJ', 'right'): 6.0, ('NOUN', 'parties'): 4.0, ('NUM', '20'): 12.0, ('VERB', 'went'): 10.0, ('ADJ', 'fundamentalist'): 2.0, ('ADJ', 'religious'): 12.0, ('NOUN', 'coalition'): 5.0, ('PROPN', 'MMA'): 4.0, ('PROPN', 'Northwest'): 1.0, ('PROPN', 'Frontier'): 2.0, ('VERB', 'shelters'): 1.0, ('ADJ', 'joint'): 1.0, ('PROPN', 'Baluchistan'): 4.0, ('PROPN', 'Afghanistan'): 36.0, ('VERB', 'instigate'): 1.0, ('PROPN', 'Islamist'): 4.0, ('VERB', 'hope'): 4.0, ('VERB', 'catapult'): 1.0, ('NOUN', 'power'): 20.0, ('ADJ', 'political'): 20.0, ('NOUN', 'allies'): 3.0, ('NOUN', 'hosts'): 1.0, ('ADV', 'thereby'): 2.0, ('VERB', 'gain'): 1.0, ('NOUN', 'control'): 7.0, ('NOUN', 'base'): 6.0, ('NOUN', 'operations'): 4.0, ('DET', 'All'): 5.0, ('ADJ', 'unlikely'): 2.0, ('NOUN', 'crackpot'): 1.0, ('NOUN', 'schemes'): 1.0, ('NOUN', 'mujahidin'): 1.0, ('ADV', 'virtually'): 3.0, ('NOUN', 'Response'): 3.0, ('PRON', 'whatever'): 1.0, ('VERB', 'strengthen'): 4.0, ('NOUN', 'legitimacy'): 1.0, ('NOUN', 'hand'): 11.0, ('VERB', 'pressure'): 2.0, ('ADP', 'off'): 18.0, ('NOUN', 'uniform'): 2.0, ('ADJ', 'fair'): 5.0, ('NOUN', 'election'): 5.0, ('VERB', 'repeal'): 1.0, ('ADJ', 'contentious'): 1.0, ('PROPN', 'Legal'): 1.0, ('PROPN', 'Framework'): 1.0, ('PROPN', 'Order'): 2.0, ('ADV', 'essentially'): 2.0, ('VERB', 'perpetuates'): 1.0, ('NOUN', 'dictatorship'): 3.0, ('NOUN', 'restrictions'): 2.0, ('VERB', 'lifted'): 1.0, ('ADJ', 'mainstream'): 4.0, ('PROPN', 'Muslim'): 1.0, ('PROPN', 'League'): 1.0, ('PROPN', 'N'): 1.0, ('PROPN', 'People'): 2.0, ('VERB', 'defeat'): 1.0, ('ADV', 'not'): 25.0, ('VERB', 'hogtied'): 1.0, ('ADJ', 'secret'): 2.0, ('ADJ', 'strong'): 16.0, ('VERB', 'arm'): 2.0, ('PROPN', 'India'): 55.0, ('ADJ', 'final'): 7.0, ('NOUN', 'settlement'): 2.0, ('PROPN', 'Kashmir'): 15.0, ('NOUN', 'issue'): 10.0, ('VERB', 'attempted'): 2.0, ('NOUN', 'lack'): 6.0, ('VERB', 'helping'): 4.0, ('ADJ', 'Indian'): 35.0, ('NOUN', 'justification'): 1.0, ('NOUN', 'overthrow'): 3.0, ('VERB', 'generates'): 1.0, ('ADV', 'far'): 15.0, ('ADJ', 'more'): 33.0, ('NOUN', 'terrorism'): 11.0, ('NOUN', 'threat'): 12.0, ('SCONJ', 'than'): 10.0, ('X', '2'): 3.0, ('VERB', 'Target'): 1.0, ('PROPN', 'Israel'): 34.0, ('VERB', 'encourage'): 4.0, ('ADJ', 'worst'): 1.0, ('PROPN', 'Palestinians'): 8.0, ('SCONJ', 'by'): 19.0, ('VERB', 'playing'): 1.0, ('ADJ', 'iron'): 2.0, ('NOUN', 'fist'): 2.0, ('NOUN', 'policies'): 2.0, ('PROPN', 'Sharon'): 7.0, ('VERB', 'succeeded'): 2.0, ('ADV', 'politically'): 3.0, ('VERB', 'isolating'): 1.0, ('PROPN', 'Hamas'): 6.0, ('NOUN', 'process'): 11.0, ('VERB', 'cutting'): 1.0, ('PRON', 'its'): 50.0, ('NOUN', 'funding'): 2.0, ('VERB', 'pull'): 2.0, ('ADV', 'less'): 8.0, ('ADJ', 'sophisticated'): 1.0, ('NOUN', 'attacks'): 11.0, ('VERB', 'defanged'): 1.0, ('ADV', 'simply'): 6.0, ('NOUN', 'means'): 3.0, ('VERB', 'establishing'): 1.0, ('ADJ', 'general'): 7.0, ('NOUN', 'peace'): 20.0, ('PROPN', 'Bush'): 100.0, ('NOUN', 'administration'): 5.0, ('ADV', 'finally'): 1.0, ('VERB', 'apply'): 2.0, ('ADJ', 'effective'): 4.0, ('NOUN', 'pressure'): 4.0, ('VERB', 'stop'): 6.0, ('NOUN', 'outrages'): 1.0, ('NOUN', 'colonization'): 1.0, ('PROPN', 'West'): 4.0, ('PROPN', 'Bank'): 4.0, ('PROPN', 'Gaza'): 2.0, ('ADJ', 'hard'): 5.0, ('NOUN', 'line'): 8.0, ('VERB', 'worked'): 2.0, ('NOUN', 'tandem'): 2.0, ('VERB', 'ratchet'): 1.0, ('NOUN', 'tensions'): 5.0, ('ADV', 'further'): 4.0, ('VERB', 'spill'): 1.0, ('ADJ', 'Muslim'): 3.0, ('VERB', 'serve'): 4.0, ('NOUN', 'recruiting'): 1.0, ('NOUN', 'tool'): 3.0, ('NOUN', 'search'): 7.0, ('ADJ', 'willing'): 3.0, ('VERB', 'hit'): 3.0, ('VERB', 'owes'): 1.0, ('ADV', 'at'): 15.0, ('ADV', 'least'): 15.0, ('ADV', 'this'): 1.0, ('ADJ', 'much'): 12.0, ('NOUN', 'crisis'): 3.0, ('VERB', 'cease'): 1.0, ('ADV', 'militarily'): 2.0, ('ADJ', 'unnecessary'): 1.0, ('NOUN', 'provocations'): 1.0, ('VERB', 'establish'): 2.0, ('ADJ', 'genuine'): 1.0, ('X', '3'): 3.0, ('VERB', 'Make'): 1.0, ('PROPN', 'Republican'): 4.0, ('PROPN', 'Right'): 1.0, ('NOUN', 'tactics'): 2.0, ('ADV', 'actually'): 18.0, ('ADJ', 'hostile'): 1.0, ('NOUN', 'territory'): 6.0, ('ADP', 'without'): 8.0, ('VERB', 'succeed'): 2.0, ('SCONJ', 'By'): 1.0, ('VERB', 'moving'): 4.0, ('ADV', 'quickly'): 1.0, ('NOUN', 'sovereignty'): 4.0, ('NOUN', 'improvement'): 1.0, ('ADJ', 'able'): 8.0, ('SCONJ', 'so'): 8.0, ('NOUN', 'foreigners'): 5.0, ('ADV', 'Certainly'): 1.0, ('PROPN', 'Shiites'): 7.0, ('VERB', 'hate'): 1.0, ('ADV', 'likewise'): 2.0, ('PROPN', 'Kurds'): 2.0, ('NOUN', 'problem'): 16.0, ('VERB', 'mollifying'): 1.0, ('PROPN', 'Sunnis'): 4.0, ('VERB', 'solved'): 2.0, ('VERB', 'avoid'): 1.0, ('VERB', 'giving'): 5.0, ('NOUN', 'entree'): 1.0, ('VERB', 'put'): 8.0, ('ADV', 'away'): 9.0, ('ADJ', 'free'): 13.0, ('NOUN', 'fetishism'): 1.0, ('NOUN', 'while'): 2.0, ('VERB', 'creating'): 3.0, ('NOUN', 'jobs'): 2.0, ('VERB', 'pumping'): 1.0, ('NOUN', 'money'): 15.0, ('NOUN', 'households'): 1.0, ('VERB', 'need'): 13.0, ('PROPN', 'FDR'): 1.0, ('PROPN', 'Ronald'): 1.0, ('PROPN', 'Reagan'): 1.0, ('ADV', 'Of'): 3.0, ('ADV', 'sooner'): 1.0, ('VERB', 'withdrawn'): 2.0, ('NOUN', 'favor'): 3.0, ('ADJ', 'provocative'): 1.0, ('ADJ', 'local'): 7.0, ('ADJ', 'international'): 22.0, ('VERB', 'Getting'): 1.0, ('ADJ', 'Spanish'): 3.0, ('NOUN', 'rallying'): 1.0, ('NOUN', 'cry'): 1.0, ('VERB', 'getting'): 4.0, ('ADJ', 'British'): 2.0, ('VERB', 'keep'): 6.0, ('NOUN', 'mind'): 7.0, ('NOUN', 'nationalists'): 1.0, ('NOUN', 'radicals'): 3.0, ('ADJ', 'likely'): 12.0, ('VERB', 'remain'): 8.0, ('ADJ', 'dangerous'): 11.0, ('NOUN', 'infiltrators'): 1.0, ('PRON', 'one'): 4.0, ('NOUN', 'eyes'): 2.0, ('NOUN', 'ball'): 1.0, ('PROPN', 'George'): 19.0, ('PROPN', 'W.'): 6.0, ('VERB', 'alleged'): 3.0, ('PROPN', 'Thursday'): 4.0, ('PROPN', 'John'): 16.0, ('PROPN', 'Edwards'): 7.0, ('VERB', 'lacks'): 1.0, ('NOUN', 'experience'): 6.0, ('ADJ', 'necessary'): 7.0, ('PROPN', 'president'): 6.0, ('NOUN', 'argument'): 4.0, ('VERB', 'lacked'): 1.0, ('VERB', 'ran'): 3.0, ('NOUN', 'sort'): 4.0, ('ADJ', 'cheap'): 1.0, ('NOUN', 'shot'): 1.0, ('VERB', 'hoists'): 1.0, ('NOUN', 'petard'): 1.0, ('VERB', 'Let'): 5.0, ('PRON', \"'s\"): 2.0, ('VERB', 'remember'): 12.0, ('ADJ', 'seminal'): 1.0, ('NOUN', 'moment'): 5.0, ('VERB', 'fails'): 1.0, ('NOUN', 'pop'): 2.0, ('NOUN', 'quiz'): 2.0, ('PROPN', 'November'): 7.0, ('NUM', '5'): 11.0, ('NOUN', 'Web'): 1.0, ('VERB', 'posted'): 3.0, ('NUM', '3:29'): 1.0, ('NOUN', 'p.m.'): 1.0, ('PROPN', 'EST'): 1.0, ('NUM', '2029'): 1.0, ('PROPN', 'GMT'): 1.0, ('PROPN', 'WASHINGTON'): 1.0, ('PROPN', 'CNN'): 1.0, ('PROPN', 'Texas'): 11.0, ('PROPN', 'Gov.'): 1.0, ('VERB', 'enduring'): 1.0, ('ADJ', 'sharp'): 1.0, ('NOUN', 'criticism'): 1.0, ('ADJ', 'unable'): 2.0, ('VERB', 'name'): 4.0, ('NUM', 'four'): 11.0, ('ADJ', 'current'): 11.0, ('ADJ', 'hot'): 3.0, ('NOUN', 'spots'): 2.0, ('PROPN', 'President'): 15.0, ('PROPN', 'Bill'): 2.0, ('PROPN', 'Clinton'): 3.0, ('VERB', 'says'): 21.0, ('VERB', 'pick'): 3.0, ('NOUN', 'names'): 1.0, ('ADJ', 'front'): 1.0, ('NOUN', 'runner'): 1.0, ('ADJ', 'Republican'): 2.0, ('ADJ', 'presidential'): 2.0, ('NOUN', 'nomination'): 1.0, ('VERB', 'faltered'): 1.0, ('NOUN', 'affairs'): 2.0, ('VERB', 'posed'): 4.0, ('PROPN', 'Andy'): 1.0, ('PROPN', 'Hiller'): 4.0, ('PROPN', 'WHDH'): 1.0, ('PROPN', 'TV'): 2.0, ('PROPN', 'Boston'): 2.0, ('VERB', 'asked'): 8.0, ('PROPN', 'Chechnya'): 2.0, ('PROPN', 'Taiwan'): 3.0, ('ADJ', 'partial'): 1.0, ('NOUN', 'response'): 3.0, ('NOUN', 'query'): 1.0, ('NOUN', 'leader'): 19.0, ('VERB', 'referring'): 1.0, ('ADJ', 'Taiwanese'): 1.0, ('PROPN', 'Lee'): 5.0, ('PROPN', 'Teng'): 1.0, ('PROPN', 'hui'): 1.0, ('NOUN', 'others'): 13.0, ('AUX', 'Can'): 6.0, ('NOUN', 'general'): 5.0, ('NOUN', 'charge'): 9.0, ('VERB', 'inquiring'): 2.0, ('PROPN', 'Pervaiz'): 2.0, ('VERB', 'seized'): 5.0, ('NUM', '12'): 4.0, ('VERB', 'Wait'): 1.0, ('VERB', 'wait'): 2.0, ('NUM', '50'): 3.0, ('NOUN', 'questions'): 13.0, ('VERB', 'replied'): 1.0, ('INTJ', 'No'): 1.0, ('VERB', 'answering'): 2.0, ('NOUN', 'question'): 11.0, ('AUX', \"'s\"): 4.0, ('VERB', 'elected'): 6.0, ('NOUN', 'guy'): 2.0, ('VERB', 'appears'): 5.0, ('VERB', 'bring'): 4.0, ('NOUN', 'stability'): 5.0, ('VERB', 'think'): 24.0, ('NOUN', 'subcontinent'): 2.0, ('PROPN', 'Gore'): 2.0, ('VERB', 'released'): 7.0, ('NOUN', 'statement'): 8.0, ('VERB', 'taking'): 10.0, ('NOUN', 'task'): 4.0, ('NOUN', 'comments'): 8.0, ('ADJ', 'troubling'): 1.0, ('NOUN', 'candidate'): 8.0, ('ADJ', 'oldest'): 2.0, ('NOUN', 'democracy'): 11.0, ('VERB', 'characterize'): 1.0, ('NOUN', 'takeover'): 1.0, ('ADV', 'Further'): 1.0, ('ADJ', 'disturbing'): 1.0, ('NOUN', 'nation'): 3.0, ('VERB', 'tested'): 1.0, ('NOUN', 'weapons'): 21.0, ('ADV', 'shortly'): 2.0, ('VERB', 'voicing'): 1.0, ('NOUN', 'opposition'): 1.0, ('PROPN', 'Comprehensive'): 1.0, ('PROPN', 'Test'): 1.0, ('PROPN', 'Ban'): 1.0, ('PROPN', 'Treaty'): 1.0, ('NOUN', 'spokesman'): 6.0, ('VERB', 'criticized'): 1.0, ('VERB', 'condone'): 1.0, ('ADV', 'democratically'): 1.0, ('NOUN', 'governments'): 3.0, ('PROPN', 'David'): 7.0, ('PROPN', 'Leavy'): 1.0, ('PROPN', 'Security'): 5.0, ('PROPN', 'Council'): 4.0, ('ADV', 'Not'): 3.0, ('PROPN', 'General'): 10.0, ('VERB', 'confused'): 1.0, ('NOUN', 'making'): 3.0, ('ADV', 'moreover'): 1.0, ('ADV', 'on'): 9.0, ('VERB', 'suggest'): 4.0, ('ADJ', 'prime'): 4.0, ('NOUN', 'minister'): 7.0, ('NOUN', 'installation'): 1.0, ('NOUN', 'military'): 7.0, ('ADJ', 'strongest'): 2.0, ('NOUN', 'supporter'): 4.0, ('ADJ', 'part'): 2.0, ('ADP', 'because'): 9.0, ('NOUN', 'anger'): 2.0, ('PROPN', 'Prime'): 3.0, ('PROPN', 'Minister'): 5.0, ('PROPN', 'Nawaz'): 1.0, ('PROPN', 'Sharif'): 1.0, ('VERB', 'back'): 3.0, ('ADP', 'down'): 5.0, ('VERB', 'confronting'): 1.0, ('ADV', 'explicitly'): 1.0, ('VERB', 'came'): 8.0, ('NOUN', 'warmonger'): 1.0, ('AUX', 'ca'): 7.0, ('ADJ', 'ominous'): 1.0, ('NOUN', 'performance'): 3.0, ('NOUN', 'interview'): 5.0, ('VERB', 'stuttering'): 1.0, ('NOUN', 'General'): 1.0, ('NOUN', 'name'): 8.0, ('ADV', 'obviously'): 3.0, ('NOUN', 'idea'): 2.0, ('VERB', 'talking'): 9.0, ('ADJ', 'ill'): 1.0, ('ADJ', 'fated'): 1.0, ('NOUN', 'instincts'): 1.0, ('VERB', 'liked'): 2.0, ('ADJ', 'authoritarian'): 1.0, ('NOUN', 'rule'): 4.0, ('ADV', 'better'): 5.0, ('VERB', 'equating'): 1.0, ('VERB', 'needed'): 6.0, ('PROPN', 'South'): 9.0, ('PROPN', 'Asia'): 9.0, ('NOUN', 'giants'): 1.0, ('ADJ', 'radical'): 4.0, ('NOUN', 'politics'): 5.0, ('ADJ', 'latter'): 3.0, ('ADJ', 'dire'): 2.0, ('NOUN', 'security'): 23.0, ('VERB', 'becoming'): 3.0, ('ADJ', 'unstable'): 1.0, ('ADJ', 'less'): 3.0, ('VERB', 'play'): 3.0, ('NOUN', 'brinkmanship'): 1.0, ('VERB', 'risking'): 1.0, ('ADV', 'twice'): 1.0, ('PROPN', 'September'): 14.0, ('NUM', '11'): 9.0, ('ADP', 'under'): 31.0, ('ADJ', 'extreme'): 3.0, ('NOUN', 'duress'): 1.0, ('VERB', 'continued'): 1.0, ('VERB', 'support'): 3.0, ('PROPN', 'Islamism'): 1.0, ('ADV', 'recently'): 8.0, ('VERB', 'implicated'): 2.0, ('NOUN', 'attempts'): 3.0, ('NOUN', 'body'): 4.0, ('VERB', 'proclaimed'): 2.0, ('VERB', 'bringing'): 2.0, ('NOUN', 'fall'): 2.0, ('NOUN', 'answer'): 3.0, ('NOUN', 'merit'): 3.0, ('DET', 'Another'): 5.0, ('ADV', 'certainly'): 5.0, ('VERB', 'knows'): 2.0, ('ADJ', 'foreign'): 6.0, ('ADV', 'Indeed'): 3.0, ('VERB', 'given'): 13.0, ('VERB', 'rampaged'): 1.0, ('VERB', 'alienating'): 1.0, ('VERB', 'ignoring'): 1.0, ('ADJ', 'vital'): 3.0, ('NOUN', 'conflicts'): 2.0, ('NOUN', 'potential'): 3.0, ('VERB', 'blow'): 3.0, ('VERB', 'argue'): 3.0, ('VERB', 'does'): 3.0, ('PART', \"'\"): 9.0, ('NOUN', 'campaign'): 20.0, ('NOUN', 'literature'): 2.0, ('VERB', 'believes'): 4.0, ('PROPN', 'U.S.'): 10.0, ('ADJ', 'active'): 9.0, ('VERB', 'resolve'): 1.0, ('VERB', 'reducing'): 1.0, ('PROPN', 'Northern'): 5.0, ('PROPN', 'Ireland'): 1.0, ('NOUN', 'role'): 5.0, ('VERB', 'promoting'): 1.0, ('PROPN', 'Israelis'): 3.0, ('ADP', 'From'): 4.0, ('PROPN', 'Daily'): 1.0, ('PROPN', 'Star'): 2.0, ('ADP', 'By'): 10.0, ('PROPN', 'Juan'): 3.0, ('PROPN', 'Cole'): 3.0, ('PROPN', 'June'): 7.0, ('NUM', '04'): 1.0, ('NUM', '2004'): 13.0, ('SCONJ', 'As'): 14.0, ('NOUN', 'public'): 2.0, ('ADV', 'gradually'): 1.0, ('VERB', 'wearies'): 1.0, ('VERB', 'begun'): 1.0, ('VERB', 'worrying'): 1.0, ('NOUN', 'conditions'): 1.0, ('ADJ', 'anti-American'): 1.0, ('ADJ', 'closer'): 2.0, ('VERB', 'suffer'): 2.0, ('NOUN', 'instability'): 2.0, ('ADV', 'acutely'): 1.0, ('ADV', 'Ironically'): 2.0, ('NOUN', 'proponents'): 1.0, ('ADJ', 'Israeli'): 9.0, ('PROPN', 'Ariel'): 2.0, ('ADJ', 'neoconservative'): 1.0, ('AUX', 'Have'): 4.0, ('VERB', 'weakened'): 2.0, ('ADJ', 'biggest'): 2.0, ('VERB', 'faces'): 2.0, ('ADJ', 'conventional'): 1.0, ('NOUN', 'armies'): 1.0, ('ADJ', 'asymmetrical'): 3.0, ('ADJ', 'Palestinian'): 7.0, ('ADJ', 'national'): 6.0, ('NOUN', 'liberation'): 1.0, ('NOUN', 'movements'): 3.0, ('NOUN', 'derailing'): 1.0, ('PROPN', 'Oslo'): 2.0, ('NOUN', 'intifada'): 1.0, ('VERB', 'encouraged'): 3.0, ('NOUN', 'suicide'): 22.0, ('NOUN', 'turn'): 2.0, ('VERB', 'discouraged'): 1.0, ('NOUN', 'immigrants'): 1.0, ('VERB', 'withdrew'): 1.0, ('ADJ', 'Lebanese'): 5.0, ('PROPN', 'May'): 13.0, ('PROPN', 'Hizbullah'): 2.0, ('VERB', 'mollified'): 1.0, ('VERB', 'estimated'): 2.0, ('NUM', '5,000'): 4.0, ('NOUN', 'fighters'): 1.0, ('VERB', 'pursued'): 1.0, ('VERB', 'compel'): 1.0, ('VERB', 'withdraw'): 3.0, ('PROPN', 'Shebaa'): 1.0, ('PROPN', 'Farms'): 1.0, ('NOUN', 'sliver'): 1.0, ('VERB', 'annexed'): 1.0, ('NUM', '1967'): 1.0, ('DET', 'Any'): 2.0, ('ADJ', 'thorough'): 1.0, ('NOUN', 'assessment'): 1.0, ('NOUN', 'impact'): 3.0, ('NOUN', 'aftermath'): 2.0, ('NOUN', 'environment'): 3.0, ('ADV', 'therefore'): 3.0, ('ADV', 'closely'): 5.0, ('VERB', 'examine'): 3.0, ('NOUN', 'conduct'): 1.0, ('NOUN', 'warfare'): 2.0, ('ADV', 'often'): 6.0, ('VERB', 'gave'): 3.0, ('ADJ', 'clear'): 13.0, ('NOUN', 'danger'): 4.0, ('PRON', 'themselves'): 10.0, ('VERB', 'end'): 3.0, ('NOUN', 'occupation'): 4.0, ('PROPN', 'Strip'): 1.0, ('VERB', 'driven'): 1.0, ('ADJ', 'economic'): 3.0, ('NOUN', 'considerations'): 1.0, ('ADJ', 'practical'): 1.0, ('ADP', 'At'): 11.0, ('NOUN', 'points'): 11.0, ('ADJ', 'late'): 7.0, ('NOUN', '1980s'): 3.0, ('ADV', 'reportedly'): 8.0, ('NOUN', 'scenes'): 1.0, ('NOUN', 'overtures'): 1.0, ('VERB', 'arrive'): 3.0, ('NOUN', 'deal'): 10.0, ('VERB', 'allow'): 3.0, ('VERB', 'launch'): 1.0, ('NOUN', '1990s'): 1.0, ('ADJ', 'biological'): 12.0, ('NOUN', 'program'): 12.0, ('VERB', 'destroyed'): 7.0, ('ADJ', 'chemical'): 4.0, ('NOUN', 'stockpiles'): 1.0, ('PRON', 'Its'): 5.0, ('ADJ', 'ramshackle'): 1.0, ('VERB', 'collapsed'): 1.0, ('ADP', 'before'): 13.0, ('NOUN', 'invasion'): 2.0, ('NUM', '2003'): 16.0, ('ADJ', 'Baathist'): 1.0, ('ADJ', 'difficult'): 3.0, ('NOUN', 'menace'): 1.0, ('NOUN', 'bungling'): 1.0, ('ADJ', 'post-war'): 3.0, ('VERB', 'created'): 3.0, ('ADJ', 'weak'): 2.0, ('VERB', 'failed'): 4.0, ('ADJ', 'Armed'): 1.0, ('NOUN', 'militias'): 2.0, ('VERB', 'staffed'): 2.0, ('NOUN', 'men'): 7.0, ('NOUN', 'training'): 11.0, ('VERB', 'proliferated'): 1.0, ('VERB', 'chose'): 1.0, ('VERB', 'ally'): 1.0, ('PRON', 'itself'): 5.0, ('ADJ', 'such'): 26.0, ('NOUN', 'groups'): 10.0, ('PROPN', 'Supreme'): 1.0, ('PROPN', 'Islamic'): 19.0, ('PROPN', 'Revolution'): 1.0, ('NUM', '15,000'): 1.0, ('PROPN', 'Badr'): 1.0, ('PROPN', 'Corps'): 2.0, ('NOUN', 'paramilitary'): 1.0, ('VERB', 'trained'): 4.0, ('PROPN', 'Iranian'): 1.0, ('PROPN', 'Revolutionary'): 2.0, ('PROPN', 'Guards'): 1.0, ('ADJ', 'Anti-Israeli'): 1.0, ('ADJ', 'pro-Palestinian'): 1.0, ('NOUN', 'feeling'): 2.0, ('ADJ', 'ideological'): 1.0, ('NOUN', 'currents'): 2.0, ('ADV', 'generally'): 1.0, ('VERB', 'follow'): 6.0, ('ADJ', 'theocratic'): 1.0, ('NOUN', 'notions'): 1.0, ('PROPN', 'Iran'): 27.0, ('PROPN', 'Ayatollah'): 3.0, ('PROPN', 'Ruhollah'): 1.0, ('PROPN', 'Khomeini'): 1.0, ('ADV', 'routinely'): 1.0, ('VERB', 'chant'): 1.0, ('VERB', 'demonstrate'): 1.0, ('ADV', 'vehemently'): 1.0, ('VERB', 'protested'): 1.0, ('PROPN', 'Sheikh'): 7.0, ('PROPN', 'Yassin'): 1.0, ('PROPN', 'March'): 12.0, ('ADJ', 'Worse'): 1.0, ('VERB', 'drew'): 1.0, ('NOUN', 'denunciation'): 1.0, ('ADJ', 'moderate'): 4.0, ('ADJ', 'cautious'): 1.0, ('PROPN', 'Grand'): 2.0, ('PROPN', 'Sistani'): 5.0, ('VERB', 'wields'): 1.0, ('ADJ', 'moral'): 2.0, ('NOUN', 'authority'): 2.0, ('VERB', 'suppressed'): 2.0, ('NOUN', 'regime'): 11.0, ('VERB', 'organized'): 1.0, ('VERB', 'armed'): 1.0, ('VERB', 'reestablished'): 1.0, ('ADJ', 'historical'): 5.0, ('ADJ', 'Iranian'): 11.0, ('ADJ', 'inevitable'): 2.0, ('VERB', 'side'): 1.0, ('NOUN', 'coreligionists'): 1.0, ('ADJ', 'rich'): 4.0, ('ADV', 'enough'): 2.0, ('NOUN', 'petroleum'): 2.0, ('NOUN', 'sales'): 1.0, ('NOUN', 'position'): 7.0, ('VERB', 'bankroll'): 1.0, ('NOUN', 'fundamentalists'): 3.0, ('ADV', 'deeply'): 3.0, ('VERB', 'sympathize'): 2.0, ('VERB', 'deep'): 1.0, ('NOUN', 'inks'): 1.0, ('PROPN', 'Jordan'): 5.0, ('PROPN', 'Palestine'): 2.0, ('NOUN', 'cities'): 3.0, ('NOUN', 'truck'): 2.0, ('NOUN', 'route'): 2.0, ('PROPN', 'Amman'): 1.0, ('NOUN', 'influence'): 4.0, ('PROPN', 'Salafi'): 1.0, ('NOUN', 'movement'): 4.0, ('ADJ', 'popular'): 6.0, ('ADJ', 'Secular'): 1.0, ('NOUN', 'nationalist'): 1.0, ('ADV', 'universally'): 1.0, ('ADJ', 'post-Saddam'): 1.0, ('NOUN', 'exception'): 2.0, ('SCONJ', 'Whereas'): 1.0, ('VERB', 'ensured'): 2.0, ('ADJ', 'populist'): 1.0, ('VERB', 'kept'): 2.0, ('ADV', 'firmly'): 3.0, ('VERB', 'organize'): 2.0, ('DET', 'An'): 9.0, ('ADJ', 'nationalist'): 1.0, ('VERB', 'proliferate'): 1.0, ('ADV', 'inevitably'): 3.0, ('NOUN', 'worry'): 1.0, ('NOUN', 'modicum'): 1.0, ('NOUN', 'normality'): 1.0, ('NOUN', 'citizens'): 4.0, ('VERB', 'benefit'): 2.0, ('NOUN', 'reserves'): 1.0, ('DET', 'That'): 5.0, ('ADJ', 'private'): 4.0, ('VERB', 'funneled'): 1.0, ('NOUN', 'aid'): 5.0, ('NOUN', 'interests'): 2.0, ('ADV', 'best'): 2.0, ('VERB', 'served'): 7.0, ('NOUN', 'neighbors'): 1.0, ('VERB', 'achieved'): 2.0, ('VERB', 'trading'): 1.0, ('ADJ', 'aggressive'): 1.0, ('ADJ', 'near'): 1.0, ('NOUN', 'annexation'): 1.0, ('VERB', 'occupied'): 1.0, ('ADJ', 'indefinite'): 2.0, ('NOUN', 'postponement'): 1.0, ('ADJ', 'unprecedented'): 2.0, ('NOUN', 'rage'): 1.0, ('NOUN', 'violence'): 4.0, ('VERB', 'spread'): 1.0, ('ADP', 'throughout'): 2.0, ('NOUN', 'promotion'): 1.0, ('ADJ', 'pro-Zionist'): 1.0, ('NOUN', 'right'): 6.0, ('NOUN', 'twin'): 1.0, ('NOUN', 'occupations'): 1.0, ('ADV', 'profoundly'): 1.0, ('VERB', 'strengthened'): 2.0, ('X', 'www.juancole.com'): 1.0, ('NOUN', 'professor'): 2.0, ('ADJ', 'modern'): 1.0, ('PROPN', 'Middle'): 4.0, ('PROPN', 'East'): 4.0, ('PROPN', 'Michigan'): 1.0, ('NOUN', 'author'): 3.0, ('PROPN', 'Sacred'): 1.0, ('PROPN', 'Space'): 1.0, ('PROPN', 'Holy'): 1.0, ('PROPN', 'War'): 5.0, ('PROPN', 'I.B.'): 1.0, ('PROPN', 'Tauris'): 1.0, ('PROPN', 'DAILY'): 1.0, ('PROPN', 'STAR'): 1.0, ('VERB', 'publishes'): 1.0, ('NOUN', 'commentary'): 1.0, ('NOUN', 'agreement'): 12.0, ('PROPN', 'Agence'): 1.0, ('PROPN', 'Global'): 1.0, ('PROPN', 'Tamils'): 5.0, ('VERB', 'feel'): 8.0, ('VERB', 'proposed'): 4.0, ('NOUN', 'defense'): 12.0, ('PROPN', 'Sri'): 17.0, ('PROPN', 'Lanka'): 17.0, ('ADJ', 'Sinhala'): 1.0, ('NOUN', 'rulers'): 3.0, ('VERB', 'prepare'): 3.0, ('VERB', 'abandoning'): 2.0, ('NOUN', 'hints'): 2.0, ('X', 'Indo'): 3.0, ('VERB', 'drawn'): 3.0, ('NOUN', 'protests'): 2.0, ('PROPN', 'Tamil'): 12.0, ('PROPN', 'TNA'): 3.0, ('VERB', 'backed'): 5.0, ('PROPN', 'LTTE'): 22.0, ('ADV', 'Apart'): 2.0, ('NUM', '1,200'): 3.0, ('VERB', 'lost'): 5.0, ('NUM', '1987'): 4.0, ('NOUN', 'peacekeeping'): 2.0, ('ADV', 'immensely'): 2.0, ('ADJ', 'unpopular'): 2.0, ('PROPN', 'Nadu'): 3.0, ('PROPN', 'Jaffna'): 4.0, ('NOUN', 'peninsula'): 4.0, ('ADJ', 'Sinhalese'): 3.0, ('NOUN', 'majority'): 6.0, ('VERB', 'considered'): 6.0, ('NOUN', 'violation'): 2.0, ('ADJ', 'defensive'): 4.0, ('PROPN', 'Ranjit'): 1.0, ('PROPN', 'Devraj'): 1.0, ('PROPN', 'NEW'): 1.0, ('PROPN', 'DELHI'): 1.0, ('SCONJ', 'While'): 4.0, ('ADJ', 'ready'): 3.0, ('VERB', 'enter'): 1.0, ('NOUN', 'cooperation'): 3.0, ('ADJ', 'wary'): 2.0, ('NOUN', 'involvement'): 3.0, ('NOUN', 'island'): 7.0, ('NOUN', 'decades'): 6.0, ('VERB', 'seen'): 5.0, ('ADJ', 'violent'): 4.0, ('NOUN', 'strife'): 2.0, ('ADJ', 'ethnic'): 5.0, ('VERB', 'leaving'): 6.0, ('NUM', '60,000'): 1.0, ('DET', 'both'): 8.0, ('NOUN', 'sides'): 3.0, ('VERB', 'explains'): 2.0, ('NOUN', 'delay'): 1.0, ('NOUN', 'signing'): 3.0, ('ADJ', 'formal'): 2.0, ('ADJ', 'Sri'): 2.0, ('ADJ', 'Lankan'): 2.0, ('PROPN', 'Chandrika'): 1.0, ('PROPN', 'Kumaratunga'): 5.0, ('NOUN', 'visit'): 3.0, ('VERB', 'According'): 13.0, ('PROPN', 'Professor'): 1.0, ('PROPN', 'S'): 2.0, ('PROPN', 'D'): 1.0, ('PROPN', 'Muni'): 2.0, ('NOUN', 'expert'): 2.0, ('PROPN', 'Jawaharal'): 1.0, ('PROPN', 'Nehru'): 1.0, ('NOUN', 'talks'): 6.0, ('PROPN', 'Colombo'): 6.0, ('PROPN', 'Liberation'): 3.0, ('PROPN', 'Tigers'): 12.0, ('PROPN', 'Eelam'): 2.0, ('VERB', 'stalemated'): 1.0, ('ADP', 'For'): 8.0, ('NOUN', 'reason'): 8.0, ('ADJ', 'keen'): 2.0, ('VERB', 'beef'): 1.0, ('NOUN', 'preparedness'): 1.0, ('NOUN', 'support'): 10.0, ('VERB', 'want'): 17.0, ('VERB', 'initiate'): 2.0, ('NOUN', 'conflict'): 9.0, ('VERB', 'deterring'): 1.0, ('VERB', 'starting'): 3.0, ('VERB', 'look'): 13.0, ('NOUN', 'brink'): 1.0, ('VERB', 'launching'): 1.0, ('NOUN', 'offensive'): 2.0, ('VERB', 'told'): 18.0, ('PROPN', 'IPS'): 2.0, ('NUM', 'six'): 4.0, ('NOUN', 'rounds'): 2.0, ('NOUN', 'rebels'): 6.0, ('ADV', 'abruptly'): 1.0, ('VERB', 'pulled'): 2.0, ('NOUN', 'negotiations'): 1.0, ('VERB', 'demanding'): 1.0, ('NOUN', 'recognition'): 1.0, ('ADV', 'first'): 11.0, ('NOUN', 'self'): 4.0, ('VERB', 'proceeding'): 1.0, ('VERB', 'preceded'): 1.0, ('ADJ', 'Norwegian'): 3.0, ('PROPN', 'Foreign'): 1.0, ('PROPN', 'Jan'): 1.0, ('PROPN', 'Petersen'): 2.0, ('NOUN', 'bid'): 1.0, ('VERB', 'revive'): 1.0, ('VERB', 'supposed'): 3.0, ('NOUN', 'ceasefire'): 1.0, ('ADV', 'successfully'): 3.0, ('VERB', 'brokered'): 1.0, ('PROPN', 'February'): 6.0, ('NOUN', 'discussions'): 1.0, ('CONJ', 'both'): 18.0, ('ADJ', 'reclusive'): 1.0, ('PROPN', 'Velupillai'): 1.0, ('PROPN', 'Prabhakaran'): 1.0, ('NOUN', 'rebel'): 2.0, ('NOUN', 'stronghold'): 1.0, ('PROPN', 'Kilinochchi'): 2.0, ('NOUN', 'success'): 1.0, ('ADJ', 'chief'): 3.0, ('ADJ', 'Tamil'): 1.0, ('NOUN', 'negotiator'): 1.0, ('PROPN', 'Anton'): 1.0, ('PROPN', 'Balasingham'): 2.0, ('NOUN', 'envoy'): 2.0, ('PROPN', 'Erik'): 1.0, ('PROPN', 'Solheim'): 1.0, ('VERB', 'closed'): 2.0, ('NOUN', 'door'): 2.0, ('NOUN', 'airport'): 4.0, ('ADV', 'late'): 2.0, ('PROPN', 'Saturday'): 2.0, ('NOUN', 'effort'): 5.0, ('NOUN', 'salvage'): 1.0, ('NOUN', 'track'): 1.0, ('ADJ', 'diplomatic'): 1.0, ('ADJ', 'intractable'): 1.0, ('PROPN', 'A'): 2.0, ('PROPN', 'Kalkat'): 6.0, ('NOUN', 'difficulty'): 1.0, ('VERB', 'lay'): 1.0, ('NOUN', 'fact'): 20.0, ('X', 'de'): 1.0, ('X', 'jure'): 1.0, ('NOUN', 'east'): 2.0, ('DET', 'every'): 10.0, ('NOUN', 'aspect'): 3.0, ('NOUN', 'veteran'): 1.0, ('NOUN', 'intervention'): 1.0, ('VERB', 'implement'): 3.0, ('PROPN', 'Peace'): 1.0, ('PROPN', 'Accord'): 3.0, ('ADV', 'ambitiously'): 1.0, ('VERB', 'provided'): 6.0, ('NOUN', 'disarming'): 1.0, ('ADJ', 'formidable'): 2.0, ('NOUN', 'reiteration'): 1.0, ('ADJ', 'older'): 4.0, ('CONJ', 'minus'): 1.0, ('NOUN', 'commitment'): 2.0, ('ADV', 'currently'): 2.0, ('VERB', 'chairs'): 1.0, ('ADJ', 'independent'): 3.0, ('PROPN', 'Conflict'): 1.0, ('PROPN', 'Resolution'): 1.0, ('NOUN', 'failure'): 4.0, ('VERB', 'disarm'): 1.0, ('VERB', 'subdue'): 1.0, ('VERB', 'remained'): 1.0, ('ADJ', 'only'): 13.0, ('ADJ', 'capable'): 2.0, ('VERB', 'influencing'): 1.0, ('NOUN', 'course'): 2.0, ('PROPN', 'Norwegians'): 1.0, ('VERB', 'mean'): 4.0, ('ADJ', 'limited'): 3.0, ('ADJ', 'honest'): 2.0, ('NOUN', 'broker'): 1.0, ('ADV', 'keenly'): 1.0, ('ADJ', 'aware'): 2.0, ('ADP', 'unlike'): 1.0, ('VERB', 'underwrite'): 1.0, ('NOUN', 'arrangement'): 1.0, ('ADV', 'reluctantly'): 1.0, ('VERB', 'accepted'): 7.0, ('NOUN', 'accord'): 2.0, ('ADP', 'Under'): 2.0, ('ADJ', 'northeastern'): 4.0, ('ADJ', 'provincial'): 1.0, ('NOUN', 'council'): 2.0, ('VERB', 'formed'): 2.0, ('VERB', 'deployed'): 2.0, ('NOUN', 'peacekeepers'): 1.0, ('NOUN', 'differences'): 2.0, ('ADV', 'soon'): 6.0, ('VERB', 'surfaced'): 1.0, ('VERB', 'led'): 11.0, ('NOUN', 'clashes'): 1.0, ('PROPN', 'Tiger'): 1.0, ('NOUN', 'keeping'): 1.0, ('ADV', 'About'): 1.0, ('NOUN', 'phase'): 1.0, ('NUM', '1989'): 1.0, ('PROPN', 'Ranasinghe'): 1.0, ('PROPN', 'Premadasa'): 1.0, ('NOUN', 'critic'): 1.0, ('NOUN', 'mediation'): 1.0, ('ADJ', 'Last'): 2.0, ('NOUN', 'initiative'): 2.0, ('PROPN', 'Japan'): 2.0, ('VERB', 'persuade'): 1.0, ('NOUN', 'negotiating'): 1.0, ('NOUN', 'table'): 1.0, ('NOUN', 'package'): 1.0, ('NOUN', 'offer'): 2.0, ('SYM', 'US$'): 1.0, ('NUM', '4.5'): 1.0, ('NUM', 'billion'): 2.0, ('ADJ', 'special'): 2.0, ('PROPN', 'Yasushi'): 1.0, ('PROPN', 'Akashi'): 1.0, ('ADJ', 'tangible'): 1.0, ('NOUN', 'visits'): 3.0, ('ADJ', 'frustrated'): 1.0, ('NOUN', 'man'): 11.0, ('VERB', 'complained'): 1.0, ('VERB', 'reaching'): 2.0, ('NOUN', 'impasse'): 1.0, ('NOUN', 'ideologue'): 1.0, ('VERB', 'sniffed'): 1.0, ('NOUN', 'proposal'): 1.0, ('NOUN', 'solution'): 9.0, ('VERB', 'predetermined'): 1.0, ('NOUN', 'resolutions'): 1.0, ('NOUN', 'declarations'): 1.0, ('NOUN', 'donor'): 1.0, ('NOUN', 'conferences'): 4.0, ('VERB', 'negotiated'): 2.0, ('NOUN', 'constraints'): 1.0, ('ADJ', 'external'): 1.0, ('PROPN', 'Sinhala'): 1.0, ('PROPN', 'P'): 1.0, ('PROPN', 'Sithamparanathan'): 1.0, ('VERB', 'quoted'): 4.0, ('PRON', 'She'): 3.0, ('NOUN', 'brass'): 2.0, ('NOUN', 'chief'): 4.0, ('PROPN', 'Nirmal'): 1.0, ('PROPN', 'Chander'): 1.0, ('PROPN', 'Vij'): 1.0, ('NOUN', 'apprehension'): 1.0, ('VERB', 'pointed'): 1.0, ('ADV', 'ill'): 1.0, ('VERB', 'advised'): 1.0, ('SCONJ', 'because'): 20.0, ('VERB', 'consider'): 4.0, ('NOUN', 'sentiments'): 1.0, ('NUM', '45'): 2.0, ('NUM', 'million'): 7.0, ('VERB', 'separated'): 3.0, ('ADJ', 'narrow'): 1.0, ('PROPN', 'Palk'): 1.0, ('PROPN', 'Straits'): 2.0, ('NOUN', 'option'): 4.0, ('ADJ', 'present'): 6.0, ('NOUN', 'circumstances'): 2.0, ('ADJ', 'dirty'): 1.0, ('PROPN', 'New'): 22.0, ('PROPN', 'Delhi'): 13.0, ('ADV', 'always'): 18.0, ('VERB', 'counted'): 1.0, ('VERB', 'render'): 1.0, ('ADJ', 'neighborly'): 1.0, ('NOUN', 'help'): 3.0, ('VERB', 'shared'): 3.0, ('NOUN', 'belief'): 4.0, ('NOUN', 'religion'): 4.0, ('NOUN', 'ethnicity'): 1.0, ('NOUN', 'basis'): 4.0, ('NOUN', 'secession'): 1.0, ('VERB', 'puts'): 1.0, ('ADV', 'succinctly'): 1.0, ('SCONJ', 'to'): 10.0, ('PROPN', 'Times'): 5.0, ('PROPN', 'Online'): 1.0, ('NUM', '16/11/2004'): 1.0, ('VERB', 'Unreported'): 2.0, ('NOUN', 'media'): 20.0, ('PROPN', 'Valley'): 5.0, ('ADJ', 'cultural'): 3.0, ('NOUN', 'genocide'): 2.0, ('VERB', 'resulted'): 3.0, ('NOUN', 'fleeing'): 2.0, ('NOUN', 'valley'): 5.0, ('ADJ', 'Hindu'): 4.0, ('SCONJ', 'since'): 14.0, ('ADJ', 'human'): 9.0, ('NOUN', 'habitation'): 2.0, ('VERB', 'recorded'): 3.0, ('ADP', 'Over'): 2.0, ('NUM', 'nine'): 3.0, ('NOUN', 'dozen'): 2.0, ('NOUN', 'temples'): 2.0, ('SCONJ', 'with'): 13.0, ('VERB', 'used'): 26.0, ('NOUN', 'building'): 3.0, ('NOUN', 'material'): 2.0, ('NOUN', 'urinals'): 2.0, ('ADV', 'Thus'): 3.0, ('NOUN', 'none'): 3.0, ('NOUN', 'busybodies'): 2.0, ('ADP', 'across'): 6.0, ('VERB', 'bothered'): 3.0, ('VERB', 'notice'): 3.0, ('NOUN', 'development'): 4.0, ('VERB', 'calls'): 2.0, ('NOUN', 'bluff'): 2.0, ('PROPN', 'M.D.'): 2.0, ('PROPN', 'Nalapat'): 2.0, ('NOUN', 'secretaries'): 1.0, ('ADP', 'save'): 1.0, ('ADV', 'perhaps'): 12.0, ('PROPN', 'Dean'): 4.0, ('PROPN', 'Rusk'): 1.0, ('VERB', 'gobbled'): 1.0, ('NOUN', 'credit'): 1.0, ('NOUN', 'outcomes'): 1.0, ('ADJ', 'brazen'): 1.0, ('PROPN', 'Colin'): 5.0, ('PROPN', 'Powell'): 5.0, ('ADJ', 'surprised'): 1.0, ('NOUN', 'alliance'): 2.0, ('PROPN', 'Maulana'): 1.0, ('PROPN', 'Fazlur'): 1.0, ('PROPN', 'Rahman'): 7.0, ('VERB', 'visited'): 1.0, ('VERB', 'issued'): 4.0, ('NOUN', 'series'): 1.0, ('ADJ', 'conciliatory'): 2.0, ('NOUN', 'statements'): 6.0, ('VERB', 'telling'): 4.0, ('VERB', 'forced'): 4.0, ('SCONJ', 'into'): 3.0, ('ADJ', 'hawkish'): 1.0, ('ADV', 'precisely'): 3.0, ('NOUN', 'likes'): 1.0, ('NOUN', 'embarrassment'): 3.0, ('NOUN', 'reality'): 5.0, ('NOUN', 'choice'): 3.0, ('NOUN', 'distinction'): 4.0, ('NOUN', 'Realists'): 1.0, ('VERB', 'includes'): 6.0, ('ADJ', 'Democratic'): 1.0, ('NOUN', 'policy'): 4.0, ('NOUN', 'establishment'): 1.0, ('PROPN', 'Strobe'): 1.0, ('PROPN', 'Talbott'): 1.0, ('ADV', 'long'): 5.0, ('VERB', 'sought'): 6.0, ('VERB', 'divest'): 1.0, ('NOUN', 'capability'): 2.0, ('ADJ', 'feasible'): 3.0, ('NOUN', 'acceptance'): 2.0, ('NOUN', 'status'): 3.0, ('NOUN', 'quo'): 2.0, ('VERB', 'keeps'): 1.0, ('PROPN', 'China'): 8.0, ('VERB', 'gifted'): 1.0, ('NOUN', 'slice'): 1.0, ('ADV', 'Simultaneously'): 1.0, ('VERB', 'ensure'): 3.0, ('NOUN', 'degree'): 2.0, ('NOUN', 'autonomy'): 2.0, ('VERB', 'cut'): 1.0, ('NOUN', 'jihadis'): 3.0, ('VERB', 'attempting'): 2.0, ('VERB', 'convert'): 1.0, ('ADJ', 'second'): 10.0, ('VERB', 'understood'): 2.0, ('NOUN', 'end'): 8.0, ('NOUN', 'term'): 3.0, ('CONJ', 'yet'): 5.0, ('NOUN', 'affinity'): 1.0, ('NOUN', 'generals'): 2.0, ('NOUN', 'pendulum'): 1.0, ('VERB', 'swung'): 1.0, ('ADP', 'toward'): 5.0, ('ADJ', 'quixotic'): 1.0, ('VERB', 'prize'): 1.0, ('ADV', 'loose'): 2.0, ('NOUN', 'lobbyist'): 1.0, ('PROPN', 'Christina'): 1.0, ('PROPN', 'Rocca'): 1.0, ('NOUN', 'minimum'): 3.0, ('VERB', 'accept'): 5.0, ('VERB', 'deliver'): 3.0, ('NOUN', 'writer'): 1.0, ('VERB', 'regarded'): 1.0, ('NOUN', 'trifurcation'): 1.0, ('NOUN', 'part'): 21.0, ('PROPN', 'Jammu'): 1.0, ('PROPN', 'Buddhist'): 1.0, ('VERB', 'dominated'): 2.0, ('PROPN', 'Ladakh'): 1.0, ('NOUN', 'little'): 1.0, ('NOUN', 'taxpayers'): 1.0, ('ADP', 'With'): 4.0, ('NOUN', 'luck'): 1.0, ('VERB', 'attract'): 1.0, ('NOUN', 'locations'): 1.0, ('NOUN', 'tourist'): 2.0, ('NOUN', 'education'): 4.0, ('NOUN', 'services'): 3.0, ('NOUN', 'haven'): 2.0, ('ADJ', 'happy'): 4.0, ('NOUN', 'outcome'): 2.0, ('ADJ', 'tiny'): 1.0, ('ADJ', 'jihadi'): 1.0, ('NOUN', 'segment'): 1.0, ('VERB', 'patronized'): 1.0, ('PROPN', 'State'): 2.0, ('PROPN', 'Department'): 5.0, ('PROPN', 'Army'): 5.0, ('VERB', 'like'): 17.0, ('VERB', 'even'): 1.0, ('NOUN', 'score'): 1.0, ('ADJ', 'catastrophic'): 1.0, ('NOUN', 'defeat'): 1.0, ('PROPN', 'Bangladesh'): 6.0, ('NUM', '1971'): 7.0, ('PROPN', 'Hindu'): 1.0, ('VERB', 'filled'): 1.0, ('NOUN', 'tales'): 1.0, ('NOUN', 'atrocities'): 2.0, ('ADJ', 'innocent'): 4.0, ('ADP', 'Despite'): 3.0, ('PROPN', 'Sept.'): 1.0, ('VERB', 'supports'): 3.0, ('NOUN', 'jihad'): 5.0, ('NOUN', 'price'): 2.0, ('VERB', 'paying'): 2.0, ('ADV', 'Unfortunately'): 3.0, ('VERB', 'decided'): 3.0, ('ADV', 'seriously'): 5.0, ('ADJ', 'frequent'): 3.0, ('NOUN', 'boasts'): 1.0, ('NOUN', 'nudging'): 1.0, ('ADV', 'that'): 6.0, ('PROPN', 'Indians'): 2.0, ('NOUN', 'gestures'): 1.0, ('PROPN', 'Islamabad'): 6.0, ('VERB', 'misnamed'): 1.0, ('NOUN', 'think'): 1.0, ('NOUN', 'tanks'): 1.0, ('NOUN', 'lead'): 2.0, ('ADJ', 'numerous'): 3.0, ('VERB', 'emerged'): 2.0, ('VERB', 'prized'): 1.0, ('NOUN', 'version'): 2.0, ('PROPN', 'Ibrahim'): 1.0, ('PROPN', 'Rugova'): 1.0, ('SCONJ', 'That'): 6.0, ('PROPN', 'Yugoslavia'): 1.0, ('NOUN', 'foreigner'): 1.0, ('PROPN', 'Indian'): 2.0, ('VERB', 'afford'): 2.0, ('VERB', 'ignore'): 1.0, ('NOUN', 'nationalism'): 1.0, ('VERB', 'struck'): 1.0, ('NOUN', 'conferees'): 1.0, ('NOUN', 'scholars'): 1.0, ('NOUN', 'analysts'): 1.0, ('VERB', 'endorse'): 1.0, ('NOUN', 'sake'): 2.0, ('NOUN', 'trip'): 1.0, ('PROPN', 'York'): 8.0, ('PROPN', 'Vienna'): 1.0, ('NOUN', 'wrinkles'): 1.0, ('VERB', 'remains'): 4.0, ('ADV', 'merely'): 4.0, ('VERB', 'signing'): 1.0, ('NOUN', 'piece'): 5.0, ('NOUN', 'paper'): 5.0, ('VERB', 'writing'): 1.0, ('NOUN', 'op'): 1.0, ('NOUN', 'ed'): 1.0, ('PROPN', 'Sonia'): 1.0, ('PROPN', 'Gandhi'): 1.0, ('PROPN', 'Atal'): 1.0, ('PROPN', 'Behari'): 1.0, ('PROPN', 'Vajpayee'): 2.0, ('ADV', 'instantly'): 1.0, ('NOUN', 'business'): 13.0, ('NOUN', 'community'): 2.0, ('VERB', 'indulged'): 1.0, ('ADJ', 'empty'): 1.0, ('NOUN', 'bout'): 1.0, ('NOUN', 'saber'): 1.0, ('NOUN', 'rattling'): 1.0, ('VERB', 'pointing'): 2.0, ('NOUN', 'beneficiary'): 1.0, ('ADJ', 'mythical'): 1.0, ('NOUN', 'perception'): 1.0, ('ADV', 'especially'): 4.0, ('NOUN', 'corner'): 2.0, ('VERB', 'emerging'): 4.0, ('ADJ', 'alternative'): 4.0, ('NOUN', 'destination'): 1.0, ('ADV', 'hence'): 1.0, ('ADJ', 'trusty'): 1.0, ('NOUN', 'supplier'): 1.0, ('NOUN', 'nukes'): 1.0, ('NOUN', 'missiles'): 1.0, ('NOUN', 'scare'): 1.0, ('VERB', 'involving'): 5.0, ('NOUN', 'flashpoints'): 1.0, ('PROPN', 'North'): 5.0, ('PROPN', 'Korea'): 1.0, ('ADP', 'Since'): 2.0, ('VERB', 'talked'): 2.0, ('VERB', 'signaling'): 1.0, ('ADJ', 'acceptable'): 1.0, ('NOUN', 'action'): 4.0, ('VERB', 'followed'): 3.0, ('NOUN', 'disputes'): 1.0, ('ADJ', 'eager'): 1.0, ('ADV', 'indefinitely'): 2.0, ('VERB', 'carry'): 3.0, ('ADJ', 'lucrative'): 1.0, ('NOUN', 'resolution'): 1.0, ('VERB', 'challenging'): 2.0, ('VERB', 'implying'): 1.0, ('VERB', 'jump'): 1.0, ('NOUN', 'commands'): 1.0, ('SCONJ', 'Once'): 2.0, ('PROPN', 'Manmohan'): 1.0, ('PROPN', 'Singh'): 1.0, ('VERB', 'shows'): 1.0, ('NOUN', 'appetite'): 1.0, ('VERB', 'face'): 3.0, ('NOUN', 'truth'): 6.0, ('VERB', 'ramp'): 2.0, ('NOUN', 'insurgency'): 3.0, ('VERB', 'spawn'): 1.0, ('ADJ', 'fresh'): 2.0, ('NOUN', 'killers'): 4.0, ('PROPN', 'Mumbai'): 1.0, ('PROPN', 'London'): 3.0, ('PROPN', 'Chicago'): 1.0, ('VERB', 'shown'): 2.0, ('VERB', 'understands'): 3.0, ('NOUN', 'realities'): 1.0, ('PROPN', 'Hu'): 1.0, ('PROPN', 'Jintao'): 1.0, ('PROPN', 'Beijing'): 1.0, ('VERB', 'opened'): 1.0, ('PROPN', 'Wen'): 1.0, ('PROPN', 'Jiabao'): 1.0, ('VERB', 'expected'): 1.0, ('VERB', 'visit'): 5.0, ('ADV', 'early'): 2.0, ('ADJ', 'next'): 13.0, ('VERB', 'making'): 7.0, ('NOUN', 'friends'): 8.0, ('ADV', 'formally'): 1.0, ('VERB', 'accepting'): 2.0, ('PROPN', 'Sikkim'): 1.0, ('VERB', 'backing'): 2.0, ('ADJ', 'permanent'): 3.0, ('PROPN', 'U.N.'): 2.0, ('ADV', 'thus'): 2.0, ('NUM', 'five'): 3.0, ('ADV', 'yet'): 9.0, ('SCONJ', 'With'): 2.0, ('ADV', 'around'): 7.0, ('NOUN', 'need'): 2.0, ('PROPN', 'Osama'): 4.0, ('NOUN', 'geopolitics'): 1.0, ('PROPN', 'Manipal'): 1.0, ('PROPN', 'Academy'): 1.0, ('PROPN', 'Higher'): 1.0, ('NUM', '28/10/2004'): 1.0, ('INTJ', 'Ok'): 1.0, ('PROPN', 'Kerry'): 14.0, ('NUM', 'Four'): 2.0, ('NOUN', 'advantage'): 2.0, ('ADV', 'little'): 3.0, ('VERB', 'known'): 14.0, ('VERB', 'invoked'): 2.0, ('NOUN', 'regulation'): 1.0, ('VERB', 'desert'): 1.0, ('NOUN', 'band'): 4.0, ('NOUN', 'brothers'): 1.0, ('PROPN', 'Dan'): 1.0, ('PROPN', 'Rather'): 1.0, ('VERB', 'libeled'): 1.0, ('VERB', 'impugned'): 1.0, ('NOUN', 'service'): 16.0, ('PROPN', 'Air'): 8.0, ('PROPN', 'Guard'): 18.0, ('NUM', '60'): 2.0, ('PROPN', 'Minutes'): 1.0, ('NUM', 'II'): 1.0, ('VERB', 'forged'): 1.0, ('NOUN', 'documents'): 3.0, ('ADP', 'During'): 4.0, ('VERB', 'flew'): 2.0, ('PROPN', 'F'): 4.0, ('NUM', '102'): 4.0, ('PROPN', 'Delta'): 1.0, ('PROPN', 'Dagger'): 1.0, ('NOUN', 'fighter'): 5.0, ('NOUN', 'interceptor'): 4.0, ('VERB', 'saw'): 3.0, ('NOUN', 'theater'): 3.0, ('NUM', '1962'): 1.0, ('PROPN', 'December'): 3.0, ('NUM', '1969'): 4.0, ('NOUN', 'squadrons'): 1.0, ('PROPN', 'Tan'): 1.0, ('PROPN', 'Son'): 1.0, ('PROPN', 'Nhut'): 1.0, ('PROPN', 'Da'): 1.0, ('PROPN', 'Nang'): 1.0, ('PROPN', 'Bien'): 1.0, ('PROPN', 'Hoa'): 1.0, ('PROPN', 'Udorn'): 1.0, ('PROPN', 'Don'): 1.0, ('PROPN', 'Muang'): 1.0, ('PROPN', 'Thailand'): 4.0, ('VERB', 'Click'): 1.0, ('ADV', 'here'): 16.0, ('NOUN', 'source'): 4.0, ('ADV', 'As'): 2.0, ('VERB', 'knew'): 2.0, ('NOUN', 'unit'): 6.0, ('VERB', 'transferred'): 3.0, ('NOUN', 'controversy'): 1.0, ('PRON', 'anyone'): 13.0, ('VERB', 'wanted'): 6.0, ('VERB', 'fly'): 3.0, ('NOUN', 'jets'): 3.0, ('VERB', 'let'): 11.0, ('VERB', 'flying'): 3.0, ('ADJ', 'supersonic'): 1.0, ('PRON', 'anybody'): 1.0, ('NOUN', 'reporters'): 3.0, ('ADJ', 'journalistic'): 1.0, ('NOUN', 'ethics'): 1.0, ('VERB', 'uncovered'): 1.0, ('NOUN', 'example'): 5.0, ('ADV', 'considerably'): 1.0, ('VERB', 'follows'): 2.0, ('NOUN', 'text'): 3.0, ('NOUN', 'e-mail'): 12.0, ('PRON', 'mine'): 2.0, ('NOUN', 'thanks'): 1.0, ('PROPN', 'Dave'): 1.0, ('PROPN', 'Manzano'): 1.0, ('NOUN', 'facts'): 3.0, ('PUNCT', '---------'): 1.0, ('SCONJ', 'Before'): 2.0, ('VERB', 'fall'): 2.0, ('PROPN', 'Dems'): 1.0, ('NOUN', 'spin'): 2.0, ('PROPN', 'Alabama'): 10.0, ('NOUN', 'physical'): 1.0, ('NOUN', 'daddy'): 1.0, ('VERB', 'got'): 4.0, ('NOUN', 'News'): 2.0, ('NOUN', 'coverage'): 4.0, ('VERB', 'tended'): 2.0, ('VERB', 'focus'): 4.0, ('ADJ', 'brief'): 1.0, ('NOUN', 'portion'): 2.0, ('PRON', 'everything'): 5.0, ('ADJ', 'else'): 4.0, ('NOUN', 'record'): 7.0, ('ADJ', 'full'): 5.0, ('ADJ', 'future'): 2.0, ('VERB', 'joined'): 3.0, ('NUM', '1968'): 3.0, ('ADV', 'Almost'): 2.0, ('ADV', 'immediately'): 3.0, ('VERB', 'began'): 3.0, ('ADJ', 'extended'): 1.0, ('NUM', 'Six'): 2.0, ('ADJ', 'basic'): 1.0, ('NUM', 'Fifty'): 1.0, ('NOUN', 'flight'): 3.0, ('NUM', 'Twenty'): 1.0, ('VERB', 'begin'): 2.0, ('NOUN', 'periods'): 1.0, ('NOUN', 'weekends'): 1.0, ('ADP', 'After'): 4.0, ('VERB', 'racking'): 1.0, ('NOUN', 'hundreds'): 4.0, ('NOUN', 'hours'): 5.0, ('VERB', 'accumulated'): 3.0, ('NOUN', 'requirements'): 2.0, ('NOUN', 'guardsmen'): 1.0, ('VERB', 'required'): 3.0, ('VERB', 'accumulate'): 1.0, ('VERB', 'meet'): 6.0, ('ADJ', 'yearly'): 1.0, ('NOUN', 'obligation'): 3.0, ('NOUN', 'records'): 3.0, ('ADV', 'earlier'): 5.0, ('VERB', 'earned'): 6.0, ('NUM', '253'): 1.0, ('ADV', 'thereafter'): 1.0, ('VERB', 'measured'): 1.0, ('NUM', '340'): 1.0, ('SYM', '-'): 13.0, ('NUM', '1970'): 5.0, ('NOUN', 'words'): 4.0, ('ADJ', 'enough'): 6.0, ('VERB', 'satisfy'): 1.0, ('ADJ', 'entire'): 9.0, ('NOUN', 'hitch'): 1.0, ('PROPN', 'acd'): 1.0, ('NUM', '137'): 1.0, ('NUM', '112'): 1.0, ('NUM', '1972'): 9.0, ('NOUN', 'numbers'): 3.0, ('VERB', 'indicate'): 3.0, ('VERB', 'showed'): 4.0, ('AUX', 'Did'): 2.0, ('VERB', 'brings'): 2.0, ('NOUN', 'focus'): 1.0, ('VERB', 'deserted'): 1.0, ('VERB', 'according'): 7.0, ('ADJ', 'anti-Bush'): 1.0, ('NOUN', 'filmmaker'): 1.0, ('PROPN', 'Michael'): 11.0, ('PROPN', 'Moore'): 2.0, ('ADJ', 'AWOL'): 2.0, ('PROPN', 'Terry'): 2.0, ('PROPN', 'McAuliffe'): 1.0, ('NOUN', 'chairman'): 1.0, ('NOUN', 'permission'): 2.0, ('VERB', 'work'): 12.0, ('PROPN', 'Senate'): 4.0, ('ADJ', 'superior'): 2.0, ('INTJ', 'OK'): 2.0, ('NOUN', 'Requests'): 1.0, ('ADJ', 'unusual'): 3.0, ('VERB', 'retired'): 4.0, ('PROPN', 'Col.'): 1.0, ('PROPN', 'William'): 3.0, ('PROPN', 'Campenni'): 2.0, ('NOUN', 'glut'): 1.0, ('NOUN', 'pilots'): 3.0, ('VERB', 'winding'): 1.0, ('PROPN', 'Force'): 2.0, ('NOUN', 'desk'): 1.0, ('NUM', \"'72\"): 2.0, ('NUM', \"'73\"): 1.0, ('NOUN', 'pilot'): 8.0, ('VERB', 'solve'): 1.0, ('NUM', '1973'): 4.0, ('NUM', '56'): 2.0, ('NOUN', 'requirement'): 2.0, ('ADV', 'Then'): 6.0, ('NOUN', 'plans'): 8.0, ('PROPN', 'Harvard'): 1.0, ('PROPN', 'Business'): 1.0, ('VERB', 'showing'): 4.0, ('ADV', 'frequently'): 1.0, ('PROPN', 'July'): 5.0, ('NUM', '1974'): 1.0, ('VERB', 'received'): 11.0, ('ADJ', 'honorable'): 2.0, ('NOUN', 'discharge'): 2.0, ('VERB', 'serving'): 4.0, ('ADJ', 'original'): 2.0, ('VERB', 'cover'): 2.0, ('NOUN', 'marks'): 1.0, ('NOUN', 'evaluation'): 3.0, ('ADV', 'clearly'): 7.0, ('VERB', 'stands'): 3.0, ('NOUN', 'notch'): 2.0, ('ADJ', 'natural'): 6.0, ('NOUN', 'contemporaries'): 1.0, ('ADV', 'exceptionally'): 1.0, ('ADJ', 'fine'): 2.0, ('ADJ', 'young'): 11.0, ('ADV', 'continually'): 1.0, ('VERB', 'flies'): 1.0, ('NOUN', 'intercept'): 1.0, ('NOUN', 'missions'): 1.0, ('VERB', 'increase'): 1.0, ('NOUN', 'proficiency'): 1.0, ('ADJ', 'exceptional'): 1.0, ('VERB', 'questioning'): 1.0, ('PROPN', 'Globe'): 1.0, ('PROPN', 'CBS'): 1.0, ('NOUN', 'outlets'): 3.0, ('PROPN', 'Democrats'): 2.0, ('ADV', 'spitting'): 1.0, ('ADJ', 'mad'): 3.0, ('PROPN', 'Swift'): 4.0, ('PROPN', 'Boat'): 1.0, ('PROPN', 'Veterans'): 1.0, ('PROPN', 'Truth'): 1.0, ('ADJ', 'reasonable'): 3.0, ('NOUN', 'Voters'): 1.0, ('ADV', 'perfectly'): 3.0, ('VERB', 'decide'): 1.0, ('SCONJ', 'whether'): 7.0, ('ADJ', 'important'): 7.0, ('NOUN', 'camp'): 2.0, ('VERB', 'blames'): 2.0, ('PROPN', 'boat'): 3.0, ('PROPN', 'veterans'): 2.0, ('PROPN', 'Swifties'): 1.0, ('VERB', 'gets'): 2.0, ('NOUN', 'sense'): 1.0, ('ADV', 'entirely'): 3.0, ('NOUN', 'reasons'): 1.0, ('VERB', 'noted'): 4.0, ('NOUN', 'passing'): 2.0, ('ADV', 'personally'): 3.0, ('VERB', 'questioned'): 2.0, ('NOUN', 'word'): 7.0, ('VERB', 'explain'): 3.0, ('ADP', 'whether'): 2.0, ('SCONJ', 'about'): 13.0, ('ADV', 'Earlier'): 1.0, ('ADV', 'Just'): 7.0, ('VERB', 'answer'): 1.0, ('NOUN', 'episode'): 2.0, ('NOUN', 'spotlight'): 1.0, ('NOUN', 'someday'): 1.0, ('VERB', 'release'): 1.0, ('PROPN', 'Byron'): 1.0, ('PROPN', 'White'): 2.0, ('PROPN', 'House'): 4.0, ('NOUN', 'correspondent'): 1.0, ('PROPN', 'Review'): 2.0, ('NOUN', 'column'): 1.0, ('PROPN', 'Hill'): 1.0, ('NOUN', 'atmosphere'): 1.0, ('ADJ', 'eerie'): 1.0, ('ADV', 'strikingly'): 1.0, ('ADJ', 'familiar'): 2.0, ('VERB', 'suspect'): 2.0, ('NOUN', 'streets'): 1.0, ('VERB', 'looming'): 1.0, ('NOUN', 'doubt'): 2.0, ('VERB', 'regard'): 2.0, ('NOUN', 'date'): 4.0, ('NUM', '30'): 5.0, ('PROPN', 'January'): 6.0, ('VERB', 'renewed'): 2.0, ('NOUN', 'hope'): 4.0, ('VERB', 'awaiting'): 1.0, ('VERB', 'dreading'): 1.0, ('ADJ', 'interim'): 1.0, ('VERB', 'promised'): 2.0, ('NOUN', 'measures'): 4.0, ('VERB', 'reduce'): 2.0, ('VERB', 'fail'): 1.0, ('NOUN', 'balloting'): 1.0, ('NOUN', 'centres'): 2.0, ('NOUN', 'voting'): 1.0, ('VERB', 'attacked'): 2.0, ('NOUN', 'picture'): 5.0, ('VERB', 'competing'): 1.0, ('VERB', 'seem'): 9.0, ('PROPN', 'Coalition'): 3.0, ('PROPN', 'Allawi'): 6.0, ('PROPN', 'Iraqiya'): 1.0, ('PROPN', 'Ayad'): 1.0, ('NOUN', 'ministers'): 1.0, ('ADV', 'quite'): 3.0, ('ADV', 'expectedly'): 1.0, ('NOUN', 'campaigning'): 2.0, ('VERB', 'handed'): 2.0, ('NUM', '100'): 5.0, ('NOUN', 'dollar'): 2.0, ('NOUN', 'gifts'): 1.0, ('NOUN', 'journalists'): 1.0, ('VERB', 'attending'): 1.0, ('NOUN', 'conference'): 4.0, ('NOUN', 'practice'): 2.0, ('ADJ', 'bad'): 4.0, ('NOUN', 'memories'): 1.0, ('PROPN', 'Naji'): 1.0, ('PROPN', 'Abbudi'): 2.0, ('VERB', 'affirmed'): 1.0, ('NOUN', 'claims'): 4.0, ('ADV', 'all'): 11.0, ('VERB', 'educating'): 1.0, ('PROPN', 'Shia'): 5.0, ('NOUN', 'merits'): 1.0, ('PROPN', 'Emminence'): 1.0, ('ADV', 'openly'): 4.0, ('NOUN', 'reference'): 1.0, ('VERB', 'abusing'): 1.0, ('ADJ', 'official'): 7.0, ('ADV', 'Again'): 3.0, ('VERB', 'written'): 1.0, ('NOUN', 'allegation'): 1.0, ('ADJ', 'intentional'): 1.0, ('PROPN', 'Hazim'): 2.0, ('PROPN', \"Sha'lan\"): 6.0, ('VERB', 'engaging'): 1.0, ('ADJ', 'shrill'): 1.0, ('PROPN', \"Ba'athist\"): 3.0, ('ADJ', 'double'): 2.0, ('NOUN', 'agent'): 4.0, ('PROPN', 'CIA'): 25.0, ('VERB', 'dismisses'): 1.0, ('NOUN', 'thief'): 1.0, ('NOUN', 'stooge'): 1.0, ('VERB', 'longs'): 1.0, ('NOUN', 'origins'): 2.0, ('VERB', 'defending'): 1.0, ('NOUN', 'remark'): 1.0, ('PROPN', 'Arabiya'): 1.0, ('VERB', 'embarrass'): 1.0, ('NOUN', 'viewers'): 1.0, ('VERB', 'roll'): 2.0, ('NOUN', 'floor'): 1.0, ('ADV', 'extremely'): 3.0, ('ADJ', 'amusing'): 1.0, ('VERB', 'watching'): 3.0, ('VERB', 'looking'): 9.0, ('ADJ', 'smug'): 1.0, ('ADJ', 'amused'): 1.0, ('VERB', 'contrasted'): 1.0, ('ADV', 'barely'): 2.0, ('VERB', 'keeping'): 2.0, ('VERB', 'swearing'): 1.0, ('NOUN', 'Fistfights'): 1.0, ('INTJ', 'please'): 3.0, ('NOUN', 'son'): 8.0, ('PROPN', \"Khaza'il\"): 1.0, ('PROPN', 'Diwaniya'): 1.0, ('VERB', 'replace'): 1.0, ('PROPN', 'M.S.'): 1.0, ('PROPN', 'Sahaf'): 3.0, ('ADJ', 'nonsensical'): 1.0, ('VERB', 'passed'): 2.0, ('NOUN', 'jokes'): 1.0, ('ADJ', 'Kurdish'): 2.0, ('PROPN', 'PUK'): 1.0, ('PROPN', 'KDP'): 1.0, ('ADP', 'outside'): 2.0, ('VERB', 'act'): 4.0, ('VERB', 'grab'): 1.0, ('VERB', 'vote'): 1.0, ('VERB', 'doubt'): 2.0, ('ADJ', 'conservative'): 2.0, ('ADV', 'surprisingly'): 1.0, ('VERB', 'rooting'): 1.0, ('PROPN', 'Communist'): 2.0, ('PROPN', 'party'): 3.0, ('NOUN', 'attempt'): 4.0, ('VERB', 'counter'): 1.0, ('PROPN', 'Islamists'): 1.0, ('ADJ', 'forthcoming'): 2.0, ('PROPN', 'Assembly'): 3.0, ('ADJ', 'largest'): 1.0, ('VERB', 'registered'): 2.0, ('ADJ', 'larger'): 6.0, ('PUNCT', '***'): 3.0, ('ADJ', 'Several'): 1.0, ('NOUN', 'candidates'): 1.0, ('VERB', 'targeted'): 4.0, ('NOUN', 'threats'): 2.0, ('ADJ', 'Sectarian'): 1.0, ('ADJ', 'highest'): 2.0, ('VERB', 'insurgents'): 1.0, ('VERB', 'attacking'): 1.0, ('PROPN', 'Husseiniyas'): 1.0, ('ADJ', 'interesting'): 3.0, ('NOUN', 'conversation'): 3.0, ('ADJ', 'aged'): 1.0, ('NOUN', 'taxi'): 4.0, ('VERB', 'live'): 8.0, ('NOUN', 'relatives'): 2.0, ('PROPN', 'Amiriya'): 1.0, ('SCONJ', 'After'): 3.0, ('VERB', 'asking'): 3.0, ('DET', 'which'): 2.0, ('VERB', 'belong'): 2.0, ('VERB', 'assessing'): 1.0, ('ADJ', 'sectarian'): 1.0, ('NOUN', 'background'): 1.0, ('VERB', 'hurling'): 1.0, ('NOUN', 'abuses'): 1.0, ('PROPN', 'Persians'): 1.0, ('PROPN', 'Majoos'): 1.0, ('NOUN', 'fire'): 6.0, ('NOUN', 'worshippers'): 1.0, ('ADJ', 'rabid'): 1.0, ('NOUN', 'dogs'): 1.0, ('NOUN', 'handful'): 2.0, ('NOUN', 'descriptions'): 1.0, ('VERB', 'described'): 5.0, ('NOUN', 'face'): 3.0, ('VERB', 'f*ed'): 1.0, ('NOUN', 'horse'): 1.0, ('VERB', 'dismissed'): 2.0, ('NOUN', 'thieves'): 1.0, ('NOUN', 'traitors'): 1.0, ('VERB', 'believed'): 4.0, ('ADJ', 'viable'): 1.0, ('NOUN', 'mess'): 1.0, ('NOUN', 'resistance'): 1.0, ('ADJ', 'commonsense'): 1.0, ('ADV', 'First'): 2.0, ('VERB', 'driving'): 2.0, ('VERB', 'fighting'): 2.0, ('NOUN', 'submission'): 1.0, ('VERB', 'contend'): 1.0, ('ADJ', 'impossible'): 2.0, ('VERB', 'hold'): 3.0, ('VERB', 'Leaving'): 1.0, ('ADV', 'aside'): 1.0, ('NOUN', 'views'): 3.0, ('VERB', 'conflicts'): 1.0, ('NOUN', 'examples'): 4.0, ('VERB', 'offered'): 4.0, ('ADV', 'eventually'): 2.0, ('VERB', 'leads'): 3.0, ('NOUN', 'opinion'): 5.0, ('ADV', 'above'): 2.0, ('VERB', 'planning'): 2.0, ('NOUN', 'timer'): 1.0, ('VERB', 'claimed'): 2.0, ('VERB', 'stage'): 1.0, ('NOUN', \"d'etat\"): 1.0, ('VERB', 'return'): 6.0, ('NUM', '10'): 3.0, ('NOUN', 'withdrawal'): 1.0, ('SCONJ', 'On'): 1.0, ('VERB', 'sensing'): 1.0, ('NOUN', 'incredulity'): 1.0, ('VERB', 'asserted'): 1.0, ('NOUN', 'parts'): 4.0, ('ADJ', 'central'): 5.0, ('NOUN', 'command'): 1.0, ('ADJ', 'seperate'): 1.0, ('NOUN', 'labels'): 1.0, ('NOUN', 'leaderships'): 1.0, ('ADJ', 'regional'): 10.0, ('VERB', 'concur'): 3.0, ('NUM', '18'): 2.0, ('NOUN', 'governorates'): 2.0, ('ADP', 'along'): 9.0, ('VERB', 'removed'): 1.0, ('ADJ', 'fatal'): 1.0, ('NOUN', 'misconception'): 1.0, ('NOUN', 'harm'): 2.0, ('ADV', 'anywhere'): 2.0, ('VERB', 'assumes'): 1.0, ('VERB', 'realises'): 1.0, ('VERB', 'call'): 3.0, ('SCONJ', 'despite'): 1.0, ('VERB', 'offer'): 2.0, ('NOUN', 'reconciliation'): 2.0, ('VERB', 'dissenting'): 1.0, ('ADJ', 'tribal'): 2.0, ('NOUN', 'Sheikhs'): 2.0, ('ADP', 'rather'): 3.0, ('ADJ', 'upper'): 1.0, ('ADV', 'effectively'): 2.0, ('VERB', 'root'): 1.0, ('NOUN', 'midst'): 2.0, ('NOUN', 'return'): 8.0, ('NOUN', 'promise'): 1.0, ('VERB', 'sharing'): 1.0, ('VERB', 'disenfranchised'): 1.0, ('VERB', 'abused'): 1.0, ('ADV', 'Very'): 1.0, ('PROPN', 'Anbar'): 3.0, ('ADJ', 'terrible'): 1.0, ('NOUN', 'blunder'): 1.0, ('ADV', 'accidentally'): 1.0, ('VERB', 'killing'): 8.0, ('PROPN', 'Razaq'): 1.0, ('PROPN', 'Inad'): 1.0, ('PROPN', \"Gu'ud\"): 3.0, ('PROPN', 'Bu'): 2.0, ('PROPN', 'Nimr'): 2.0, ('ADJ', 'powerful'): 5.0, ('PROPN', 'Dulaym'): 1.0, ('VERB', 'favoured'): 1.0, ('NOUN', 'terms'): 4.0, ('NOUN', 'family'): 11.0, ('NOUN', 'seat'): 1.0, ('NOUN', 'governor'): 2.0, ('VERB', 'rose'): 1.0, ('NOUN', 'mid-nineties'): 1.0, ('NOUN', 'execution'): 1.0, ('PROPN', 'Thamir'): 1.0, ('PROPN', 'Madhlum'): 1.0, ('PROPN', 'Dulaymi'): 1.0, ('VERB', 'belonging'): 2.0, ('NOUN', 'revolt'): 1.0, ('NOUN', 'step'): 1.0, ('NOUN', 'arrest'): 10.0, ('PROPN', 'Hassan'): 1.0, ('PROPN', 'Lihabi'): 1.0, ('PROPN', 'Lihaib'): 1.0, ('VERB', 'scattered'): 1.0, ('PROPN', 'Salah'): 1.0, ('PROPN', 'Din'): 1.0, ('PROPN', 'Lihaibi'): 1.0, ('NOUN', 'incident'): 2.0, ('NOUN', 'path'): 2.0, ('ADV', 'forward'): 4.0, ('ADV', 'vice'): 1.0, ('ADV', 'versa'): 1.0, ('DET', 'Both'): 1.0, ('NUM', '14'): 2.0, ('NOUN', 'centuries'): 1.0, ('ADJ', 'possible'): 11.0, ('NOUN', 'partition'): 1.0, ('NOUN', 'possibility'): 1.0, ('NOUN', 'borders'): 2.0, ('PROPN', 'Luis'): 1.0, ('PROPN', 'Posada'): 15.0, ('PROPN', 'Carriles'): 1.0, ('PROPN', 'Miami'): 4.0, ('PROPN', 'Herald'): 1.0, ('ADJ', 'relaxed'): 1.0, ('NOUN', 'terrorist'): 6.0, ('NOUN', 'luxury'): 1.0, ('NOUN', 'condo'): 1.0, ('NOUN', 'yesterday'): 3.0, ('ADV', 'At'): 4.0, ('VERB', 'hid'): 1.0, ('VERB', 'thought'): 9.0, ('ADP', 'Because'): 1.0, ('VERB', 'taken'): 10.0, ('NOUN', 'custody'): 2.0, ('NOUN', 'Immigration'): 1.0, ('NOUN', 'Authorities'): 1.0, ('NOUN', 'whereabouts'): 2.0, ('NOUN', 'shrug'): 1.0, ('NOUN', 'knowledge'): 3.0, ('VERB', 'entered'): 1.0, ('ADJ', 'comfortable'): 3.0, ('NOUN', 'thugs'): 1.0, ('VERB', 'overplayed'): 1.0, ('VERB', 'embarrassed'): 1.0, ('NOUN', 'patron'): 1.0, ('AUX', \"'d\"): 7.0, ('ADJ', 'faithful'): 1.0, ('NOUN', 'servant'): 1.0, ('NOUN', 'player'): 2.0, ('NOUN', 'cue'): 1.0, ('NOUN', 'bomber'): 2.0, ('ADJ', 'pesky'): 1.0, ('ADJ', 'dark'): 2.0, ('ADJ', 'skinned'): 1.0, ('VERB', 'extradited'): 3.0, ('PROPN', 'Meiring'): 6.0, ('ADJ', 'inquisitive'): 1.0, ('NOUN', 'journalist'): 2.0, ('NOUN', 'doorstep'): 1.0, ('ADJ', 'single'): 2.0, ('NOUN', 'trail'): 1.0, ('PROPN', 'Philippines'): 5.0, ('VERB', 'harms'): 1.0, ('AUX', \"'ll\"): 10.0, ('VERB', 'am'): 11.0, ('ADV', 'maybe'): 4.0, ('VERB', 'saved'): 2.0, ('ADV', 'either'): 1.0, ('NOUN', 'crimes'): 6.0, ('VERB', 'wave'): 1.0, ('NOUN', 'flag'): 1.0, ('NOUN', 'War'): 2.0, ('NOUN', 'Terror'): 2.0, ('ADJ', 'vulnerable'): 4.0, ('NOUN', 'node'): 1.0, ('NOUN', 'agencies'): 1.0, ('NOUN', 'organizations'): 5.0, ('VERB', 'conduct'): 2.0, ('VERB', 'Ask'): 2.0, ('PROPN', 'Nichols'): 1.0, ('VERB', 'exacerbated'): 1.0, ('ADJ', 'Philippine'): 3.0, ('VERB', 'exploit'): 2.0, ('ADJ', 'ancient'): 2.0, ('PROPN', 'Venezuela'): 4.0, ('VERB', 'answered'): 1.0, ('NOUN', 'Yes'): 1.0, ('VERB', 'determined'): 3.0, ('ADJ', 'Strong'): 1.0, ('NOUN', 'Man'): 1.0, ('PROPN', 'Hugo'): 1.0, ('PROPN', 'Chavez'): 2.0, ('VERB', 'guaranteed'): 1.0, ('ADJ', 'post-Chavez'): 1.0, ('ADJ', 'like'): 2.0, ('NOUN', 'puppet'): 1.0, ('PROPN', 'Caracas'): 1.0, ('SCONJ', 'Whether'): 2.0, ('NOUN', 'justice'): 3.0, ('ADJ', 'Venezuelan'): 1.0, ('ADJ', 'vain'): 2.0, ('ADV', 'instead'): 5.0, ('ADJ', 'embarrassing'): 1.0, ('NOUN', 'killer'): 1.0, ('VERB', 'strut'): 1.0, ('VERB', 'gloating'): 1.0, ('NOUN', 'freedom'): 2.0, ('VERB', 'threatens'): 1.0, ('NOUN', 'consensus'): 2.0, ('NOUN', 'fiction'): 2.0, ('ADJ', 'unspoken'): 1.0, ('NOUN', 'connection'): 13.0, ('VERB', 'smuggling'): 2.0, ('VERB', 'dating'): 1.0, ('NOUN', 'book'): 5.0, ('PROPN', 'Disposable'): 1.0, ('PROPN', 'Patriot'): 1.0, ('PROPN', 'Jack'): 1.0, ('PROPN', 'Terrell'): 4.0, ('NOUN', 'asset'): 3.0, ('VERB', 'recruited'): 2.0, ('PROPN', 'Contra'): 3.0, ('VERB', 'identifies'): 1.0, ('VERB', 'recruit'): 2.0, ('NOUN', 'scheme'): 1.0, ('PROPN', 'Oliver'): 1.0, ('VERB', 'recall'): 1.0, ('NOUN', 'object'): 1.0, ('VERB', 'smuggle'): 1.0, ('NOUN', 'explosives'): 4.0, ('X', 'http://www.amazon.ca/exec/obidos/ASIN/0915765381/701-3377456-8181939'): 1.0, ('AUX', \"'ve\"): 2.0, ('VERB', 'read'): 11.0, ('NOUN', 'memoir'): 1.0, ('NOUN', 'ring'): 2.0, ('ADJ', 'wider'): 2.0, ('NOUN', 'context'): 2.0, ('VERB', 'confirmed'): 4.0, ('NOUN', 'allegations'): 4.0, ('NOUN', 'drug'): 7.0, ('NOUN', 'connections'): 1.0, ('NOUN', 'target'): 2.0, ('NOUN', 'discrediting'): 1.0, ('PROPN', 'TWIG'): 1.0, ('NOUN', 'counterintelligence'): 1.0, ('NOUN', 'interference'): 4.0, ('ADJ', 'covert'): 2.0, ('NOUN', 'ops'): 2.0, ('NOUN', 'types'): 1.0, ('PROPN', 'Ollie'): 1.0, ('PROPN', 'Robert'): 1.0, ('PROPN', 'Owen'): 1.0, ('PROPN', 'Vince'): 9.0, ('PROPN', 'Cannistaro'): 1.0, ('PROPN', 'Buck'): 1.0, ('PROPN', 'Revell'): 1.0, ('PROPN', 'Peter'): 3.0, ('PROPN', 'Dale'): 1.0, ('PROPN', 'Scott'): 1.0, ('NOUN', 'account'): 4.0, ('NOUN', 'whistleblowing'): 1.0, ('VERB', 'stifled'): 1.0, ('PROPN', 'Cocaine'): 1.0, ('PROPN', 'Politics'): 1.0, ('ADJ', 'ole'): 2.0, ('ADP', 'Due'): 1.0, ('NOUN', 'Judge'): 1.0, ('NUM', '1983'): 2.0, ('VERB', 'ruling'): 1.0, ('VERB', 'testify'): 1.0, ('NOUN', 'pseudonym'): 1.0, ('VERB', 'opening'): 2.0, ('NOUN', 'cross-examination'): 1.0, ('PROPN', 'Wilson'): 7.0, ('NOUN', 'prosecution'): 3.0, ('VERB', 'convince'): 2.0, ('NOUN', 'jury'): 4.0, ('ADJ', 'relevant'): 1.0, ('NOUN', 'files'): 4.0, ('VERB', 'claiming'): 3.0, ('VERB', 'sold'): 3.0, ('NOUN', 'C'): 1.0, ('PROPN', 'Quaddaffi'): 1.0, ('VERB', 'charging'): 1.0, ('NOUN', 'rogue'): 1.0, ('NOUN', 'contacts'): 2.0, ('PROPN', 'Office'): 2.0, ('PROPN', 'Naval'): 2.0, ('PROPN', 'Intelligence'): 4.0, ('ADJ', 'non-social'): 1.0, ('NOUN', 'retirement'): 1.0, ('NOUN', 'turmoil'): 1.0, ('PROPN', 'Charles'): 1.0, ('PROPN', 'A.'): 1.0, ('PROPN', 'Briggs'): 3.0, ('NOUN', 'rescue'): 1.0, ('ADV', 'Third'): 2.0, ('VERB', 'ranking'): 1.0, ('NOUN', 'declaration'): 1.0, ('NOUN', '3rd'): 1.0, ('NOUN', '8th'): 1.0, ('VERB', 'authorized'): 1.0, ('ADJ', 'material'): 1.0, ('VERB', 'pertains'): 1.0, ('PROPN', 'Edwin'): 1.0, ('PROPN', 'P.'): 1.0, ('ADJ', 'various'): 6.0, ('VERB', 'concerning'): 1.0, ('NOUN', '28th'): 1.0, ('PROPN', 'Declaration'): 1.0, ('VERB', 'states'): 1.0, ('ADV', 'directly'): 1.0, ('ADV', 'indirectly'): 1.0, ('VERB', 'retiring'): 1.0, ('X', 'http://www.disinfo.com/archive/pages/dossier/id334/pg1/'): 1.0, ('PROPN', 'Federal'): 1.0, ('PROPN', 'District'): 1.0, ('PROPN', 'Judge'): 2.0, ('PROPN', 'Lynn'): 1.0, ('PROPN', 'Hughes'): 3.0, ('PROPN', 'Huston'): 1.0, ('VERB', 'threw'): 1.0, ('NOUN', 'conviction'): 4.0, ('VERB', 'wrote'): 3.0, ('PUNCT', '`'): 2.0, ('ADV', 'knowingly'): 1.0, ('VERB', 'concluding'): 2.0, ('NOUN', 'honesty'): 1.0, ('VERB', 'comes'): 4.0, ('X', 'http://www.bigeye.com/111003.htm'): 1.0, ('X', 'http://www.thekcrachannel.com/news/4503872/detail.html'): 1.0, ('VERB', 'looks'): 1.0, ('VERB', 'rearing'): 1.0, ('PROPN', 'Louisianna'): 1.0, ('PUNCT', '..'): 2.0, ('ADJ', 'Former'): 3.0, ('NOUN', 'Pastor'): 1.0, ('NOUN', 'Deputy'): 1.0, ('VERB', 'Implicated'): 1.0, ('NOUN', 'Church'): 2.0, ('NOUN', 'Child'): 2.0, ('NOUN', 'Sex'): 1.0, ('NOUN', 'Abuse'): 1.0, ('PROPN', 'PONCHATOULA'): 1.0, ('PROPN', 'La.'): 2.0, ('NOUN', 'Sheriff'): 1.0, ('NOUN', 'deputies'): 1.0, ('PROPN', 'Louisiana'): 1.0, ('ADJ', 'ongoing'): 4.0, ('NOUN', 'investigation'): 7.0, ('ADJ', 'sexual'): 5.0, ('NOUN', 'abuse'): 4.0, ('NOUN', 'animals'): 2.0, ('PROPN', 'Ponchatoula'): 1.0, ('NOUN', 'church'): 6.0, ('PROPN', 'Austin'): 1.0, ('PROPN', 'Aaron'): 1.0, ('PROPN', 'Bernard'): 3.0, ('PROPN', 'III'): 2.0, ('NUM', '36'): 1.0, ('VERB', 'arrested'): 6.0, ('VERB', 'aggravated'): 3.0, ('NOUN', 'rape'): 3.0, ('NOUN', 'age'): 4.0, ('NUM', '13'): 1.0, ('NOUN', 'Police'): 2.0, ('VERB', 'confessed'): 1.0, ('NOUN', 'detectives'): 1.0, ('NOUN', 'sex'): 2.0, ('NOUN', 'girl'): 2.0, ('VERB', 'admitted'): 4.0, ('VERB', 'knowing'): 1.0, ('NOUN', 'acts'): 6.0, ('NOUN', 'dog'): 3.0, ('VERB', 'occurred'): 3.0, ('PROPN', 'Hosanna'): 2.0, ('PROPN', 'Church'): 2.0, ('PROPN', 'Tangipahoa'): 1.0, ('PROPN', 'Parish'): 2.0, ('NOUN', 'sheriff'): 1.0, ('PROPN', 'Christopher'): 1.0, ('PROPN', 'Blair'): 5.0, ('PROPN', 'Labat'): 1.0, ('NUM', '24'): 2.0, ('VERB', 'booked'): 2.0, ('PROPN', 'Tuesday'): 6.0, ('NOUN', 'count'): 3.0, ('NOUN', 'nature'): 4.0, ('PROPN', 'Monday'): 3.0, ('PROPN', 'Louis'): 2.0, ('PROPN', 'Lamonica'): 1.0, ('NOUN', 'pastor'): 1.0, ('NOUN', 'counts'): 1.0, ('VERB', 'walked'): 1.0, ('PROPN', 'Livingston'): 1.0, ('PROPN', 'Sheriff'): 1.0, ('VERB', 'implicate'): 1.0, ('VERB', 'unravelling'): 1.0, ('NOUN', 'northwest'): 1.0, ('NOUN', 'murder'): 14.0, ('ADJ', 'missing'): 4.0, ('NOUN', 'purpose'): 3.0, ('PROPN', 'Rummy'): 1.0, ('VERB', 'send'): 7.0, ('INTJ', 'yeah'): 1.0, ('NOUN', 'evening'): 3.0, ('VERB', 'featured'): 1.0, ('NOUN', 'television'): 3.0, ('NOUN', 'show'): 3.0, ('PROPN', 'Germany'): 3.0, ('PROPN', 'ARD'): 1.0, ('PROPN', 'Tagesthemen'): 1.0, ('ADV', 'newly'): 2.0, ('NOUN', 'papers'): 4.0, ('ADJ', 'complicit'): 1.0, ('NOUN', 'airliner'): 1.0, ('PROPN', 'gandalf'): 1.0, ('VERB', 'relate'): 3.0, ('VERB', 'thinking'): 6.0, ('VERB', 'enjoy'): 3.0, ('VERB', 'reading'): 3.0, ('NOUN', 'blog'): 7.0, ('AUX', \"'m\"): 8.0, ('ADV', 'definitely'): 5.0, ('NOUN', 'immigration'): 1.0, ('PROPN', 'Canada'): 4.0, ('ADJ', 'A3l'): 1.0, ('VERB', 'ai'): 2.0, ('PART', 'nt'): 2.0, ('PRON', 'yuor'): 1.0, ('INTJ', 'Please'): 15.0, ('VERB', 'check'): 5.0, ('INTJ', 'hey'): 2.0, ('ADJ', 'great'): 5.0, ('NOUN', 'site'): 3.0, ('PROPN', 'los'): 2.0, ('PROPN', 'angeles'): 2.0, ('ADJ', 'online'): 3.0, ('NOUN', 'dating'): 6.0, ('ADJ', 'excellent'): 5.0, ('VERB', 'Keep'): 1.0, ('ADV', 'pretty'): 4.0, ('VERB', 'covers'): 1.0, ('NOUN', 'stuff'): 2.0, ('ADJ', 'sure'): 10.0, ('INTJ', 'Hey'): 1.0, ('VERB', 'love'): 2.0, ('NOUN', 'advice'): 3.0, ('SYM', '/'): 7.0, ('VERB', 'deals'): 1.0, ('VERB', 'searching'): 1.0, ('NOUN', 'info'): 3.0, ('VERB', 'agree'): 6.0, ('ADV', 'totally'): 6.0, ('PROPN', 'Paul'): 4.0, ('ADV', 'How'): 10.0, ('AUX', 'Would'): 3.0, ('VERB', 'Like'): 1.0, ('PART', 'To'): 4.0, ('VERB', 'Know'): 2.0, ('PRON', 'YOU'): 1.0, ('VERB', 'Live'): 1.0, ('ADJ', 'Beautiful'): 1.0, ('ADJ', 'NEW'): 1.0, ('NOUN', 'House'): 1.0, ('AUX', 'Is'): 1.0, ('NOUN', 'Custom'): 1.0, ('VERB', 'Designed'): 1.0, ('ADP', 'To'): 6.0, ('PRON', 'YOUR'): 1.0, ('NOUN', 'Specifications'): 1.0, ('PUNCT', '.........'): 1.0, ('DET', 'NO'): 1.0, ('NOUN', 'COST'): 1.0, ('AUX', 'CAN'): 1.0, ('VERB', 'Do'): 2.0, ('PART', 'Not'): 4.0, ('ADJ', 'Hard'): 1.0, ('SCONJ', 'IF'): 1.0, ('ADV', 'HOW'): 1.0, ('VERB', 'Pass'): 1.0, ('ADV', 'By'): 1.0, ('SCONJ', 'Without'): 1.0, ('ADV', 'Least'): 1.0, ('VERB', 'Taking'): 1.0, ('NOUN', 'Look'): 1.0, ('ADJ', 'Only'): 1.0, ('NOUN', 'Catch'): 1.0, ('VERB', 'Is'): 4.0, ('VERB', 'HAVE'): 2.0, ('VERB', 'Have'): 1.0, ('NOUN', 'Income'): 1.0, ('ADP', 'ABOVE'): 1.0, ('SYM', '$'): 20.0, ('NUM', '21,000'): 2.0, ('PUNCT', '......'): 2.0, ('VERB', 'Follow'): 1.0, ('ADJ', 'Easy'): 1.0, ('NOUN', 'Instructions'): 4.0, ('VERB', 'Provided'): 1.0, ('X', 'Get-A-Free-House.com'): 1.0, ('SYM', '+'): 3.0, ('NOUN', 'implications'): 2.0, ('PROPN', 'Andaman'): 21.0, ('PROPN', 'Islands'): 13.0, ('VERB', 'sink'): 3.0, ('NOUN', 'scrutiny'): 2.0, ('NOUN', 'fallibility'): 2.0, ('ADJ', 'evident'): 3.0, ('NOUN', 'inability'): 2.0, ('VERB', 'foresee'): 2.0, ('NOUN', 'advance'): 2.0, ('ADJ', 'pro-India'): 2.0, ('ADJ', 'Bangladeshi'): 3.0, ('PROPN', 'Hasina'): 2.0, ('PROPN', 'Wazed'): 2.0, ('PROPN', 'August'): 6.0, ('NUM', '21'): 3.0, ('NOUN', 'rally'): 2.0, ('PROPN', 'Dhaka'): 2.0, ('ADJ', 'Nepalese'): 2.0, ('PROPN', 'King'): 2.0, ('PROPN', 'Gyanendra'): 2.0, ('VERB', 'dismissing'): 2.0, ('VERB', 'imposing'): 2.0, ('ADJ', 'virtual'): 2.0, ('ADJ', 'absolute'): 4.0, ('NOUN', 'monarchy'): 2.0, ('ADJ', 'glaring'): 2.0, ('NOUN', 'Trouble'): 1.0, ('NOUN', 'islands'): 10.0, ('PROPN', 'Ramtanu'): 2.0, ('PROPN', 'Maitra'): 2.0, ('VERB', 'charged'): 5.0, ('NUM', '34'): 1.0, ('ADJ', 'Arakan'): 2.0, ('NOUN', 'separatists'): 1.0, ('PROPN', 'Myanmar'): 8.0, ('VERB', 'hiding'): 3.0, ('PROPN', 'Landfall'): 1.0, ('NOUN', 'note'): 2.0, ('ADJ', 'alleged'): 4.0, ('PROPN', 'Arakan'): 2.0, ('NOUN', 'wing'): 3.0, ('PROPN', 'Unity'): 1.0, ('ADJ', 'illegal'): 2.0, ('NOUN', 'entry'): 3.0, ('VERB', 'deported'): 1.0, ('NOUN', 'tip'): 1.0, ('NOUN', 'iceberg'): 1.0, ('VERB', 'growing'): 3.0, ('NOUN', 'concern'): 5.0, ('NOUN', 'Reports'): 2.0, ('VERB', 'circulating'): 1.0, ('ADJ', 'thick'): 1.0, ('NOUN', 'depot'): 1.0, ('VERB', 'operate'): 2.0, ('ADV', 'freely'): 1.0, ('PROPN', 'Sea'): 9.0, ('NOUN', 'Information'): 1.0, ('NOUN', 'direction'): 1.0, ('NOUN', 'light'): 3.0, ('NUM', '26'): 5.0, ('NOUN', 'tsunami'): 2.0, ('ADJ', 'heavy'): 8.0, ('NOUN', 'toll'): 1.0, ('PROPN', 'Andamans'): 4.0, ('NOUN', 'Correspondents'): 1.0, ('VERB', 'flocked'): 1.0, ('ADJ', 'miserable'): 2.0, ('VERB', 'surrounding'): 2.0, ('NOUN', 'navy'): 4.0, ('PROPN', 'Far'): 2.0, ('PROPN', 'Eastern'): 1.0, ('PROPN', 'Command'): 1.0, ('VERB', 'established'): 1.0, ('ADJ', 'blue'): 1.0, ('NOUN', 'water'): 1.0, ('VERB', 'Deteriorating'): 1.0, ('NOUN', 'Security'): 2.0, ('VERB', 'deteriorated'): 1.0, ('PROPN', 'Nepal'): 1.0, ('PROPN', 'Bhutan'): 3.0, ('ADV', 'particularly'): 4.0, ('NOUN', 'states'): 4.0, ('PROPN', 'Manipur'): 2.0, ('PROPN', 'Nagaland'): 1.0, ('PROPN', 'Tripura'): 1.0, ('NOUN', 'rock'): 1.0, ('ADJ', 'solid'): 1.0, ('ADJ', 'western'): 3.0, ('NOUN', 'front'): 4.0, ('VERB', 'developed'): 3.0, ('VERB', 'withstanding'): 1.0, ('NOUN', 'adventure'): 1.0, ('NOUN', 'sector'): 2.0, ('ADJ', 'eastern'): 2.0, ('NOUN', 'multitude'): 1.0, ('ADJ', 'secessionist'): 1.0, ('PROPN', 'Southeast'): 2.0, ('NOUN', 'conduit'): 1.0, ('NOUN', 'traffic'): 2.0, ('NUM', '572'): 1.0, ('ADJ', 'large'): 11.0, ('VERB', 'constitute'): 2.0, ('PROPN', 'Nicobar'): 6.0, ('NOUN', 'transit'): 1.0, ('NOUN', 'drugs'): 6.0, ('VERB', 'travel'): 2.0, ('NOUN', 'directions'): 2.0, ('SCONJ', 'Since'): 2.0, ('VERB', 'rule'): 5.0, ('NOUN', 'use'): 6.0, ('PROPN', 'Aceh'): 1.0, ('ADJ', 'east'): 2.0, ('NOUN', 'coast'): 4.0, ('PROPN', 'Africa'): 4.0, ('NOUN', 'network'): 5.0, ('ADJ', 'unknown'): 1.0, ('PROPN', 'Port'): 4.0, ('ADJ', 'unnamed'): 3.0, ('NOUN', 'official'): 6.0, ('ADV', 'permanently'): 1.0, ('VERB', 'settled'): 1.0, ('ADJ', 'fake'): 2.0, ('NOUN', 'ration'): 1.0, ('NOUN', 'cards'): 1.0, ('PROPN', 'Indonesia'): 4.0, ('PROPN', 'Malaysia'): 3.0, ('VERB', 'migrated'): 1.0, ('ADV', 'temporarily'): 1.0, ('VERB', 'plunder'): 1.0, ('NOUN', 'resources'): 3.0, ('PROPN', 'Havelock'): 1.0, ('PROPN', 'Diglipur'): 1.0, ('PROPN', 'Campbell'): 1.0, ('PROPN', 'Bay'): 3.0, ('PROPN', 'Neil'): 3.0, ('PROPN', 'Rangott'): 1.0, ('VERB', 'overrun'): 1.0, ('NOUN', 'estimate'): 1.0, ('VERB', 'suggests'): 1.0, ('NUM', '50,000'): 1.0, ('ADJ', 'unofficial'): 1.0, ('ADJ', 'higher'): 1.0, ('PROPN', 'Bangladeshis'): 1.0, ('NOUN', 'millions'): 5.0, ('NOUN', 'countrymen'): 2.0, ('ADV', 'densely'): 1.0, ('VERB', 'populated'): 5.0, ('NOUN', 'homeland'): 2.0, ('VERB', 'settle'): 2.0, ('ADV', 'elsewhere'): 1.0, ('ADJ', 'technical'): 3.0, ('NOUN', 'skills'): 1.0, ('NOUN', 'demand'): 1.0, ('NOUN', 'smuggling'): 2.0, ('ADJ', 'unlawful'): 1.0, ('NOUN', 'presence'): 5.0, ('NOUN', 'guns'): 11.0, ('NOUN', 'cash'): 4.0, ('NOUN', 'ground'): 5.0, ('PROPN', 'Coast'): 1.0, ('ADV', 'grossly'): 1.0, ('ADJ', 'unequipped'): 1.0, ('VERB', 'deal'): 6.0, ('NOUN', 'surge'): 1.0, ('NOUN', 'migrants'): 1.0, ('ADJ', 'naval'): 2.0, ('NOUN', 'Arms'): 1.0, ('ADJ', 'profitable'): 1.0, ('VERB', 'Considering'): 1.0, ('ADJ', 'huge'): 3.0, ('ADJ', 'strategic'): 2.0, ('NOUN', 'importance'): 1.0, ('ADJ', 'amazing'): 3.0, ('ADJ', 'lax'): 2.0, ('VERB', 'sit'): 1.0, ('ADP', 'aside'): 2.0, ('NOUN', 'sea'): 2.0, ('NOUN', 'lanes'): 1.0, ('PROPN', 'Strait'): 1.0, ('PROPN', 'Malacca'): 1.0, ('NOUN', 'tankers'): 1.0, ('NOUN', 'merchant'): 3.0, ('NOUN', 'ships'): 1.0, ('VERB', 'pass'): 3.0, ('ADV', 'daily'): 2.0, ('NOUN', 'oil'): 2.0, ('NOUN', 'Intelligence'): 2.0, ('NOUN', 'lapses'): 1.0, ('ADJ', 'dependent'): 1.0, ('ADJ', 'smaller'): 5.0, ('NOUN', 'nations'): 2.0, ('VERB', 'maintain'): 2.0, ('SCONJ', 'Unless'): 1.0, ('VERB', 'providing'): 1.0, ('NOUN', 'nationals'): 1.0, ('NOUN', 'secessionists'): 1.0, ('NOUN', 'runners'): 1.0, ('NOUN', 'traffickers'): 1.0, ('VERB', 'generate'): 1.0, ('NOUN', 'confidence'): 1.0, ('NOUN', 'capitals'): 1.0, ('VERB', 'tend'): 2.0, ('ADJ', 'organized'): 5.0, ('ADJ', 'anti-India'): 1.0, ('NOUN', 'outfits'): 1.0, ('X', 'Inter-'): 1.0, ('PROPN', 'Services'): 2.0, ('PROPN', 'ISI'): 6.0, ('ADV', 'outside'): 1.0, ('VERB', 'linked'): 3.0, ('PROPN', 'Maoists'): 1.0, ('NOUN', 'networks'): 2.0, ('VERB', 'weaken'): 1.0, ('NOUN', 'flank'): 1.0, ('ADJ', 'lackluster'): 1.0, ('VERB', 'dealing'): 2.0, ('ADJ', 'startling'): 1.0, ('ADJ', 'black'): 6.0, ('NOUN', 'eye'): 3.0, ('NOUN', 'mid-1980s'): 1.0, ('ADJ', 'long'): 6.0, ('ADV', 'wide'): 1.0, ('VERB', 'ranging'): 1.0, ('NOUN', 'operation'): 3.0, ('VERB', 'inhabited'): 1.0, ('VERB', 'neighboring'): 1.0, ('NOUN', 'caches'): 1.0, ('VERB', 'search'): 3.0, ('VERB', 'destroy'): 3.0, ('VERB', 'carried'): 2.0, ('VERB', 'repeated'): 1.0, ('NOUN', 'requests'): 2.0, ('NOUN', 'addition'): 2.0, ('NOUN', 'activity'): 3.0, ('ADJ', 'known'): 3.0, ('ADJ', 'maritime'): 1.0, ('ADJ', 'defunct'): 1.0, ('PROPN', 'Asiaweek'): 1.0, ('NOUN', 'shipment'): 3.0, ('NOUN', 'ammunition'): 4.0, ('VERB', 'purchased'): 4.0, ('PROPN', 'Cambodia'): 2.0, ('ADJ', 'worth'): 1.0, ('NOUN', 'dollars'): 4.0, ('NOUN', 'port'): 3.0, ('PROPN', 'Phuket'): 3.0, ('ADP', 'aboard'): 1.0, ('NOUN', 'freighter'): 2.0, ('PROPN', 'Comex'): 1.0, ('PROPN', 'Joux'): 1.0, ('ADJ', 'standard'): 1.0, ('NOUN', 'procedure'): 1.0, ('NOUN', 'vessel'): 1.0, ('VERB', 'changed'): 7.0, ('PROPN', 'Horizon'): 1.0, ('NOUN', 'journey'): 2.0, ('PROPN', 'Bengal'): 3.0, ('VERB', 'tracked'): 1.0, ('PROPN', 'Orissa'): 1.0, ('NOUN', 'spy'): 1.0, ('NOUN', 'planes'): 3.0, ('PROPN', 'Aviation'): 1.0, ('PROPN', 'Research'): 1.0, ('PROPN', 'Center'): 6.0, ('VERB', 'intercepted'): 2.0, ('NOUN', 'vessels'): 1.0, ('NOUN', 'running'): 2.0, ('NUM', '1997'): 5.0, ('ADJ', 'Thai'): 3.0, ('NOUN', 'interception'): 1.0, ('NUM', '16'): 4.0, ('NOUN', 'meter'): 1.0, ('NOUN', 'boat'): 2.0, ('NOUN', 'chase'): 1.0, ('PROPN', 'Ranong'): 1.0, ('NOUN', 'confiscation'): 1.0, ('NOUN', 'tons'): 1.0, ('ADP', 'Among'): 1.0, ('NOUN', 'rocket'): 1.0, ('VERB', 'propelled'): 1.0, ('NOUN', 'grenade'): 2.0, ('NOUN', 'launchers'): 2.0, ('NOUN', 'assault'): 3.0, ('NOUN', 'rifles'): 2.0, ('NOUN', 'M'): 1.0, ('NUM', '79'): 1.0, ('NUM', '10,000'): 3.0, ('NOUN', 'persons'): 1.0, ('PROPN', 'Front'): 5.0, ('NOUN', 'crew'): 1.0, ('VERB', 'heading'): 1.0, ('PROPN', 'Cox'): 2.0, ('PROPN', 'Bazar'): 2.0, ('ADP', 'Until'): 3.0, ('NUM', '1995'): 3.0, ('VERB', 'maintained'): 2.0, ('PROPN', 'Twante'): 1.0, ('NOUN', 'coat'): 1.0, ('ADV', 'west'): 1.0, ('ADV', 'Subsequently'): 1.0, ('VERB', 'became'): 6.0, ('NOUN', 'backup'): 2.0, ('VERB', 'born'): 2.0, ('NOUN', 'passport'): 1.0, ('NOUN', 'suspect'): 2.0, ('ADV', 'allegedly'): 5.0, ('VERB', 'constructing'): 1.0, ('NOUN', 'submarine'): 1.0, ('NOUN', 'shipyard'): 1.0, ('PROPN', 'Sirae'): 1.0, ('ADJ', 'Steady'): 1.0, ('NOUN', 'buildup'): 1.0, ('NOUN', 'Islands'): 1.0, ('ADJ', 'steady'): 1.0, ('NOUN', 'up'): 3.0, ('NOUN', 'ports'): 1.0, ('NOUN', 'conduits'): 1.0, ('NOUN', 'host'): 1.0, ('ADJ', 'militant'): 12.0, ('ADV', 'steadily'): 1.0, ('VERB', 'moved'): 2.0, ('NOUN', 'lawlessness'): 1.0, ('NOUN', 'extremist'): 4.0, ('NOUN', 'cover'): 2.0, ('NOUN', 'trafficking'): 2.0, ('NOUN', 'gun'): 2.0, ('ADJ', 'anti-Indian'): 1.0, ('ADV', 'widely'): 2.0, ('VERB', 'acknowledged'): 2.0, ('VERB', 'nurtured'): 1.0, ('PROPN', 'Harkat'): 1.0, ('PROPN', 'ul'): 1.0, ('PROPN', 'Jehad'): 1.0, ('PROPN', 'Islami'): 2.0, ('PROPN', 'Chittagong'): 3.0, ('NOUN', 'secessionist'): 1.0, ('NOUN', 'northeast'): 1.0, ('NOUN', 'touch'): 1.0, ('NOUN', 'result'): 6.0, ('ADV', 'north'): 1.0, ('NOUN', 'coastal'): 1.0, ('NOUN', 'nest'): 2.0, ('NOUN', 'pattern'): 3.0, ('NOUN', 'arrests'): 1.0, ('NOUN', 'seizures'): 1.0, ('VERB', 'brought'): 1.0, ('PROPN', 'Laos'): 1.0, ('VERB', 'transported'): 2.0, ('ADV', 'northward'): 1.0, ('PROPN', 'Kalikhola'): 1.0, ('VERB', 'passes'): 1.0, ('ADJ', 'northern'): 2.0, ('PROPN', 'Assam'): 1.0, ('PROPN', 'Meghalaya'): 1.0, ('NOUN', 'Note'): 1.0, ('VERB', 'comprise'): 1.0, ('VERB', 'lying'): 3.0, ('NUM', '1,000'): 1.0, ('NOUN', 'kilometers'): 2.0, ('ADV', 'east'): 1.0, ('VERB', 'Stretching'): 1.0, ('NUM', '750'): 1.0, ('VERB', 'reach'): 4.0, ('PROPN', 'Sumatra'): 1.0, ('PROPN', 'Ten'): 1.0, ('PROPN', 'Degree'): 1.0, ('PROPN', 'Channel'): 1.0, ('VERB', 'divides'): 1.0, ('NOUN', 'archipelago'): 1.0, ('NOUN', 'Travel'): 1.0, ('ADJ', 'forbidden'): 1.0, ('PROPN', 'non-Indians'): 1.0, ('PROPN', 'Island'): 1.0, ('VERB', 'writes'): 2.0, ('NOUN', 'journals'): 1.0, ('ADJ', 'regular'): 3.0, ('NOUN', 'contributor'): 1.0, ('PROPN', 'EIR'): 1.0, ('PROPN', 'Defense'): 2.0, ('PROPN', 'Aakrosh'): 1.0, ('VERB', 'tied'): 2.0, ('ADJ', 'quarterly'): 1.0, ('NOUN', 'journal'): 1.0, ('PROPN', 'Karzai'): 20.0, ('NOUN', 'rivalry'): 2.0, ('VERB', 'accuses'): 2.0, ('NOUN', 'consulates'): 2.0, ('PROPN', 'Kandahar'): 6.0, ('PROPN', 'Jalalabad'): 4.0, ('VERB', 'train'): 2.0, ('ADJ', 'Balochi'): 2.0, ('PROPN', 'Balochistan'): 2.0, ('VERB', 'claim'): 7.0, ('NUM', '42'): 2.0, ('PROPN', 'RAW'): 2.0, ('SCONJ', 'unless'): 2.0, ('VERB', 'undermining'): 3.0, ('PUNCT', \"''\"): 32.0, ('CONJ', 'Both'): 2.0, ('VERB', 'deny'): 2.0, ('NOUN', 'claim'): 3.0, ('AUX', 'Should'): 3.0, ('ADJ', 'Afghan'): 13.0, ('NOUN', 'soil'): 4.0, ('NOUN', 'neighbours'): 7.0, ('PROPN', 'Amrullah'): 2.0, ('PROPN', 'Saleh'): 2.0, ('PROPN', 'Directorate'): 2.0, ('NOUN', 'equations'): 1.0, ('PROPN', 'KABUL'): 1.0, ('ADJ', 'landlocked'): 1.0, ('VERB', 'suffered'): 2.0, ('ADJ', 'constant'): 1.0, ('ADJ', 'Central'): 1.0, ('ADJ', 'Asian'): 1.0, ('NOUN', 'Republics'): 1.0, ('NOUN', 'powers'): 1.0, ('VERB', 'interfering'): 1.0, ('NOUN', 'signs'): 2.0, ('NOUN', 'reassurance'): 1.0, ('ADJ', 'stable'): 1.0, ('ADJ', 'peaceful'): 2.0, ('PRON', 'Nobody'): 2.0, ('NOUN', 'looser'): 1.0, ('PROPN', 'English'): 3.0, ('NOUN', 'daily'): 1.0, ('PROPN', 'Nation'): 1.0, ('PUNCT', 'All'): 1.0, ('ADV', 'publicly'): 4.0, ('VERB', 'supported'): 1.0, ('ADJ', 'electoral'): 1.0, ('NOUN', 'undercurrents'): 1.0, ('ADJ', 'favourite'): 1.0, ('NOUN', 'contenders'): 1.0, ('ADV', 'repeatedly'): 3.0, ('ADJ', 'Western'): 15.0, ('VERB', 'harbouring'): 1.0, ('NOUN', 'extremists'): 4.0, ('VERB', 'pledged'): 1.0, ('VERB', 'disrupt'): 3.0, ('VERB', 'avoided'): 2.0, ('VERB', 'criticising'): 1.0, ('VERB', 'catch'): 2.0, ('ADP', 'inside'): 3.0, ('NUM', '22'): 3.0, ('NOUN', 'meeting'): 18.0, ('NOUN', 'sidelines'): 1.0, ('PROPN', 'UN'): 5.0, ('ADV', 'desperately'): 1.0, ('ADJ', 'anxious'): 1.0, ('VERB', 'secure'): 1.0, ('NOUN', 'non-interference'): 1.0, ('NOUN', 'diplomats'): 4.0, ('ADV', 'intimately'): 1.0, ('SCONJ', 'on'): 9.0, ('VERB', 'reigning'): 1.0, ('ADV', 'peacefully'): 1.0, ('ADV', 'Where'): 1.0, ('PROPN', 'Mullah'): 3.0, ('PROPN', 'Omar'): 2.0, ('PROPN', 'Usmani'): 2.0, ('PROPN', 'Gulbuddin'): 1.0, ('PROPN', 'Hikmetar'): 1.0, ('ADJ', 'flustered'): 1.0, ('PROPN', 'Hikmetyar'): 1.0, ('VERB', 'heads'): 1.0, ('PROPN', 'Hizb'): 1.0, ('PROPN', 'e'): 1.0, ('VERB', 'focused'): 1.0, ('NOUN', 'diplomat'): 2.0, ('VERB', 'briefed'): 1.0, ('VERB', 'realise'): 1.0, ('ADJ', 'bigger'): 2.0, ('ADJ', 'pleased'): 1.0, ('NOUN', 'results'): 2.0, ('VERB', 'cooperate'): 1.0, ('VERB', 'curbing'): 1.0, ('ADJ', 'angry'): 1.0, ('ADV', 'categorically'): 2.0, ('NOUN', 'snub'): 1.0, ('NOUN', 'options'): 4.0, ('VERB', 'insist'): 1.0, ('NOUN', 'decision'): 3.0, ('ADJ', 'unconnected'): 1.0, ('ADJ', 'tripartite'): 2.0, ('PROPN', 'NATO'): 2.0, ('PROPN', 'Kabul'): 6.0, ('ADJ', 'tough'): 1.0, ('NOUN', 'message'): 6.0, ('ADJ', 'instrumental'): 1.0, ('VERB', 'persuading'): 1.0, ('VERB', 'restrain'): 1.0, ('VERB', 'disrupting'): 1.0, ('NOUN', 'crackdown'): 1.0, ('ADV', 'before'): 5.0, ('PROPN', 'Lt.'): 2.0, ('PROPN', 'Barno'): 3.0, ('ADJ', 'tactical'): 1.0, ('VERB', 'goes'): 3.0, ('VERB', 'adds'): 1.0, ('ADV', 'Also'): 6.0, ('NOUN', 'crossing'): 1.0, ('NOUN', 'point'): 13.0, ('PROPN', 'Chaman'): 2.0, ('ADJ', 'Pashtun'): 2.0, ('VERB', 'leading'): 2.0, ('NOUN', 'units'): 2.0, ('ADJ', 'sympathetic'): 1.0, ('NOUN', 'mullahs'): 1.0, ('PROPN', 'JUI'): 1.0, ('ADJ', 'oblivious'): 1.0, ('VERB', 'facing'): 2.0, ('PROPN', 'Waziristan'): 1.0, ('NOUN', 'anti-Americanism'): 1.0, ('ADJ', 'anti-army'): 1.0, ('ADJ', 'actionable'): 1.0, ('VERB', 'change'): 2.0, ('VERB', 'presenting'): 1.0, ('VERB', 'suspected'): 3.0, ('ADV', 'Moreover'): 1.0, ('VERB', 'try'): 5.0, ('ADV', 'quietly'): 3.0, ('ADJ', 'isolated'): 1.0, ('VERB', 'facilitated'): 1.0, ('VERB', 'happen'): 7.0, ('NOUN', 'pull'): 1.0, ('NOUN', 'push'): 2.0, ('VERB', 'win'): 6.0, ('NOUN', 'demotion'): 1.0, ('PROPN', 'Fahim'): 1.0, ('NOUN', 'warlord'): 2.0, ('PROPN', 'Ismail'): 2.0, ('PROPN', 'Khan'): 6.0, ('VERB', 'fear'): 2.0, ('NOUN', 'reprisals'): 1.0, ('PRON', 'Their'): 1.0, ('NOUN', 'removal'): 1.0, ('VERB', 'provide'): 7.0, ('VERB', 'increased'): 1.0, ('NOUN', 'motivation'): 2.0, ('VERB', 'welcomed'): 1.0, ('NOUN', 'appointment'): 1.0, ('PROPN', 'Ashfaq'): 1.0, ('PROPN', 'Kiyani'): 1.0, ('PROPN', 'Kayani'): 1.0, ('NOUN', 'delegation'): 1.0, ('NOUN', 'meetings'): 3.0, ('NOUN', 'issues'): 3.0, ('VERB', 'assured'): 1.0, ('ADJ', 'adverse'): 1.0, ('VERB', 'acted'): 1.0, ('ADP', 'upon'): 8.0, ('ADV', 'swiftly'): 1.0, ('ADJ', 'Senior'): 3.0, ('VERB', 'warned'): 2.0, ('NOUN', 'shift'): 1.0, ('VERB', 'reflecting'): 1.0, ('ADV', 'traditionally'): 1.0, ('PROPN', 'Tajik'): 1.0, ('ADJ', 'strenuous'): 1.0, ('NOUN', 'efforts'): 1.0, ('PROPN', 'Younis'): 1.0, ('PROPN', 'Qanooni'): 4.0, ('VERB', 'strike'): 2.0, ('VERB', 'oppose'): 2.0, ('ADJ', 'particular'): 5.0, ('VERB', 'feared'): 2.0, ('VERB', 'loose'): 1.0, ('VERB', 'isolated'): 4.0, ('NOUN', 'mainstream'): 1.0, ('VERB', 'refused'): 3.0, ('ADJ', 'fellow'): 2.0, ('ADJ', 'Panjsheri'): 1.0, ('PROPN', 'Tajiks'): 1.0, ('VERB', 'urged'): 2.0, ('VERB', 'stand'): 5.0, ('VERB', 'proved'): 1.0, ('ADJ', 'critical'): 1.0, ('VERB', 'convincing'): 2.0, ('PROPN', 'Hazara'): 1.0, ('PROPN', 'Mohammed'): 18.0, ('PROPN', 'Mohaqeq'): 1.0, ('VERB', 'posing'): 1.0, ('VERB', 'stabilise'): 1.0, ('VERB', 'reduced'): 1.0, ('NOUN', 'hardliners'): 2.0, ('VERB', 'undermine'): 1.0, ('NOUN', 'strategy'): 2.0, ('VERB', 'deepen'): 1.0, ('NOUN', 'rift'): 1.0, ('NOUN', 'moderates'): 2.0, ('ADJ', 'concerned'): 2.0, ('PROPN', 'Shindand'): 4.0, ('ADJ', 'massive'): 1.0, ('NOUN', 'era'): 1.0, ('NOUN', 'airbase'): 1.0, ('NOUN', 'kilometres'): 1.0, ('VERB', 'enhanced'): 1.0, ('NOUN', 'ousting'): 1.0, ('PROPN', 'Governor'): 1.0, ('PROPN', 'Herat'): 1.0, ('NOUN', 'province'): 1.0, ('NOUN', 'month'): 8.0, ('NOUN', 'ally'): 1.0, ('NOUN', 'objections'): 1.0, ('NOUN', 'ouster'): 1.0, ('NOUN', 'warlords'): 1.0, ('VERB', 'heightened'): 1.0, ('NOUN', 'calls'): 5.0, ('NOUN', 'neo-conservatives'): 1.0, ('ADV', 'aggressively'): 1.0, ('PROPN', 'Iranians'): 1.0, ('NOUN', 'listening'): 1.0, ('NOUN', 'spying'): 1.0, ('NOUN', 'facility'): 1.0, ('NOUN', 'launching'): 2.0, ('NOUN', 'pad'): 1.0, ('ADJ', 'Special'): 1.0, ('NOUN', 'Forces'): 1.0, ('VERB', 'insists'): 1.0, ('VERB', 'poses'): 1.0, ('ADV', 'Nevertheless'): 2.0, ('VERB', 'places'): 1.0, ('ADJ', 'sensitive'): 1.0, ('NOUN', 'relations'): 1.0, ('NOUN', 'benefit'): 1.0, ('X', 'post'): 2.0, ('ADJ', 'election'): 1.0, ('NOUN', 'scenario'): 2.0, ('ADV', 'abundantly'): 2.0, ('CONJ', 'nor'): 2.0, ('PROPN', 'Pashtuns'): 1.0, ('NOUN', 'move'): 2.0, ('NOUN', 'tribes'): 1.0, ('PROPN', 'Quetta'): 2.0, ('VERB', 'resisted'): 2.0, ('VERB', 'voted'): 1.0, ('ADV', 'rapidly'): 1.0, ('VERB', 'changing'): 1.0, ('NOUN', 'demise'): 1.0, ('ADJ', 'fluid'): 1.0, ('VERB', 'reconsider'): 1.0, ('ADJ', 'unlimited'): 1.0, ('NOUN', 'sanctuary'): 1.0, ('PROPN', 'Link'): 1.0, ('NUM', '16/10/2004'): 1.0, ('NOUN', 'debate'): 2.0, ('NOUN', 'multi-millionnaires'): 1.0, ('ADV', 'cleverly'): 1.0, ('VERB', 'manufactured'): 1.0, ('ADJ', 'absurd'): 1.0, ('NOUN', 'charges'): 3.0, ('NOUN', 'vets'): 1.0, ('VERB', 'flip'): 1.0, ('VERB', 'flopped'): 1.0, ('VERB', 'contradicting'): 1.0, ('CONJ', 'Or'): 3.0, ('NOUN', 'eyewitnesses'): 1.0, ('VERB', 'address'): 2.0, ('NOUN', 'substance'): 1.0, ('ADJ', 'Big'): 1.0, ('NOUN', 'Lie'): 1.0, ('VERB', 'risk'): 1.0, ('VERB', 'falling'): 1.0, ('NOUN', 'logic'): 2.0, ('NOUN', 'absurdity'): 1.0, ('VERB', 'appreciated'): 1.0, ('NOUN', 'bravery'): 1.0, ('NOUN', 'life'): 7.0, ('VERB', 'lived'): 3.0, ('NOUN', 'zone'): 2.0, ('NOUN', 'wounds'): 3.0, ('ADJ', 'superficial'): 1.0, ('NOUN', 'shrapnel'): 2.0, ('NOUN', 'forearm'): 1.0, ('ADJ', 'minor'): 3.0, ('NOUN', 'wound'): 1.0, ('NOUN', 'brain'): 2.0, ('VERB', 'demonstrates'): 1.0, ('ADJ', 'mortal'): 1.0, ('VERB', 'absent'): 1.0, ('PRON', 'yourself'): 2.0, ('NOUN', 'medal'): 1.0, ('NOUN', 'youth'): 2.0, ('VERB', 'drinking'): 5.0, ('NOUN', 'fish'): 1.0, ('ADJ', 'wee'): 1.0, ('VERB', 'risked'): 1.0, ('VERB', 'slack'): 2.0, ('ADV', 'efficiently'): 1.0, ('NOUN', 'alcoholism'): 1.0, ('ADV', 'possibly'): 2.0, ('VERB', 'speaks'): 1.0, ('NOUN', 'character'): 1.0, ('ADJ', 'addictive'): 1.0, ('NOUN', 'personality'): 1.0, ('ADJ', 'erratic'): 2.0, ('ADJ', 'alarming'): 1.0, ('ADJ', 'explosive'): 2.0, ('NOUN', 'temper'): 2.0, ('VERB', 'provoked'): 1.0, ('ADJ', 'disastrous'): 1.0, ('NOUN', 'siege'): 1.0, ('NOUN', 'spring'): 1.0, ('NUM', '600'): 1.0, ('NOUN', 'revenge'): 1.0, ('ADJ', 'civilian'): 1.0, ('NOUN', 'mercenaries'): 1.0, ('PROPN', 'African'): 1.0, ('PROPN', 'Newsweek'): 3.0, ('VERB', 'commanded'): 1.0, ('NOUN', 'cabinet'): 1.0, ('ADJ', 'sadistic'): 1.0, ('NOUN', 'streak'): 1.0, ('VERB', 'enjoyed'): 2.0, ('NOUN', 'executions'): 1.0, ('NOUN', 'delight'): 1.0, ('VERB', 'seemed'): 1.0, ('NOUN', 'prospect'): 2.0, ('VERB', 'executing'): 2.0, ('ADJ', 'wrong'): 2.0, ('NOUN', 'doers'): 1.0, ('VERB', 'enjoying'): 1.0, ('NOUN', 'scale'): 1.0, ('NOUN', 'Drug'): 1.0, ('NOUN', 'ability'): 6.0, ('ADJ', 'deep'): 4.0, ('NOUN', 'emotions'): 1.0, ('NOUN', 'empathy'): 1.0, ('VERB', 'pickling'): 1.0, ('ADJ', 'nervous'): 1.0, ('NOUN', 'system'): 7.0, ('ADJ', 'toxic'): 1.0, ('NOUN', 'substances'): 1.0, ('VERB', 'damaged'): 1.0, ('NOUN', 'goods'): 1.0, ('VERB', 'abstain'): 1.0, ('ADJ', 'visual'): 1.0, ('ADJ', 'spatial'): 1.0, ('NOUN', 'abilities'): 1.0, ('NOUN', 'abstraction'): 1.0, ('NOUN', 'solving'): 1.0, ('ADJ', 'short'): 2.0, ('NOUN', 'memory'): 1.0, ('ADJ', 'slowest'): 1.0, ('VERB', 'recover'): 1.0, ('NOUN', 'wagon'): 1.0, ('NOUN', 'pretzel'): 1.0, ('ADJ', 'laudable'): 1.0, ('VERB', 'suffers'): 1.0, ('ADJ', 'severe'): 2.0, ('VERB', 'suffering'): 1.0, ('VERB', 'absenting'): 1.0, ('NOUN', 'senate'): 1.0, ('PROPN', 'Winton'): 2.0, ('PROPN', 'Red'): 8.0, ('PROPN', 'Blount'): 12.0, ('NOUN', 'stint'): 1.0, ('ADJ', 'unclear'): 1.0, ('VERB', 'slacked'): 1.0, ('NOUN', 'nephew'): 3.0, ('PROPN', 'Murph'): 3.0, ('PROPN', 'Archibald'): 12.0, ('VERB', 'appeared'): 3.0, ('PROPN', 'Public'): 1.0, ('PROPN', 'Radio'): 1.0, ('PROPN', 'Things'): 2.0, ('VERB', 'Considered'): 2.0, ('VERB', 'gives'): 2.0, ('ADJ', 'devastating'): 1.0, ('NOUN', 'insight'): 1.0, ('NUM', '8:00'): 1.0, ('NOUN', 'PM'): 4.0, ('PROPN', 'ET'): 1.0, ('PROPN', 'NPR'): 2.0, ('NOUN', 'season'): 1.0, ('VERB', 'fulfilled'): 1.0, ('NOUN', 'obligations'): 1.0, ('NOUN', 'lieutenant'): 3.0, ('NOUN', '1970s'): 1.0, ('VERB', 'scoured'): 1.0, ('VERB', 'remembered'): 1.0, ('VERB', 'seeing'): 1.0, ('PROPN', 'Mr.'): 24.0, ('ADV', 'every'): 1.0, ('NOUN', 'headquarters'): 4.0, ('PROPN', 'Montgomery'): 3.0, ('PROPN', 'Wade'): 1.0, ('PROPN', 'Goodwyn'): 2.0, ('PROPN', 'WADE'): 1.0, ('PROPN', 'GOODWYN'): 7.0, ('VERB', 'reporting'): 2.0, ('PROPN', 'Baba'): 2.0, ('PROPN', 'Groom'): 4.0, ('ADJ', 'smart'): 1.0, ('ADJ', 'funny'): 1.0, ('NOUN', 'woman'): 4.0, ('ADV', 'smack'): 1.0, ('ADV', 'dab'): 1.0, ('NOUN', 'middle'): 1.0, ('ADJ', 'exciting'): 1.0, ('NOUN', 'scheduler'): 1.0, ('NOUN', 'job'): 8.0, ('NOUN', 'hub'): 1.0, ('NOUN', 'wheel'): 1.0, ('PRON', 'her'): 17.0, ('ADJ', 'handsome'): 1.0, ('VERB', 'remembers'): 1.0, ('NUM', '32'): 1.0, ('PROPN', 'Ms.'): 1.0, ('PROPN', 'BABA'): 1.0, ('PROPN', 'GROOM'): 1.0, ('NOUN', 'Campaign'): 1.0, ('NOUN', 'Worker'): 1.0, ('VERB', 'wear'): 2.0, ('NOUN', 'khaki'): 1.0, ('NOUN', 'trousers'): 1.0, ('NOUN', 'jacket'): 1.0, ('NOUN', 'phone'): 4.0, ('NOUN', 'accent'): 2.0, ('VERB', 'melded'): 1.0, ('PRON', 'everybody'): 5.0, ('VERB', 'gotten'): 2.0, ('NOUN', 'construction'): 1.0, ('ADJ', 'Prominent'): 1.0, ('ADJ', 'Southern'): 1.0, ('PROPN', 'Republicans'): 2.0, ('ADJ', 'rare'): 2.0, ('NOUN', 'breed'): 1.0, ('VERB', 'appointed'): 1.0, ('PROPN', 'Richard'): 2.0, ('PROPN', 'Nixon'): 1.0, ('PROPN', 'postmaster'): 1.0, ('PROPN', 'general'): 1.0, ('NOUN', 'tennis'): 1.0, ('NOUN', 'partners'): 1.0, ('NOUN', 'father'): 5.0, ('PROPN', 'Congressman'): 1.0, ('PROPN', 'Lieutenant'): 2.0, ('NOUN', 'urging'): 1.0, ('NOUN', 'county'): 3.0, ('NOUN', 'chairpersons'): 1.0, ('NUM', '67'): 1.0, ('NOUN', 'counties'): 3.0, ('ADV', 'Back'): 1.0, ('ADJ', 'Deep'): 1.0, ('NOUN', 'South'): 1.0, ('ADJ', 'rural'): 1.0, ('NOUN', 'apparatus'): 1.0, ('PROPN', 'Texan'): 1.0, ('NOUN', 'supplies'): 3.0, ('VERB', 'helped'): 2.0, ('ADJ', 'statewide'): 1.0, ('NOUN', 'marriage'): 4.0, ('VERB', 'coming'): 5.0, ('NOUN', 'infantry'): 1.0, ('ADJ', 'dedicated'): 1.0, ('NOUN', 'workers'): 2.0, ('PROPN', 'MURPH'): 1.0, ('PROPN', 'ARCHIBALD'): 6.0, ('NOUN', 'Nephew'): 1.0, ('INTJ', 'Well'): 3.0, ('NOUN', 'morning'): 7.0, ('NOUN', 'mid-evenings'): 1.0, ('ADV', 'Ordinarily'): 1.0, ('NOUN', 'noon'): 1.0, ('ADV', 'ordinarily'): 1.0, ('NUM', '5:30'): 1.0, ('NUM', '6:00'): 1.0, ('NOUN', 'manager'): 1.0, ('NOUN', 'materials'): 1.0, ('NOUN', 'concerns'): 2.0, ('ADJ', 'overall'): 1.0, ('NOUN', 'impression'): 1.0, ('PRON', 'somebody'): 1.0, ('VERB', 'talk'): 10.0, ('VERB', 'figured'): 1.0, ('VERB', 'broached'): 1.0, ('NOUN', 'subject'): 5.0, ('ADJ', 'unpleasant'): 1.0, ('ADV', 'Far'): 1.0, ('PROPN', 'DC'): 1.0, ('VERB', 'dated'): 2.0, ('ADJ', 'beautiful'): 2.0, ('NOUN', 'evenings'): 1.0, ('NOUN', 'house'): 3.0, ('VERB', 'rented'): 1.0, ('NOUN', 'disrepair'): 1.0, ('NOUN', 'damage'): 1.0, ('NOUN', 'walls'): 1.0, ('NOUN', 'chandelier'): 1.0, ('VERB', 'owned'): 2.0, ('VERB', 'grumble'): 1.0, ('ADJ', 'unpaid'): 1.0, ('NOUN', 'repair'): 1.0, ('NOUN', 'bill'): 2.0, ('ADJ', 'friendly'): 1.0, ('NOUN', 'stories'): 7.0, ('NOUN', 'drinking'): 2.0, ('ADV', 'kind'): 2.0, ('NOUN', 'starter'): 1.0, ('NOUN', 'People'): 1.0, ('AUX', \"'re\"): 6.0, ('NOUN', 'kids'): 5.0, ('NOUN', 'football'): 3.0, ('NOUN', 'weather'): 1.0, ('NOUN', 'gambit'): 1.0, ('NOUN', 'frequency'): 1.0, ('VERB', 'discussed'): 2.0, ('ADV', 'off'): 2.0, ('ADJ', 'putting'): 1.0, ('NOUN', 'mid-20s'): 1.0, ('NOUN', 'conversations'): 2.0, ('ADV', 'serious'): 1.0, ('ADJ', 'senatorial'): 1.0, ('ADV', 'frankly'): 1.0, ('ADJ', 'inappropriate'): 1.0, ('ADV', 'sometimes'): 4.0, ('PROPN', 'Yale'): 1.0, ('PROPN', 'Haven'): 1.0, ('ADV', 'whenever'): 4.0, ('AUX', 'got'): 1.0, ('NOUN', 'driving'): 1.0, ('NOUN', 'grandson'): 2.0, ('PROPN', 'Connecticut'): 1.0, ('PROPN', 'senator'): 1.0, ('NOUN', 'class'): 1.0, ('NOUN', 'boy'): 1.0, ('PROPN', 'Democrat'): 1.0, ('NOUN', 'law'): 18.0, ('NOUN', 'enforcement'): 6.0, ('PROPN', 'Prescott'): 1.0, ('VERB', 'laugh'): 1.0, ('ADV', 'dutifully'): 1.0, ('VERB', 'notes'): 1.0, ('ADJ', 'drunk'): 1.0, ('VERB', 'means'): 6.0, ('NOUN', 'guys'): 5.0, ('NOUN', 'girls'): 8.0, ('VERB', 'buddy'): 1.0, ('VERB', 'impress'): 1.0, ('NOUN', 'behavior'): 4.0, ('ADJ', 'untouched'): 1.0, ('PRON', 'Our'): 1.0, ('NOUN', 'Republic'): 1.0, ('NOUN', 'hands'): 3.0, ('ADV', 'When'): 11.0, ('VERB', 'burning'): 1.0, ('NOUN', 'flesh'): 1.0, ('NOUN', 'death'): 21.0, ('VERB', 'abiding'): 1.0, ('NOUN', 'citizen'): 1.0, ('PROPN', 'Bob'): 1.0, ('PROPN', 'Dylan'): 1.0, ('ADJ', 'Afraid'): 1.0, ('NOUN', 'today'): 9.0, ('VERB', 'discuss'): 2.0, ('PRON', 'these'): 3.0, ('NOUN', 'attention'): 2.0, ('PROPN', 'Mirror'): 1.0, ('NOUN', 'headline'): 1.0, ('NUM', '200,000'): 7.0, ('NOUN', 'AK47s'): 2.0, ('VERB', 'Fallen'): 1.0, ('ADP', 'Into'): 1.0, ('NOUN', 'Hands'): 1.0, ('NOUN', 'Terrorists'): 1.0, ('NOUN', 'thread'): 3.0, ('PROPN', 'RI'): 2.0, ('NOUN', 'discussion'): 3.0, ('NOUN', 'board'): 2.0, ('VERB', 'smuggled'): 2.0, ('NUM', '99'): 2.0, ('NOUN', 'tonne'): 1.0, ('NOUN', 'cache'): 1.0, ('VERB', 'flown'): 1.0, ('PROPN', 'Bosnia'): 3.0, ('NOUN', 'planeloads'): 2.0, ('VERB', 'vanished'): 1.0, ('NOUN', 'Orders'): 9.0, ('ADV', 'ahead'): 1.0, ('VERB', 'contracted'): 2.0, ('ADP', 'via'): 2.0, ('ADJ', 'complex'): 2.0, ('NOUN', 'web'): 1.0, ('NOUN', 'traders'): 2.0, ('ADJ', 'Moldovan'): 3.0, ('NOUN', 'airline'): 3.0, ('VERB', 'transport'): 1.0, ('VERB', 'blasted'): 1.0, ('PROPN', 'Liberia'): 1.0, ('PROPN', 'Amnesty'): 2.0, ('ADJ', 'separate'): 4.0, ('NOUN', 'probe'): 1.0, ('VERB', 'meant'): 1.0, ('PROPN', 'Mike'): 7.0, ('PROPN', 'Blakemore'): 1.0, ('ADJ', 'unbelievable'): 2.0, ('VERB', 'account'): 1.0, ('ADJ', 'terrifying'): 1.0, ('NOUN', 'defence'): 1.0, ('NOUN', 'chiefs'): 2.0, ('VERB', 'hired'): 1.0, ('NOUN', '90s'): 1.0, ('ADJ', 'Bosnian'): 2.0, ('NOUN', 'controllers'): 1.0, ('NOUN', 'flights'): 1.0, ('ADV', 'supposedly'): 1.0, ('NUM', '2005'): 1.0, ('NOUN', 'purchases'): 1.0, ('PROPN', 'Nato'): 1.0, ('VERB', 'voiced'): 1.0, ('NOUN', 'fears'): 1.0, ('ADJ', 'Swiss'): 1.0, ('NOUN', 'firms'): 2.0, ('NOUN', 'tracking'): 2.0, ('NOUN', 'mechanism'): 2.0, ('VERB', 'siphoned'): 1.0, ('PROPN', 'UK'): 1.0, ('VERB', 're-routed'): 1.0, ('PROPN', 'Aerocom'): 3.0, ('INTJ', 'yes'): 3.0, ('PROPN', 'Victor'): 2.0, ('PROPN', 'Bout'): 4.0, ('VERB', 'bang'): 1.0, ('NOUN', 'wall'): 2.0, ('NOUN', 'play'): 2.0, ('NOUN', 'incompetence'): 2.0, ('NOUN', 'theory'): 7.0, ('VERB', 'receives'): 5.0, ('NOUN', 'Administration'): 1.0, ('ADJ', 'harshest'): 1.0, ('NOUN', 'critics'): 1.0, ('NOUN', 'misdirection'): 1.0, ('ADJ', 'representative'): 2.0, ('VERB', 'privatized'): 1.0, ('ADJ', 'global'): 3.0, ('NOUN', 'gangland'): 1.0, ('ADP', 'Like'): 2.0, ('NOUN', 'billions'): 2.0, ('VERB', 'vanish'): 1.0, ('PROPN', 'Pentagon'): 2.0, ('VERB', 'contracts'): 1.0, ('NOUN', 'criminal'): 1.0, ('ADJ', 'untouchable'): 1.0, ('NOUN', 'stature'): 1.0, ('DET', 'whatever'): 1.0, ('VERB', 'sinks'): 1.0, ('ADP', 'beneath'): 1.0, ('VERB', 'frothing'): 1.0, ('NOUN', 'triviality'): 1.0, ('ADV', 'Meanwhile'): 1.0, ('PROPN', 'Toledo'): 1.0, ('NOUN', 'priest'): 2.0, ('PROPN', 'Gerald'): 2.0, ('PROPN', 'Robinson'): 6.0, ('NOUN', 'guilty'): 2.0, ('PROPN', 'Rev.'): 1.0, ('ADJ', 'stony'): 1.0, ('ADJ', 'faced'): 1.0, ('NOUN', 'verdict'): 1.0, ('VERB', 'blinked'): 1.0, ('VERB', 'glanced'): 1.0, ('NOUN', 'lawyers'): 2.0, ('NOUN', 'handcuffs'): 1.0, ('SYM', '...'): 4.0, ('NOUN', 'sacristy'): 1.0, ('VERB', 'adjoining'): 1.0, ('NOUN', 'hospital'): 4.0, ('NOUN', 'chapel'): 1.0, ('ADJ', 'downtown'): 1.0, ('NOUN', 'Toledo'): 1.0, ('PROPN', 'Easter'): 1.0, ('NUM', '1980'): 3.0, ('NOUN', 'Investigators'): 1.0, ('NOUN', 'nun'): 1.0, ('PROPN', 'Margaret'): 1.0, ('PROPN', 'Ann'): 1.0, ('PROPN', 'Pahl'): 1.0, ('NUM', '71'): 1.0, ('VERB', 'strangled'): 1.0, ('VERB', 'stabbed'): 1.0, ('NOUN', 'chest'): 1.0, ('VERB', 'forming'): 1.0, ('NOUN', 'shape'): 3.0, ('VERB', 'inverted'): 1.0, ('NOUN', 'cross'): 1.0, ('VERB', 'recognized'): 1.0, ('ADJ', 'Satanic'): 1.0, ('NOUN', 'symbol'): 1.0, ('NOUN', 'altar'): 1.0, ('NOUN', 'cloth'): 1.0, ('VERB', 'draped'): 1.0, ('ADV', 'half'): 1.0, ('ADJ', 'naked'): 3.0, ('ADV', 'sexually'): 1.0, ('VERB', 'assaulted'): 1.0, ('VERB', 'humiliate'): 1.0, ('NOUN', 'prosecutor'): 1.0, ('PROPN', 'Mandros'): 1.0, ('VERB', 'closing'): 1.0, ('NOUN', 'arguments'): 1.0, ('PRON', 'everyone'): 2.0, ('PROPN', 'God'): 2.0, ('NOUN', 'sentencing'): 1.0, ('ADJ', 'tearful'): 1.0, ('PROPN', 'Claudia'): 1.0, ('PROPN', 'Vercellotti'): 1.0, ('PROPN', 'Survivors'): 1.0, ('PROPN', 'Network'): 2.0, ('PRON', 'Those'): 1.0, ('VERB', 'Abused'): 1.0, ('PROPN', 'Priests'): 1.0, ('PROPN', 'SNAP'): 3.0, ('VERB', 'reopen'): 1.0, ('VERB', 'rot'): 1.0, ('NOUN', 'hell'): 3.0, ('ADV', 'More'): 1.0, ('NOUN', 'prosecutors'): 3.0, ('NOUN', 'tools'): 2.0, ('NOUN', 'will'): 3.0, ('ADJ', 'horrific'): 1.0, ('NOUN', 'defendants'): 1.0, ('ADV', 'seemingly'): 1.0, ('NOUN', 'institutions'): 2.0, ('NOUN', 'victims'): 3.0, ('NOUN', 'witnesses'): 2.0, ('VERB', 'stay'): 3.0, ('ADJ', 'silent'): 2.0, ('VERB', 'changes'): 1.0, ('VERB', 'speak'): 4.0, ('VERB', 'protected'): 2.0, ('NOUN', 'weapon'): 9.0, ('NOUN', 'letter'): 17.0, ('NOUN', 'opener'): 1.0, ('ADV', 'Finally'): 3.0, ('NOUN', 'email'): 5.0, ('NOUN', 'follow'): 1.0, ('NOUN', 'reopening'): 1.0, ('PROPN', 'Atlanta'): 3.0, ('NOUN', 'Murders'): 1.0, ('PROPN', 'Dekalb'): 1.0, ('PROPN', 'County'): 1.0, ('PROPN', 'Police'): 1.0, ('PROPN', 'Chief'): 1.0, ('PROPN', 'Graham'): 1.0, ('VERB', 'reopened'): 1.0, ('NOUN', 'investigations'): 1.0, ('ADV', 'mysteriously'): 1.0, ('VERB', 'stepping'): 1.0, ('ADV', 'down'): 3.0, ('NOUN', 'replacement'): 2.0, ('PROPN', 'Brown'): 1.0, ('NOUN', 'supervisor'): 1.0, ('PROPN', 'PD'): 1.0, ('NOUN', 'murders'): 11.0, ('ADJ', 'subsequent'): 1.0, ('NOUN', 'matter'): 3.0, ('ADV', 'individually'): 1.0, ('ADV', 'institutionally'): 1.0, ('NOUN', 'domain'): 1.0, ('NOUN', 'lots'): 1.0, ('PROPN', 'jeff'): 1.0, ('VERB', 'thank'): 1.0, ('PROPN', 'god'): 3.0, ('NOUN', 'souls'): 2.0, ('PROPN', 'congress'): 1.0, ('NOUN', 'CRACKDOWN'): 1.0, ('ADP', 'ON'): 1.0, ('NOUN', 'POLYGAMY'): 1.0, ('NOUN', 'GROUP'): 1.0, ('ADJ', 'Small'): 1.0, ('ADJ', 'polygamous'): 4.0, ('ADJ', 'southwestern'): 2.0, ('ADJ', 'watchful'): 1.0, ('ADV', 'fairly'): 1.0, ('ADJ', 'benign'): 1.0, ('NOUN', 'sect'): 4.0, ('PROPN', 'Fundamentalist'): 1.0, ('PROPN', 'Latter'): 1.0, ('PROPN', 'Day'): 1.0, ('PROPN', 'Saints'): 1.0, ('PROPN', 'FLDS'): 3.0, ('PROPN', 'Mormonism'): 1.0, ('NUM', '1890'): 1.0, ('PROPN', 'Warren'): 2.0, ('PROPN', 'Jeffs'): 7.0, ('NUM', 'Ten'): 1.0, ('ADV', 'Most'): 1.0, ('VERB', 'Wanted'): 1.0, ('NOUN', 'Fugitives'): 1.0, ('VERB', 'caps'): 1.0, ('NOUN', 'approach'): 3.0, ('NOUN', 'practices'): 1.0, ('VERB', 'grew'): 2.0, ('NOUN', 'followers'): 2.0, ('NOUN', 'communities'): 4.0, ('NOUN', 'assets'): 5.0, ('NUM', '110'): 1.0, ('NOUN', 'school'): 4.0, ('ADJ', 'intertwined'): 1.0, ('NOUN', 'ex-members'): 1.0, ('ADV', 'mainly'): 1.0, ('ADJ', 'underage'): 2.0, ('VERB', 'marry'): 2.0, ('ADJ', 'secluded'): 1.0, ('NOUN', 'compounds'): 1.0, ('ADJ', 'federal'): 1.0, ('VERB', 'crack'): 1.0, ('ADV', 'vigorously'): 1.0, ('ADV', 'Specifically'): 1.0, ('PROPN', 'Utah'): 1.0, ('PROPN', 'Arizona'): 1.0, ('VERB', 'arranging'): 1.0, ('ADJ', 'spiritual'): 5.0, ('NOUN', 'marriages'): 1.0, ('NOUN', 'weekend'): 3.0, ('PROPN', 'Salt'): 2.0, ('PROPN', 'Lake'): 2.0, ('PROPN', 'City'): 6.0, ('PROPN', 'Phoenix'): 2.0, ('VERB', 'traveling'): 2.0, ('NOUN', 'bodyguards'): 1.0, ('NOUN', 'past'): 2.0, ('ADJ', 'apocalyptic'): 1.0, ('PROPN', 'Christian'): 2.0, ('PROPN', 'Identity'): 1.0, ('NOUN', 'hate'): 1.0, ('NOUN', 'philosophies'): 1.0, ('PROPN', 'Aryan'): 1.0, ('ADJ', 'neo-Nazi'): 1.0, ('PROPN', 'Creativity'): 1.0, ('PROPN', 'Movement'): 1.0, ('VERB', 'preached'): 1.0, ('NOUN', 'racism'): 1.0, ('NOUN', 'race'): 1.0, ('NOUN', 'devil'): 1.0, ('NOUN', 'evil'): 4.0, ('ADP', 'unto'): 1.0, ('NOUN', 'earth'): 1.0, ('VERB', 'cited'): 2.0, ('PROPN', 'Southern'): 1.0, ('PROPN', 'Poverty'): 1.0, ('PROPN', 'Law'): 2.0, ('NOUN', 'editorial'): 1.0, ('PROPN', 'Deseret'): 1.0, ('PROPN', 'Morning'): 1.0, ('PROPN', 'News'): 2.0, ('NOUN', 'unwillingness'): 1.0, ('VERB', 'stir'): 1.0, ('NOUN', 'hornet'): 1.0, ('VERB', 'contributed'): 1.0, ('NOUN', 'kid'): 1.0, ('NOUN', 'glove'): 1.0, ('NOUN', 'lawmakers'): 1.0, ('X', 'http://www.csmonitor.com/2006/0509/p02s01-ussc.html?s=t5'): 1.0, ('ADV', 'Maybe'): 4.0, ('VERB', 'missing'): 1.0, ('ADV', 'exactly'): 3.0, ('VERB', 'Loose'): 1.0, ('NOUN', \"AK47's\"): 1.0, ('PUNCT', '?!?!?'): 1.0, ('INTJ', 'Never'): 1.0, ('INTJ', 'mind'): 1.0, ('DET', 'No'): 8.0, ('VERB', 'worry'): 1.0, ('NOUN', 'kind'): 1.0, ('ADV', 'regularly'): 1.0, ('VERB', 'scheduled'): 3.0, ('NOUN', 'programming'): 1.0, ('VERB', 's'): 1.0, ('PROPN', 'Raw'): 1.0, ('PROPN', 'Story'): 1.0, ('PRON', 'i'): 4.0, ('VERB', 'count'): 1.0, ('INTJ', 'like'): 1.0, ('NOUN', 'headlines'): 2.0, ('ADJ', 'Funny'): 1.0, ('VERB', 'squirelled'): 1.0, ('VERB', 'meaning'): 1.0, ('VERB', 'hint'): 1.0, ('NOUN', 'conspiracy'): 1.0, ('NOUN', 'abusers'): 1.0, ('ADJ', 'satanic'): 2.0, ('NOUN', 'underground'): 1.0, ('NOUN', 'judge'): 6.0, ('NOUN', 'bench'): 1.0, ('VERB', 'obeying'): 1.0, ('PRON', \"it's\"): 1.0, ('NOUN', 'feet'): 1.0, ('ADV', 'Personally'): 1.0, ('NOUN', 'damn'): 1.0, ('NOUN', 'adults'): 1.0, ('VERB', 'invovled'): 1.0, ('NOUN', 'marrying'): 1.0, ('NOUN', 'off'): 1.0, ('VERB', 'enfurates'): 1.0, ('NOUN', 'cults'): 1.0, ('NOUN', 'multimillions'): 1.0, ('NOUN', 'poverty'): 4.0, ('NOUN', 'cult'): 2.0, ('VERB', 'reassigns'): 1.0, ('NOUN', 'wives'): 1.0, ('NOUN', 'husband'): 2.0, ('VERB', 'displeases'): 1.0, ('VERB', 'reward'): 1.0, ('NOUN', 'Husbands'): 1.0, ('NOUN', 'mothers'): 2.0, ('NOUN', 'teen'): 2.0, ('NOUN', 'daughters'): 2.0, ('NOUN', 'retardation'): 1.0, ('ADV', 'else'): 1.0, ('ADJ', 'genetic'): 1.0, ('VERB', 'needs'): 2.0, ('NOUN', 'gene'): 1.0, ('NOUN', 'parents'): 4.0, ('NOUN', 'vegetable'): 1.0, ('NOUN', 'doctor'): 1.0, ('VERB', 'begs'): 1.0, ('VERB', 'intermarrying'): 1.0, ('ADJ', 'pure'): 1.0, ('NOUN', 'fumarase'): 1.0, ('NOUN', 'deficiency'): 1.0, ('PROPN', 'kingston'): 1.0, ('VERB', 'convicted'): 1.0, ('VERB', 'beating'): 1.0, ('NOUN', 'daughter'): 2.0, ('ADJ', 'unconcious'): 1.0, ('NOUN', 'beating'): 1.0, ('NOUN', 'shit'): 1.0, ('VERB', 'shoot'): 3.0, ('ADJ', 'juicy'): 1.0, ('NOUN', 'profits'): 1.0, ('VERB', 'evaporate'): 1.0, ('NOUN', 'comparison'): 4.0, ('NOUN', 'contractors'): 1.0, ('ADJ', 'awash'): 1.0, ('NOUN', 'tax'): 6.0, ('VERB', 'milking'): 1.0, ('NOUN', 'angle'): 1.0, ('NOUN', 'whoopsie'): 1.0, ('NOUN', 'daisy'): 1.0, ('INTJ', 'huh'): 1.0, ('INTJ', 'Christ'): 1.0, ('NOUN', 'bit'): 5.0, ('ADJ', 'tired'): 1.0, ('VERB', 'treating'): 1.0, ('ADP', 'but'): 2.0, ('ADJ', 'gullible'): 2.0, ('NOUN', 'idiots'): 2.0, ('VERB', 'suppose'): 1.0, ('ADV', 'damn'): 2.0, ('ADJ', 'lucky'): 1.0, ('PROPN', 'GOP'): 1.0, ('VERB', 'outsourcing'): 1.0, ('VERB', 'bypassing'): 1.0, ('NOUN', 'channels'): 1.0, ('VERB', 'turning'): 1.0, ('ADJ', 'unruly'): 1.0, ('NOUN', 'cast'): 1.0, ('NOUN', 'characters'): 1.0, ('NOUN', 'order'): 7.0, ('VERB', 'create'): 1.0, ('NOUN', 'preparation'): 1.0, ('X', 'http://www.rawstory.com/news/2006/US_outsourcing_special_operations_intelligence_gathering_0413.html'): 1.0, ('VERB', 'deserves'): 1.0, ('PROPN', 'MEK'): 1.0, ('ADV', 'itself'): 1.0, ('ADV', 'unquestionably'): 1.0, ('VERB', 'boast'): 1.0, ('PROPN', 'Mujahedeen'): 9.0, ('NOUN', 'affair'): 1.0, ('NOUN', 'generations'): 2.0, ('NOUN', 'martyrs'): 4.0, ('NOUN', 'grandmothers'): 1.0, ('VERB', 'meeting'): 1.0, ('VERB', 'grown'): 1.0, ('NOUN', 'schools'): 4.0, ('PROPN', 'Ashraf'): 1.0, ('NOUN', 'Family'): 2.0, ('NOUN', 'nights'): 1.0, ('PROPN', 'Fridays'): 2.0, ('VERB', 'invaded'): 1.0, ('PROPN', 'Kuwait'): 2.0, ('PROPN', 'France'): 5.0, ('PROPN', 'Denmark'): 1.0, ('PROPN', 'England'): 1.0, ('VERB', 'raised'): 3.0, ('NOUN', 'guardians'): 1.0, ('ADV', 'usually'): 4.0, ('NUM', '19'): 2.0, ('VERB', 'fill'): 1.0, ('NOUN', 'ranks'): 1.0, ('ADJ', 'youngest'): 1.0, ('NOUN', 'generation'): 5.0, ('SCONJ', 'Though'): 1.0, ('NOUN', 'decided'): 1.0, ('NOUN', 'boys'): 3.0, ('VERB', 'taught'): 1.0, ('ADV', 'blindly'): 1.0, ('DET', 'Every'): 2.0, ('VERB', 'beginning'): 3.0, ('ADV', 'young'): 1.0, ('NUM', '1'): 8.0, ('NOUN', 'poster'): 2.0, ('PROPN', 'Massoud'): 2.0, ('PROPN', 'Maryam'): 2.0, ('VERB', 'salute'): 1.0, ('VERB', 'shout'): 2.0, ('NOUN', 'praises'): 1.0, ('PROPN', 'Nadereh'): 1.0, ('PROPN', 'Afshari'): 3.0, ('NOUN', 'believer'): 1.0, ('ADJ', 'responsible'): 6.0, ('PROPN', 'gulf'): 1.0, ('PROPN', 'war'): 1.0, ('ADJ', 'German'): 2.0, ('VERB', 'tried'): 1.0, ('VERB', 'absorb'): 1.0, ('PROPN', 'Rajavis'): 1.0, ('VERB', 'brainwash'): 2.0, ('VERB', 'control'): 2.0, ('PRON', 'Which'): 1.0, ('NOUN', 'empowerment'): 1.0, ('NOUN', 'enlightenment'): 1.0, ('NOUN', 'sacrifice'): 1.0, ('VERB', 'inspired'): 3.0, ('NOUN', 'wisdom'): 1.0, ('NOUN', 'Link'): 1.0, ('PRON', 'One'): 1.0, ('ADJ', 'bright'): 1.0, ('NOUN', 'fellows'): 1.0, ('VERB', 'set'): 1.0, ('NOUN', 'deals'): 2.0, ('VERB', 'care'): 1.0, ('PROPN', 'Jeff'): 4.0, ('PROPN', 'SPLOID.com'): 1.0, ('NOUN', 'topic'): 2.0, ('X', 'http://www.sploid.com/news/2006/05/evil_priest_gui.php'): 1.0, ('INTJ', 'Umm'): 1.0, ('NOUN', 'Question'): 1.0, ('NOUN', 'Mark'): 1.0, ('PROPN', 'Jimmy'): 1.0, ('PROPN', 'Plant'): 1.0, ('PROPN', 'Blogshares'): 1.0, ('NOUN', 'fantasy'): 1.0, ('NOUN', 'blogosphere'): 1.0, ('NOUN', 'trading'): 2.0, ('NOUN', 'game'): 4.0, ('ADJ', 'pretend'): 1.0, ('PROPN', 'Satan'): 2.0, ('X', 'http://www.laweekly.com/general/features/satan-loves-you/13454/'): 1.0, ('NOUN', 'metal'): 23.0, ('INTJ', 'sorry'): 1.0, ('NOUN', 'strippers'): 2.0, ('NOUN', 'dope'): 2.0, ('INTJ', 'Okay'): 1.0, ('ADV', 'partly'): 2.0, ('VERB', 'evolves'): 1.0, ('NOUN', 'music'): 9.0, ('VERB', 'belongs'): 1.0, ('ADJ', 'weirder'): 1.0, ('NOUN', 'guardian'): 2.0, ('NOUN', 'morality'): 9.0, ('NOUN', 'Praise'): 1.0, ('VERB', 'Given'): 2.0, ('NOUN', 'void'): 2.0, ('ADJ', 'educational'): 3.0, ('NOUN', 'establishments'): 2.0, ('NOUN', 'religions'): 1.0, ('VERB', 'guiding'): 1.0, ('NOUN', 'righteousness'): 1.0, ('VERB', 'fallen'): 1.0, ('INTJ', 'Hmmmmmm'): 2.0, ('PUNCT', '....................'): 1.0, ('ADJ', 'underground'): 1.0, ('NOUN', 'trauma'): 1.0, ('VERB', 'controlled'): 1.0, ('NOUN', 'slaves'): 1.0, ('PROPN', 'wikipedia'): 1.0, ('NOUN', 'stub'): 1.0, ('VERB', 'waiting'): 2.0, ('X', 'http://en.wikipedia.org/wiki/Aerocom'): 1.0, ('VERB', 'hates'): 1.0, ('SYM', ':)'): 3.0, ('NOUN', 'chants'): 1.0, ('PROPN', 'Manson'): 1.0, ('NOUN', 'tees'): 1.0, ('VERB', 'bearing'): 1.0, ('VERB', 'Kill'): 1.0, ('PROPN', 'Columbine'): 2.0, ('VERB', 'acting'): 1.0, ('ADV', 'indescriminately'): 1.0, ('VERB', 'murdered'): 1.0, ('NOUN', 'classmates'): 1.0, ('VERB', 'brainwashed'): 1.0, ('NOUN', 'fans'): 3.0, ('ADV', 'literally'): 1.0, ('VERB', 'fulfilling'): 1.0, ('NOUN', 'paradigm'): 1.0, ('INTJ', 'er'): 1.0, ('INTJ', 'no'): 2.0, ('ADJ', 'phoney'): 1.0, ('PRON', 'thy'): 3.0, ('NOUN', 'neighbor'): 3.0, ('INTJ', 'hell'): 1.0, ('ADJ', 'REAL'): 1.0, ('NOUN', 'MORALITY'): 1.0, ('PUNCT', '?????'): 1.0, ('INTJ', 'hmmmm'): 1.0, ('ADV', 'indeed'): 1.0, ('VERB', 'detect'): 1.0, ('VERB', 'hissing'): 1.0, ('NOUN', 'lisp'): 1.0, ('NOUN', 'serpent'): 1.0, ('NOUN', 'article'): 4.0, ('ADJ', 'deathly'): 1.0, ('NOUN', 'inversion'): 3.0, ('ADP', 'ala'): 1.0, ('PROPN', 'satanism'): 7.0, ('NOUN', 'brainwashing'): 1.0, ('NOUN', 'regard'): 3.0, ('ADV', 'Perhaps'): 3.0, ('NOUN', 'phenomenon'): 1.0, ('NOUN', 'invitation'): 3.0, ('ADJ', 'couter-cultural'): 1.0, ('X', 'over'): 1.0, ('NOUN', 'hypocrisy'): 1.0, ('NOUN', 'percentage'): 1.0, ('NOUN', 'Morality'): 1.0, ('NOUN', 'crowd'): 2.0, ('NOUN', 'lip'): 1.0, ('VERB', 'loving'): 1.0, ('NOUN', 'love'): 2.0, ('ADP', 'beyond'): 2.0, ('NOUN', 'congregation'): 1.0, ('VERB', 'checked'): 5.0, ('PROPN', 'Satanists'): 1.0, ('NOUN', 'destruction'): 2.0, ('VERB', 'dealt'): 1.0, ('PROPN', 'Christians'): 5.0, ('NOUN', 'Irony'): 2.0, ('ADV', 'Long'): 1.0, ('NOUN', 'editor'): 1.0, ('ADJ', 'simple'): 1.0, ('ADJ', 'cut'): 1.0, ('VERB', 'entering'): 1.0, ('ADV', 'previously'): 1.0, ('ADJ', 'unexplored'): 1.0, ('ADJ', 'socio-political'): 1.0, ('VERB', 'require'): 2.0, ('ADJ', 'further'): 5.0, ('NOUN', 'examination'): 1.0, ('NOUN', 'introspection'): 1.0, ('PROPN', 'Kevin'): 3.0, ('PROPN', 'Coogan'): 2.0, ('NOUN', 'publication'): 1.0, ('ADJ', 'previous'): 2.0, ('NOUN', 'quote'): 1.0, ('ADJ', 'Imaginary'): 2.0, ('ADJ', 'romantic'): 1.0, ('ADJ', 'varied'): 1.0, ('ADJ', 'gloomy'): 1.0, ('ADJ', 'monotonous'): 1.0, ('ADJ', 'barren'): 1.0, ('ADJ', 'boring'): 2.0, ('NOUN', 'good'): 3.0, ('ADJ', 'marvelous'): 1.0, ('ADJ', 'intoxicating'): 1.0, ('PROPN', 'Simone'): 1.0, ('PROPN', 'Weil'): 1.0, ('PROPN', 'Starroute'): 1.0, ('NOUN', 'D'): 1.0, ('ADJ', 'above'): 3.0, ('ADP', 'due'): 3.0, ('NOUN', 'thought'): 2.0, ('ADV', 'appartently'): 1.0, ('ADV', 'cognitively'): 2.0, ('ADJ', 'functional'): 2.0, ('ADJ', 'senseless'): 2.0, ('ADJ', 'incompetent'): 2.0, ('NOUN', 'sign'): 3.0, ('ADJ', 'ulterior'): 2.0, ('ADJ', 'hidden'): 2.0, ('NOUN', 'motives'): 2.0, ('VERB', 'trust'): 2.0, ('VERB', 'befuddled'): 2.0, ('AUX', 'wilt'): 2.0, ('X', 'http://youtube.com/watch?v=d46_ctqDmI4'): 1.0, ('SCONJ', 'To'): 1.0, ('VERB', 'buying'): 1.0, ('NOUN', 'BS'): 1.0, ('ADV', 'specifically'): 2.0, ('NOUN', 'tidbit'): 1.0, ('ADV', 'waaaaaaaaaaaaay'): 1.0, ('NOUN', 'definitions'): 1.0, ('X', 'http://news.bbc.co.uk/2/hi/programmes/this_world/4446342.stm'): 1.0, ('ADJ', 'relentless'): 1.0, ('VERB', 'uncover'): 1.0, ('ADJ', 'shocking'): 1.0, ('PROPN', 'Italy'): 1.0, ('NOUN', 'tale'): 1.0, ('VERB', 'gripped'): 1.0, ('NUM', '1998'): 5.0, ('PROPN', 'Fabio'): 5.0, ('PROPN', 'Tollis'): 4.0, ('PROPN', 'Chiara'): 2.0, ('PROPN', 'Marino'): 1.0, ('ADV', 'both'): 1.0, ('VERB', 'disappeared'): 2.0, ('NOUN', 'pub'): 1.0, ('PROPN', 'Midnight'): 1.0, ('NOUN', 'centre'): 1.0, ('NOUN', 'scene'): 2.0, ('PROPN', 'Milan'): 1.0, ('PROPN', 'Michele'): 3.0, ('VERB', 'attend'): 4.0, ('NOUN', 'concerts'): 1.0, ('NOUN', 'festivals'): 1.0, ('PROPN', 'Europe'): 13.0, ('VERB', 'handing'): 1.0, ('NOUN', 'leaflets'): 1.0, ('VERB', 'quizzing'): 1.0, ('NOUN', 'forms'): 2.0, ('ADJ', 'obsessed'): 2.0, ('NOUN', 'images'): 2.0, ('NOUN', 'collection'): 2.0, ('NOUN', 'paraphernalia'): 1.0, ('NOUN', 'bedroom'): 1.0, ('ADJ', 'convinced'): 1.0, ('NOUN', 'disappearance'): 1.0, ('VERB', \"'re\"): 3.0, ('ADJ', 'inseparable'): 1.0, ('INTJ', 'Yeah'): 1.0, ('NOUN', 'posts'): 1.0, ('ADV', 'wholly'): 1.0, ('VERB', 'expand'): 2.0, ('VERB', 'include'): 6.0, ('ADJ', 'freaky'): 1.0, ('NOUN', 'bozos'): 1.0, ('VERB', 'Talk'): 1.0, ('VERB', 'beat'): 1.0, ('PROPN', 'Smartwolf'): 1.0, ('NOUN', 'principle'): 2.0, ('VERB', 'employed'): 1.0, ('ADV', 'deliberately'): 3.0, ('PROPN', 'GWB'): 1.0, ('NOUN', 'neocons'): 1.0, ('PROPN', 'Jesus'): 1.0, ('PROPN', 'Nazareth'): 1.0, ('ADJ', 'ignorant'): 1.0, ('ADV', 'why'): 2.0, ('NOUN', 'phonies'): 1.0, ('NOUN', 'appellation'): 1.0, ('NOUN', 'imposters'): 1.0, ('PRON', 'Everything'): 1.0, ('NOUN', 'lie'): 2.0, ('ADP', 'above'): 1.0, ('VERB', 'credit'): 2.0, ('VERB', 'serves'): 3.0, ('NOUN', 'purposes'): 2.0, ('VERB', 'identified'): 1.0, ('ADV', 'fully'): 3.0, ('NOUN', 'subterfuge'): 1.0, ('NOUN', 'magician'): 1.0, ('ADJ', 'rigorous'): 1.0, ('VERB', 'perceive'): 1.0, ('VERB', 'See'): 3.0, ('CONJ', 'not'): 1.0, ('NOUN', 'avant'): 1.0, ('NOUN', 'guard'): 1.0, ('NOUN', 'art'): 4.0, ('ADJ', '20th'): 1.0, ('NOUN', 'Century'): 1.0, ('ADJ', 'Italian'): 1.0, ('NOUN', 'Futurism'): 1.0, ('VERB', 'labeled'): 1.0, ('ADJ', 'degenerate'): 3.0, ('PROPN', 'Socialists'): 1.0, ('PROPN', 'Adolf'): 1.0, ('PROPN', 'Hitler'): 1.0, ('NOUN', 'exhibition'): 1.0, ('NUM', '1937'): 2.0, ('VERB', 'curating'): 1.0, ('ADJ', 'parallel'): 1.0, ('ADJ', 'Aryan'): 1.0, ('NOUN', 'Expressionist'): 1.0, ('ADJ', 'quick'): 3.0, ('NOUN', 'overview'): 1.0, ('PROPN', 'Entartete'): 1.0, ('PROPN', 'Kunst'): 1.0, ('PROPN', 'Degenerate'): 1.0, ('PROPN', 'Art'): 1.0, ('NOUN', 'exhibit'): 1.0, ('X', 'http://en.wikipedia.org/wiki/Degenerate_art'): 1.0, ('NOUN', 'notion'): 2.0, ('NOUN', 'degenerate'): 1.0, ('VERB', 'appropriated'): 1.0, ('PROPN', 'Nazis'): 1.0, ('VERB', 'traced'): 1.0, ('ADJ', 'Jewish'): 3.0, ('NOUN', 'intellectual'): 1.0, ('PROPN', 'Max'): 1.0, ('PROPN', 'Nordau'): 2.0, ('NUM', '1892'): 1.0, ('VERB', 'authored'): 1.0, ('PROPN', 'Degeneration'): 1.0, ('X', 'co'): 1.0, ('NOUN', 'founder'): 1.0, ('PROPN', 'World'): 10.0, ('PROPN', 'Zionist'): 1.0, ('PROPN', 'Organization'): 1.0, ('ADV', 'apparently'): 3.0, ('AUX', 'Are'): 3.0, ('NOUN', 'adherent'): 1.0, ('ADJ', 'Crowleyan'): 2.0, ('PROPN', 'Thelema'): 2.0, ('NOUN', 'ilk'): 1.0, ('VERB', 'appropriating'): 2.0, ('NOUN', 'appropriators'): 1.0, ('VERB', 'redirect'): 1.0, ('NOUN', 'energy'): 1.0, ('AUX', 'Does'): 3.0, ('NOUN', 'reappropriation'): 1.0, ('VERB', 'lead'): 2.0, ('VERB', 'originate'): 1.0, ('VERB', 'Seems'): 1.0, ('ADJ', 'imaginary'): 1.0, ('PROPN', 'Whore'): 2.0, ('PROPN', 'Babalon'): 1.0, ('PROPN', 'Babylon'): 1.0, ('ADJ', 'accurate'): 1.0, ('ADJ', 'valid'): 1.0, ('X', 'socio'): 1.0, ('NOUN', 'conundrum'): 1.0, ('ADJ', 'Confused'): 1.0, ('INTJ', 'Good'): 1.0, ('PROPN', 'enemy'): 1.0, ('ADV', 'Why'): 3.0, ('VERB', 'state'): 1.0, ('ADJ', 'undecided'): 1.0, ('NOUN', 'answers'): 2.0, ('PROPN', 'Balance'): 2.0, ('PROPN', 'Coil'): 2.0, ('VERB', 'wish'): 1.0, ('PROPN', 'Wikipedia'): 1.0, ('ADJ', 'easier'): 1.0, ('VERB', 'spending'): 1.0, ('ADJ', 'obscure'): 1.0, ('ADJ', 'detailed'): 2.0, ('X', 'http://en.wikipedia.org/wiki/John_Balance'): 1.0, ('X', 'http://www.guardian.co.uk/obituaries/story/0,3604,1371372,00.html'): 1.0, ('VERB', 'blended'): 3.0, ('NOUN', 'systems'): 3.0, ('PROPN', 'Shamanism'): 1.0, ('PROPN', 'Christianity'): 2.0, ('PROPN', 'Buddhism'): 1.0, ('PROPN', 'Paganism'): 1.0, ('PROPN', 'Hermeticism'): 1.0, ('PROPN', 'Gnosticism'): 1.0, ('VERB', 'imbued'): 1.0, ('ADJ', 'vast'): 4.0, ('NOUN', 'output'): 1.0, ('ADJ', 'magickal'): 1.0, ('NOUN', 'current'): 1.0, ('VERB', 'designed'): 1.0, ('ADV', 'functionally'): 1.0, ('ADJ', 'transcendent'): 1.0, ('NOUN', 'listeners'): 1.0, ('NOUN', 'kg'): 1.0, ('ADP', 'per'): 4.0, ('NUM', '1000'): 2.0, ('NOUN', 'tonnes'): 2.0, ('NOUN', 'containers'): 1.0, ('NOUN', 'load'): 1.0, ('NUM', '20,000'): 2.0, ('PROPN', 'E@tG'): 1.0, ('VERB', 'formulate'): 1.0, ('VERB', 'acquiring'): 1.0, ('VERB', 'Reporting'): 1.0, ('PROPN', 'Susan'): 1.0, ('PROPN', 'Polk'): 1.0, ('NOUN', 'bay'): 1.0, ('NOUN', 'farmer'): 1.0, ('NOUN', 'event'): 2.0, ('INTJ', 'Lo'): 1.0, ('INTJ', 'behold'): 1.0, ('PROPN', 'Mormons'): 1.0, ('NOUN', 'booth'): 1.0, ('VERB', 'delivering'): 1.0, ('ADJ', 'Mormon'): 1.0, ('NOUN', 'genealogy'): 1.0, ('VERB', 'track'): 1.0, ('VERB', 'amazed'): 1.0, ('NOUN', 'spiel'): 1.0, ('VERB', 'delivered'): 3.0, ('ADJ', 'Kindly'): 1.0, ('ADV', 'utterly'): 1.0, ('VERB', 'programmed'): 1.0, ('DET', 'either'): 2.0, ('VERB', 'watch'): 3.0, ('AUX', 'Has'): 1.0, ('ADV', 'purportedly'): 1.0, ('PROPN', 'Ahmadinejad'): 1.0, ('X', 'http://www.thetruthseeker.co.uk/article.asp?id=4503'): 1.0, ('VERB', 'verified'): 1.0, ('ADV', 'succintly'): 1.0, ('ADJ', 'succint'): 1.0, ('ADV', 'regardless'): 1.0, ('NOUN', 'processes'): 2.0, ('PROPN', 'W'): 1.0, ('VERB', 'picture'): 1.0, ('NOUN', 'contrast'): 1.0, ('PRON', 'whoever'): 1.0, ('CONJ', 'vs.'): 1.0, ('VERB', 'Bring'): 1.0, ('NOUN', 'condition'): 2.0, ('ADP', 'Of'): 1.0, ('NOUN', 'farce'): 1.0, ('ADV', 'definately'): 1.0, ('NOUN', 'in'): 1.0, ('PROPN', 'Loose'): 1.0, ('PROPN', 'Change'): 1.0, ('PROPN', 'Alex'): 1.0, ('PROPN', 'Jones'): 1.0, ('PROPN', 'Rio'): 1.0, ('PROPN', 'Grande'): 1.0, ('PROPN', 'Blood'): 1.0, ('ADJ', 'Important'): 1.0, ('PROPN', 'Kos'): 1.0, ('PROPN', 'Diary'): 1.0, ('PROPN', 'bush'): 1.0, ('VERB', 'gathering'): 1.0, ('NOUN', 'storm'): 1.0, ('NOUN', 'scandals'): 1.0, ('X', 'http://www.dailykos.com/story/2006/5/12/232746/857'): 1.0, ('SYM', '='): 1.0, ('ADJ', 'Executive'): 5.0, ('ADJ', 'creepy'): 1.0, ('ADJ', 'scary'): 1.0, ('VERB', 'accumulating'): 1.0, ('ADV', 'basically'): 1.0, ('VERB', 'sieze'): 1.0, ('VERB', 'feels'): 1.0, ('NOUN', 'minute'): 1.0, ('SYM', '=='): 1.0, ('VERB', 'revoke'): 1.0, ('PROPN', 'Executive'): 1.0, ('NUM', '13011'): 2.0, ('PROPN', 'exmearden'): 1.0, ('PROPN', 'Fri'): 1.0, ('NUM', '2006'): 1.0, ('NUM', '08:27:46'): 1.0, ('PROPN', 'PDT'): 1.0, ('SCONJ', 'In'): 1.0, ('VERB', 'scanning'): 1.0, ('PROPN', 'whitehouse.gov'): 1.0, ('VERB', 'noticed'): 2.0, ('NOUN', 'Order'): 3.0, ('NOUN', 'Amendments'): 1.0, ('NUM', '11030'): 1.0, ('NUM', '13279'): 1.0, ('NUM', '13339'): 1.0, ('NUM', '13381'): 1.0, ('NUM', '13389'): 1.0, ('NOUN', 'Revocation'): 1.0, ('VERB', 'frequent'): 1.0, ('VERB', 'jumping'): 1.0, ('NOUN', 'fray'): 1.0, ('PROPN', 'blogger'): 1.0, ('VERB', 'sign'): 2.0, ('PROPN', 'TroubleFunk'): 1.0, ('PROPN', 'Tronicus'): 2.0, ('ADV', 'Simply'): 1.0, ('VERB', 'permeate'): 1.0, ('NOUN', 'Mystery'): 1.0, ('NOUN', 'Tradition'): 1.0, ('ADJ', 'Eastern'): 1.0, ('NOUN', 'confluence'): 1.0, ('NOUN', 'spirituality'): 1.0, ('ADJ', 'alchemical'): 2.0, ('ADV', 'initially'): 2.0, ('NOUN', 'appropriation'): 1.0, ('ADV', 'Similarly'): 1.0, ('VERB', 'invoking'): 1.0, ('ADJ', 'Egyptian'): 21.0, ('NOUN', 'gods'): 1.0, ('NOUN', 'goddesses'): 1.0, ('NOUN', 'reaffirmation'): 1.0, ('ADJ', 'oppressive'): 1.0, ('ADJ', 'hierarchical'): 1.0, ('VERB', 'partition'): 1.0, ('VERB', 'compartmentalize'): 1.0, ('ADJ', 'personal'): 4.0, ('NOUN', 'weirdness'): 1.0, ('ADJ', 'hardcore'): 1.0, ('NOUN', 'factions'): 1.0, ('VERB', 'integrate'): 1.0, ('NOUN', 'synthesis'): 1.0, ('NOUN', 'regards'): 3.0, ('ADJ', 'vacuous'): 1.0, ('NOUN', 'perspective'): 2.0, ('NOUN', 'culture'): 2.0, ('AUX', 'Shall'): 1.0, ('VERB', 'censor'): 1.0, ('NOUN', 'works'): 4.0, ('NOUN', 'poet'): 1.0, ('NOUN', 'Charles'): 1.0, ('PROPN', 'Baudelaire'): 2.0, ('NOUN', 'artists'): 4.0, ('VERB', 'checking'): 1.0, ('NOUN', 'knee'): 1.0, ('NOUN', 'jerk'): 1.0, ('NOUN', 'reaction'): 1.0, ('VERB', 'apologize'): 1.0, ('NOUN', 'over-generalizations'): 1.0, ('NOUN', 'literalist'): 1.0, ('ADJ', 'True'): 1.0, ('NOUN', 'believers'): 1.0, ('ADJ', 'hung'): 1.0, ('ADJ', 'up'): 1.0, ('NOUN', 'top'): 3.0, ('ADJ', 'pyrimidal'): 1.0, ('NOUN', 'structure'): 1.0, ('NOUN', 'cloak'): 1.0, ('NOUN', 'motivations'): 1.0, ('NOUN', 'masses'): 1.0, ('VERB', 'boggles'): 1.0, ('PRON', 'ANYONE'): 1.0, ('NOUN', 'value'): 3.0, ('PROPN', 'christian'): 1.0, ('AUX', 'shall'): 1.0, ('PRON', 'ye'): 1.0, ('NOUN', 'hackles'): 1.0, ('VERB', 'haul'): 1.0, ('NOUN', 'Rock'): 1.0, ('CONJ', 'N'): 1.0, ('NOUN', 'Roll'): 1.0, ('SCONJ', 'Whie'): 1.0, ('VERB', 'subscribe'): 1.0, ('VERB', 'listen'): 4.0, ('ADV', 'patently'): 1.0, ('ADJ', 'rediculous'): 1.0, ('VERB', 'bother'): 1.0, ('VERB', 'single'): 1.0, ('ADJ', 'specific'): 2.0, ('VERB', 'pin'): 1.0, ('NOUN', 'Sub-cultures'): 1.0, ('NOUN', 'lable'): 1.0, ('VERB', 'implies'): 1.0, ('VERB', 'develope'): 1.0, ('NOUN', 'vacuum'): 1.0, ('ADV', 'to'): 1.0, ('NOUN', 'egos'): 1.0, ('NOUN', 'metaphors'): 1.0, ('ADV', 'literaly'): 1.0, ('SCONJ', 'wether'): 1.0, ('VERB', 'banging'): 1.0, ('PROPN', 'Canibal'): 1.0, ('PROPN', 'Corpse'): 1.0, ('VERB', 'humming'): 1.0, ('ADV', 'along'): 2.0, ('ADV', 'Onward'): 1.0, ('PROPN', 'Soldiers'): 1.0, ('NOUN', 'mixing'): 1.0, ('NOUN', 'militarism'): 1.0, ('NOUN', 'respect'): 1.0, ('VERB', 'adhear'): 1.0, ('NOUN', 'non-violence'): 1.0, ('NOUN', 'forgiveness'): 2.0, ('NOUN', 'charity'): 1.0, ('PROPN', 'Smartwolves'): 1.0, ('NOUN', 'monopoly'): 1.0, ('NOUN', 'lifestyle'): 1.0, ('NOUN', 'Peace'): 1.0, ('NOUN', 'hissy'): 1.0, ('NOUN', 'fit'): 1.0, ('SCONJ', 'over'): 1.0, ('VERB', 'wanting'): 1.0, ('VERB', 'buy'): 4.0, ('VERB', 'wants'): 2.0, ('ADJ', 'sus'): 1.0, ('VERB', 'located'): 1.0, ('ADV', 'enroute'): 1.0, ('NOUN', 'storage'): 2.0, ('NOUN', 'stooges'): 1.0, ('NOUN', 'pretext'): 1.0, ('NOUN', 'Thanks'): 18.0, ('PROPN', 'Nick'): 1.0, ('PROPN', 'Keck'): 1.0, ('VERB', 'passing'): 1.0, ('NOUN', 'speech'): 4.0, ('NOUN', 'gentleman'): 1.0, ('ADJ', 'advisory'): 1.0, ('ADJ', 'multi-nation'): 1.0, ('NOUN', 'corporation'): 2.0, ('ADJ', 'worthwhile'): 1.0, ('NOUN', 'read'): 1.0, ('PROPN', 'View'): 1.0, ('PROPN', 'Eye'): 1.0, ('PROPN', 'Storm'): 1.0, ('NOUN', 'Talk'): 1.0, ('PROPN', 'Haim'): 1.0, ('PROPN', 'Harari'): 1.0, ('PROPN', 'Advisory'): 1.0, ('PROPN', 'Board'): 1.0, ('ADJ', 'multi-national'): 2.0, ('ADJ', 'technological'): 1.0, ('NOUN', 'entertainment'): 1.0, ('NOUN', 'occasion'): 1.0, ('PROPN', 'Chairman'): 1.0, ('VERB', 'suggested'): 1.0, ('NOUN', 'events'): 3.0, ('NOUN', 'Government'): 6.0, ('ADJ', 'privileged'): 1.0, ('NUM', '200'): 1.0, ('ADJ', 'proverbial'): 1.0, ('VERB', 'question'): 1.0, ('ADJ', 'fascinating'): 1.0, ('NOUN', 'thoughts'): 2.0, ('VERB', 'touch'): 1.0, ('VERB', 'prefer'): 1.0, ('VERB', 'devote'): 1.0, ('NOUN', 'remarks'): 1.0, ('ADJ', 'broader'): 1.0, ('VERB', 'refer'): 5.0, ('PROPN', 'Morocco'): 1.0, ('ADV', 'predominantly'): 2.0, ('ADJ', 'Moslem'): 10.0, ('ADJ', 'non-Arab'): 1.0, ('ADJ', 'non-Moslem'): 1.0, ('NOUN', 'minorities'): 1.0, ('ADJ', 'immediate'): 3.0, ('NOUN', 'neighborhood'): 1.0, ('NOUN', 'spite'): 2.0, ('NOUN', 'upheaval'): 2.0, ('INTJ', 'Yes'): 1.0, ('ADJ', 'mass'): 2.0, ('VERB', 'happening'): 2.0, ('ADV', 'right'): 3.0, ('PROPN', 'Sudan'): 3.0, ('VERB', 'massacring'): 1.0, ('ADJ', 'Christian'): 1.0, ('PROPN', 'Algeria'): 2.0, ('NOUN', 'village'): 1.0, ('PROPN', 'Algerians'): 1.0, ('VERB', 'invade'): 1.0, ('VERB', 'endanger'): 1.0, ('VERB', 'butcher'): 1.0, ('PROPN', 'Egypt'): 6.0, ('ADJ', 'poison'): 1.0, ('NOUN', 'gas'): 2.0, ('PROPN', 'Yemen'): 1.0, ('NOUN', \"60's\"): 1.0, ('PROPN', 'Assad'): 1.0, ('PROPN', 'Father'): 1.0, ('PROPN', 'El'): 3.0, ('PROPN', 'Hamma'): 1.0, ('PROPN', 'Syria'): 7.0, ('ADJ', 'Libyan'): 1.0, ('NOUN', 'blowing'): 1.0, ('PROPN', 'Pan'): 1.0, ('PROPN', 'Am'): 1.0, ('NOUN', 'root'): 1.0, ('NOUN', 'entire'): 1.0, ('ADJ', 'dysfunctional'): 3.0, ('NOUN', 'standard'): 2.0, ('PROPN', 'Arab'): 2.0, ('PROPN', 'league'): 2.0, ('PROPN', 'Mauritania'): 1.0, ('PROPN', 'Gulf'): 2.0, ('ADJ', 'total'): 7.0, ('PROPN', 'EU'): 1.0, ('NOUN', 'expansion'): 2.0, ('CONJ', 'either'): 5.0, ('VERB', 'combined'): 1.0, ('PROPN', 'GDP'): 3.0, ('PROPN', 'Netherlands'): 1.0, ('CONJ', 'plus'): 1.0, ('PROPN', 'Belgium'): 2.0, ('ADJ', 'equal'): 3.0, ('PROPN', 'California'): 1.0, ('ADP', 'Within'): 1.0, ('ADJ', 'meager'): 1.0, ('NOUN', 'gaps'): 2.0, ('ADJ', 'poor'): 2.0, ('VERB', 'succeeding'): 1.0, ('ADJ', 'corrupt'): 1.0, ('ADJ', 'social'): 3.0, ('SCONJ', 'below'): 1.0, ('PROPN', 'Western'): 3.0, ('NUM', '150'): 1.0, ('ADJ', 'Human'): 1.0, ('ADP', 'below'): 1.0, ('ADJ', 'grotesque'): 1.0, ('PROPN', 'Libya'): 3.0, ('NOUN', 'Chair'): 1.0, ('PROPN', 'Human'): 1.0, ('PROPN', 'Rights'): 1.0, ('NOUN', 'commission'): 1.0, ('VERB', 'prepared'): 2.0, ('NOUN', 'committee'): 1.0, ('NOUN', 'intellectuals'): 2.0, ('NOUN', 'auspices'): 1.0, ('NOUN', 'books'): 1.0, ('VERB', 'translated'): 1.0, ('PROPN', 'Greece'): 1.0, ('VERB', 'translates'): 1.0, ('NOUN', 'publications'): 2.0, ('NOUN', 'Birth'): 1.0, ('NOUN', 'rates'): 1.0, ('NOUN', 'decline'): 2.0, ('ADJ', 'wealthy'): 1.0, ('ADJ', 'advanced'): 2.0, ('NOUN', 'cultures'): 2.0, ('VERB', 'creates'): 2.0, ('NOUN', 'breeding'): 2.0, ('ADJ', 'cruel'): 3.0, ('NOUN', 'dictators'): 1.0, ('NOUN', 'fanaticism'): 1.0, ('NOUN', 'incitement'): 8.0, ('NOUN', 'Civilization'): 1.0, ('PROPN', 'Judaism'): 1.0, ('ADJ', 'decent'): 1.0, ('ADJ', 'devout'): 1.0, ('PROPN', 'Moslems'): 3.0, ('ADJ', 'outside'): 3.0, ('VERB', 'develops'): 1.0, ('NOUN', 'Islamophobia'): 1.0, ('NOUN', 'accomplices'): 1.0, ('NOUN', 'omission'): 1.0, ('VERB', 'applies'): 2.0, ('NOUN', 'wrong'): 1.0, ('ADJ', 'afraid'): 3.0, ('VERB', 'express'): 1.0, ('VERB', 'amplified'): 1.0, ('ADJ', 'rampant'): 1.0, ('VERB', 'acknowledges'): 1.0, ('NOUN', 'pillars'): 1.0, ('NOUN', 'World'): 4.0, ('NOUN', 'Conflict'): 1.0, ('ADJ', 'undeclared'): 1.0, ('PUNCT', '*'): 8.0, ('NOUN', 'element'): 2.0, ('NOUN', 'Suicide'): 2.0, ('NOUN', 'invention'): 1.0, ('NOUN', 'expression'): 1.0, ('ADV', 'lately'): 1.0, ('ADJ', 'potent'): 1.0, ('ADJ', 'psychological'): 1.0, ('ADJ', 'direct'): 2.0, ('ADV', 'relatively'): 1.0, ('NOUN', 'accidents'): 1.0, ('ADV', 'quantitatively'): 1.0, ('ADJ', 'lethal'): 2.0, ('NOUN', 'earthquakes'): 1.0, ('ADJ', 'More'): 1.0, ('VERB', 'die'): 1.0, ('NOUN', 'AIDS'): 1.0, ('PROPN', 'Russians'): 2.0, ('NOUN', 'murderers'): 6.0, ('ADJ', 'every'): 1.0, ('NOUN', 'fuss'): 1.0, ('NOUN', 'killings'): 1.0, ('ADJ', 'spectacular'): 1.0, ('ADJ', 'frightening'): 1.0, ('VERB', 'dismembered'): 1.0, ('ADJ', 'horrible'): 2.0, ('ADJ', 'lifelong'): 1.0, ('NOUN', 'injuries'): 1.0, ('NOUN', 'detail'): 3.0, ('ADJ', 'hysterical'): 1.0, ('NOUN', 'tourism'): 1.0, ('NOUN', 'industry'): 1.0, ('DET', 'quite'): 1.0, ('PROPN', 'Bali'): 1.0, ('PROPN', 'Turkey'): 4.0, ('NOUN', 'fear'): 2.0, ('ADJ', 'undisputed'): 1.0, ('ADJ', 'preventive'): 2.0, ('ADJ', 'determined'): 2.0, ('NOUN', 'murderer'): 2.0, ('VERB', 'penetrated'): 1.0, ('NOUN', 'thinking'): 1.0, ('ADV', 'constantly'): 1.0, ('VERB', 'improving'): 1.0, ('VERB', 'arrange'): 2.0, ('VERB', 'murder'): 1.0, ('VERB', 'board'): 1.0, ('NOUN', 'plane'): 2.0, ('VERB', 'explode'): 3.0, ('PRON', 'Who'): 3.0, ('ADJ', 'crowded'): 2.0, ('NOUN', 'detector'): 2.0, ('NOUN', 'lines'): 1.0, ('NOUN', 'counters'): 1.0, ('ADJ', 'busy'): 1.0, ('NOUN', 'travel'): 4.0, ('VERB', 'Put'): 2.0, ('NOUN', 'train'): 3.0, ('NOUN', 'station'): 2.0, ('PROPN', 'Spain'): 6.0, ('NOUN', 'buses'): 2.0, ('VERB', 'Protect'): 1.0, ('NOUN', 'movie'): 1.0, ('NOUN', 'theaters'): 1.0, ('NOUN', 'concert'): 2.0, ('NOUN', 'halls'): 1.0, ('NOUN', 'supermarkets'): 1.0, ('NOUN', 'shopping'): 1.0, ('NOUN', 'malls'): 1.0, ('NOUN', 'hospitals'): 4.0, ('NOUN', 'guards'): 3.0, ('NOUN', 'hall'): 1.0, ('ADV', 'somewhat'): 1.0, ('NOUN', 'vulnerability'): 1.0, ('ADJ', 'strict'): 1.0, ('NOUN', 'controls'): 2.0, ('VERB', 'eliminate'): 3.0, ('NOUN', 'Money'): 1.0, ('ADJ', 'cold'): 1.0, ('ADJ', 'blooded'): 1.0, ('ADJ', 'murderous'): 1.0, ('ADJ', 'fanatic'): 2.0, ('NOUN', 'beliefs'): 1.0, ('VERB', 'blown'): 2.0, ('NOUN', 'politician'): 1.0, ('NOUN', 'relative'): 2.0, ('ADJ', 'influential'): 1.0, ('ADV', 'truly'): 3.0, ('ADJ', 'supreme'): 2.0, ('NOUN', 'act'): 5.0, ('NOUN', 'fervor'): 1.0, ('VERB', 'Are'): 1.0, ('NOUN', 'benefits'): 1.0, ('PROPN', 'Heaven'): 1.0, ('ADV', 'Instead'): 1.0, ('ADJ', 'outcast'): 1.0, ('ADJ', 'naive'): 2.0, ('ADJ', 'retarded'): 1.0, ('VERB', 'incited'): 1.0, ('NOUN', 'hotheads'): 1.0, ('VERB', 'promise'): 1.0, ('NOUN', 'delights'): 1.0, ('VERB', 'pay'): 4.0, ('ADV', 'handsomely'): 1.0, ('VERB', 'performed'): 3.0, ('NOUN', 'despair'): 2.0, ('ADJ', 'poorest'): 1.0, ('VERB', 'happens'): 1.0, ('ADJ', 'desperate'): 1.0, ('NOUN', 'continents'): 1.0, ('NOUN', 'Desperation'): 1.0, ('NOUN', 'reconnaissance'): 1.0, ('NOUN', 'transportation'): 1.0, ('PROPN', 'Bremmer'): 1.0, ('VERB', 'exploded'): 1.0, ('ADJ', 'vicious'): 1.0, ('ADJ', 'inhuman'): 1.0, ('ADJ', 'cynical'): 1.0, ('VERB', 'funded'): 1.0, ('ADJ', 'affluent'): 1.0, ('NOUN', 'being'): 1.0, ('NOUN', 'hunger'): 1.0, ('ADJ', 'identical'): 2.0, ('NOUN', 'pirates'): 1.0, ('NOUN', 'seas'): 1.0, ('ADJ', 'offensive'): 2.0, ('ADJ', 'crucial'): 3.0, ('ADJ', 'united'): 1.0, ('NOUN', 'pyramid'): 1.0, ('VERB', 'arresting'): 1.0, ('NOUN', 'dealer'): 1.0, ('VERB', 'tolerate'): 1.0, ('VERB', 'thrive'): 1.0, ('ADJ', 'humble'): 2.0, ('NOUN', 'trains'): 1.0, ('PROPN', 'Istanbul'): 1.0, ('NOUN', 'beginning'): 1.0, ('NOUN', 'unity'): 3.0, ('ADJ', 'Civilized'): 1.0, ('NOUN', 'horror'): 2.0, ('ADV', 'absolutely'): 1.0, ('ADJ', 'indispensable'): 1.0, ('SCONJ', 'Until'): 1.0, ('VERB', 'wakes'): 1.0, ('NOUN', 'ingredient'): 1.0, ('NOUN', 'lies'): 5.0, ('NOUN', 'Words'): 1.0, ('NOUN', 'politicians'): 1.0, ('VERB', 'lie'): 2.0, ('ADJ', 'professional'): 1.0, ('NOUN', 'norms'): 1.0, ('NOUN', 'diplomacy'): 1.0, ('ADJ', 'childish'): 1.0, ('ADJ', 'deliberate'): 2.0, ('NOUN', 'fabrications'): 3.0, ('NOUN', 'heights'): 1.0, ('ADJ', 'incredible'): 1.0, ('NOUN', 'provocation'): 1.0, ('NOUN', 'plot'): 5.0, ('PROPN', 'Information'): 2.0, ('PROPN', 'Mouhamad'): 1.0, ('PROPN', 'Said'): 1.0, ('NOUN', 'Disinformation'): 1.0, ('NOUN', 'tactic'): 2.0, ('ADJ', 'preposterous'): 1.0, ('SCONJ', 'without'): 4.0, ('VERB', 'ridiculed'): 1.0, ('NOUN', 'milieu'): 1.0, ('NOUN', 'icon'): 1.0, ('NOUN', 'jester'): 1.0, ('ADJ', 'respectable'): 1.0, ('NOUN', 'newspapers'): 2.0, ('VERB', 'prevent'): 2.0, ('NOUN', 'credence'): 1.0, ('NOUN', 'liars'): 1.0, ('ADV', 'After'): 1.0, ('NOUN', 'anti-Semite'): 1.0, ('ADJ', 'subtle'): 3.0, ('PROPN', 'holocaust'): 1.0, ('NOUN', 'temple'): 1.0, ('PROPN', 'Jerusalem'): 1.0, ('VERB', 'report'): 2.0, ('ADJ', 'daily'): 2.0, ('NOUN', 'occurrence'): 1.0, ('VERB', 'finance'): 1.0, ('VERB', 'dispatch'): 1.0, ('VERB', 'condemn'): 3.0, ('NOUN', 'TV'): 4.0, ('NOUN', 'cameras'): 1.0, ('NOUN', 'audience'): 1.0, ('NOUN', 'routine'): 1.0, ('ADJ', 'opposite'): 1.0, ('PROPN', 'Arabic'): 2.0, ('NOUN', 'Incitement'): 1.0, ('VERB', 'accompanied'): 2.0, ('NOUN', 'pictures'): 1.0, ('VERB', 'mutilated'): 1.0, ('VERB', 'distort'): 1.0, ('ADJ', 'Little'): 1.0, ('NOUN', 'hatred'): 3.0, ('NOUN', 'admiration'): 1.0, ('NOUN', 'sets'): 1.0, ('VERB', 'tuned'): 1.0, ('NOUN', 'soap'): 1.0, ('NOUN', 'operas'): 1.0, ('NOUN', 'shows'): 1.0, ('VERB', 'recommend'): 4.0, ('NOUN', 'demonstration'): 2.0, ('PROPN', 'Berlin'): 1.0, ('VERB', 'carrying'): 7.0, ('NOUN', 'banners'): 1.0, ('VERB', 'supporting'): 2.0, ('VERB', 'featuring'): 1.0, ('NOUN', 'babies'): 1.0, ('VERB', 'dressed'): 1.0, ('VERB', 'defined'): 1.0, ('PROPN', 'Arafat'): 3.0, ('NOUN', 'activists'): 1.0, ('VERB', 'walks'): 1.0, ('NOUN', 'restaurant'): 2.0, ('NOUN', 'mid-day'): 1.0, ('VERB', 'eats'): 1.0, ('VERB', 'observes'): 1.0, ('VERB', 'eating'): 2.0, ('NOUN', 'lunch'): 7.0, ('ADJ', 'adjacent'): 1.0, ('NOUN', 'tables'): 1.0, ('VERB', 'pays'): 2.0, ('VERB', 'blows'): 1.0, ('PRON', 'herself'): 2.0, ('VERB', 'rolling'): 1.0, ('NOUN', 'martyr'): 1.0, ('NOUN', 'activist'): 1.0, ('ADJ', 'European'): 4.0, ('NOUN', 'Dignitaries'): 1.0, ('ADJ', 'bereaved'): 1.0, ('VERB', 'flows'): 1.0, ('ADJ', 'actual'): 1.0, ('VERB', 'equips'): 1.0, ('VERB', 'sends'): 2.0, ('ADJ', 'Orwellian'): 1.0, ('NOUN', 'nomenclature'): 1.0, ('VERB', 'realize'): 4.0, ('ADJ', 'emotional'): 1.0, ('NOUN', 'infrastructure'): 3.0, ('PROPN', 'Joseph'): 1.0, ('PROPN', 'Goebbels'): 1.0, ('VERB', 'repeat'): 1.0, ('VERB', 'outperformed'): 1.0, ('NOUN', 'successors'): 1.0, ('ADJ', 'Huge'): 1.0, ('NOUN', 'amounts'): 1.0, ('VERB', 'channeled'): 1.0, ('ADJ', 'concentric'): 1.0, ('NOUN', 'spheres'): 1.0, ('ADJ', 'inner'): 7.0, ('NOUN', 'circle'): 16.0, ('VERB', 'funds'): 1.0, ('NOUN', 'hideouts'): 1.0, ('ADJ', 'soft'): 1.0, ('NOUN', 'circles'): 7.0, ('ADV', 'primarily'): 2.0, ('VERB', 'financed'): 3.0, ('ADP', 'until'): 2.0, ('ADJ', 'Communist'): 1.0, ('NOUN', 'regimes'): 2.0, ('PROPN', 'Palestinian'): 2.0, ('PROPN', 'Authority'): 2.0, ('ADJ', 'safe'): 2.0, ('NOUN', 'havens'): 1.0, ('ADJ', 'wholesale'): 1.0, ('NOUN', 'vendors'): 1.0, ('NOUN', 'planners'): 1.0, ('NOUN', 'preachers'): 2.0, ('NOUN', 'living'): 3.0, ('NOUN', 'welfare'): 1.0, ('VERB', 'feed'): 1.0, ('ADJ', 'hungry'): 1.0, ('NOUN', 'schooling'): 1.0, ('NOUN', 'ignorance'): 2.0, ('VERB', 'operates'): 1.0, ('NOUN', 'madrasas'): 1.0, ('VERB', 'inciting'): 2.0, ('ADJ', 'electronic'): 1.0, ('VERB', 'printed'): 2.0, ('ADJ', 'inferior'): 1.0, ('ADJ', 'unthinkable'): 2.0, ('ADJ', 'minimal'): 2.0, ('VERB', 'blaming'): 1.0, ('NOUN', 'miseries'): 1.0, ('ADJ', 'outer'): 8.0, ('NOUN', 'donations'): 3.0, ('NOUN', 'Governments'): 1.0, ('NOUN', \"NGO's\"): 1.0, ('NOUN', 'goals'): 2.0, ('ADJ', 'noble'): 1.0, ('VERB', 'infested'): 1.0, ('VERB', 'exploited'): 1.0, ('ADJ', 'Saudi'): 2.0, ('NOUN', 'victim'): 1.0, ('PROPN', 'Saudis'): 1.0, ('VERB', 'financing'): 1.0, ('ADV', 'Figuratively'): 1.0, ('VERB', 'speaking'): 1.0, ('ADV', 'inwards'): 1.0, ('ADV', 'rather'): 1.0, ('NOUN', 'blackmail'): 2.0, ('ADJ', 'horrifying'): 1.0, ('NOUN', 'factor'): 1.0, ('NOUN', 'Half'): 1.0, ('ADJ', 'receptive'): 1.0, ('VERB', 'guaranteeing'): 1.0, ('ADJ', 'blind'): 5.0, ('ADV', 'comfortably'): 2.0, ('NOUN', 'loot'): 1.0, ('NOUN', 'camps'): 1.0, ('NOUN', 'Jihad'): 1.0, ('VERB', 'join'): 1.0, ('VERB', 'packaged'): 1.0, ('NOUN', 'tours'): 1.0, ('NOUN', 'hotspots'): 1.0, ('VERB', 'ski'): 1.0, ('PROPN', 'Switzerland'): 1.0, ('PROPN', 'Mrs.'): 1.0, ('VERB', 'lives'): 1.0, ('PROPN', 'Paris'): 1.0, ('VERB', 'allegedly'): 1.0, ('ADJ', 'bankrupt'): 1.0, ('ADJ', 'typical'): 2.0, ('NOUN', 'ringleader'): 1.0, ('PROPN', 'Aksa'): 1.0, ('NOUN', 'brigade'): 1.0, ('NOUN', 'payment'): 1.0, ('NOUN', 'couple'): 6.0, ('VERB', 'performing'): 1.0, ('NOUN', 'retail'): 1.0, ('X', '4'): 1.0, ('ADJ', 'fourth'): 1.0, ('NOUN', 'breaking'): 1.0, ('NOUN', 'laws'): 3.0, ('ADJ', 'civilized'): 5.0, ('NOUN', 'liberties'): 2.0, ('ADJ', 'fashioned'): 2.0, ('NOUN', 'habits'): 1.0, ('VERB', 'respecting'): 1.0, ('NOUN', 'sites'): 1.0, ('NOUN', 'symbols'): 1.0, ('VERB', 'avoiding'): 1.0, ('NOUN', 'mutilation'): 1.0, ('NOUN', 'shields'): 1.0, ('ADV', 'Never'): 1.0, ('PROPN', 'Nazi'): 1.0, ('NOUN', 'disregard'): 1.0, ('VERB', 'observe'): 1.0, ('NOUN', 'student'): 2.0, ('NOUN', 'science'): 2.0, ('VERB', 'debates'): 1.0, ('ADJ', 'anti-democratic'): 1.0, ('VERB', 'winning'): 1.0, ('ADJ', 'democratic'): 2.0, ('VERB', 'abolishing'): 1.0, ('ADJ', 'Other'): 2.0, ('NOUN', 'aspects'): 1.0, ('NOUN', 'society'): 4.0, ('NOUN', 'limitations'): 1.0, ('NOUN', 'policeman'): 1.0, ('VERB', 'open'): 2.0, ('NOUN', 'dealers'): 1.0, ('NOUN', 'penalty'): 1.0, ('ADJ', 'multiple'): 1.0, ('NOUN', 'dilemmas'): 1.0, ('NOUN', 'set'): 1.0, ('VERB', 'raid'): 2.0, ('VERB', 'storm'): 1.0, ('NOUN', 'priests'): 1.0, ('NOUN', 'hostages'): 1.0, ('NOUN', 'ambulance'): 1.0, ('VERB', 'strip'): 1.0, ('VERB', 'pretended'): 1.0, ('NOUN', 'bomb'): 1.0, ('NOUN', 'belly'): 1.0, ('VERB', 'standing'): 1.0, ('VERB', 'hidden'): 1.0, ('ADJ', 'mental'): 1.0, ('NOUN', 'arch-murderer'): 1.0, ('VERB', 'moves'): 1.0, ('NOUN', 'location'): 1.0, ('NOUN', 'dilemma'): 1.0, ('VERB', 'Suppose'): 1.0, ('NOUN', 'address'): 3.0, ('PROPN', 'Teheran'): 1.0, ('NOUN', 'atrocity'): 1.0, ('NOUN', 'responsibility'): 4.0, ('VERB', 'promising'): 1.0, ('NOUN', 'interviews'): 5.0, ('VERB', 'issues'): 1.0, ('NOUN', 'condemnations'): 1.0, ('VERB', 'host'): 1.0, ('VERB', 'invite'): 1.0, ('NOUN', 'functions'): 1.0, ('VERB', 'treat'): 1.0, ('NOUN', 'dignitary'): 1.0, ('NOUN', 'homework'): 1.0, ('VERB', 'figure'): 1.0, ('NOUN', 'illusions'): 1.0, ('ADJ', 'lawless'): 2.0, ('NOUN', 'ice'): 2.0, ('NOUN', 'hockey'): 1.0, ('VERB', 'sending'): 2.0, ('NOUN', 'ballerina'): 1.0, ('NOUN', 'skater'): 1.0, ('VERB', 'knock'): 1.0, ('NOUN', 'heavyweight'): 1.0, ('NOUN', 'boxer'): 1.0, ('NOUN', 'chess'): 1.0, ('SCONJ', 'against'): 1.0, ('NOUN', 'cannibals'): 1.0, ('ADJ', 'International'): 2.0, ('VERB', 'handle'): 1.0, ('VERB', 'throw'): 1.0, ('NOUN', 'stones'): 1.0, ('VERB', 'shoots'): 1.0, ('NOUN', 'immunity'): 1.0, ('VERB', 'sheltered'): 1.0, ('ADV', 'royally'): 1.0, ('VERB', 'pretends'): 1.0, ('VERB', 'claims'): 2.0, ('VERB', 'arrest'): 1.0, ('NOUN', 'crooks'): 1.0, ('VERB', 'demand'): 1.0, ('NOUN', 'protection'): 1.0, ('VERB', 'attack'): 2.0, ('NOUN', 'criminals'): 1.0, ('VERB', 'repeating'): 1.0, ('ADJ', 'temporary'): 2.0, ('NOUN', 'evolution'): 1.0, ('VERB', 'adapted'): 1.0, ('NOUN', 'punishment'): 1.0, ('ADV', 'during'): 1.0, ('ADV', 'after'): 2.0, ('NOUN', 'rules'): 1.0, ('NOUN', 'twilight'): 1.0, ('ADJ', 'pretty'): 1.0, ('NOUN', 'run'): 3.0, ('VERB', 'educate'): 1.0, ('VERB', 'eliminated'): 1.0, ('ADJ', 'financial'): 6.0, ('NOUN', 'starvation'): 1.0, ('VERB', 'organizing'): 1.0, ('ADJ', 'elite'): 1.0, ('NOUN', 'counter-propaganda'): 1.0, ('NOUN', 'boycott'): 1.0, ('NOUN', 'access'): 6.0, ('NOUN', 'internet'): 2.0, ('ADP', 'Above'): 2.0, ('NOUN', 'determination'): 2.0, ('VERB', 'Allow'): 1.0, ('VERB', 'depart'): 1.0, ('ADJ', 'malignant'): 1.0, ('NOUN', 'tumor'): 3.0, ('VERB', 'remove'): 1.0, ('ADV', 'surgically'): 1.0, ('VERB', 'starve'): 1.0, ('VERB', 'preventing'): 2.0, ('VERB', 'expanding'): 2.0, ('VERB', 'comment'): 1.0, ('ADJ', 'justified'): 1.0, ('ADJ', 'pre-war'): 1.0, ('NOUN', 'map'): 1.0, ('PROPN', 'Lebanon'): 2.0, ('NOUN', 'colony'): 1.0, ('NOUN', 'conquest'): 1.0, ('NOUN', 'territories'): 1.0, ('ADJ', 'unfriendly'): 1.0, ('VERB', 'encircled'): 1.0, ('NOUN', 'republics'): 1.0, ('ADJ', 'surprising'): 2.0, ('VERB', 'incite'): 1.0, ('NOUN', 'uprising'): 1.0, ('NOUN', 'plan'): 2.0, ('VERB', 'encircle'): 1.0, ('VERB', 'resulting'): 2.0, ('NOUN', 'ambitions'): 1.0, ('NOUN', 'ideology'): 1.0, ('NOUN', 'supremacy'): 1.0, ('ADJ', 'ruthless'): 1.0, ('VERB', 'execute'): 2.0, ('ADJ', 'elaborate'): 1.0, ('NOUN', 'traces'): 1.0, ('PROPN', 'Embassies'): 1.0, ('VERB', 'develop'): 1.0, ('NOUN', 'conservatives'): 1.0, ('NOUN', 'virtuoso'): 1.0, ('NOUN', 'cop'): 2.0, ('ADP', 'versus'): 1.0, ('VERB', 'sponsors'): 1.0, ('VERB', 'funding'): 1.0, ('PROPN', 'Hezbollah'): 2.0, ('PROPN', 'Uzbekistan'): 1.0, ('NOUN', 'consortium'): 1.0, ('NOUN', 'players'): 1.0, ('VERB', 'trade'): 2.0, ('VERB', 'appease'): 1.0, ('VERB', 'refuse'): 1.0, ('NOUN', 'signals'): 1.0, ('VERB', 'dry'): 1.0, ('NOUN', 'conglomerate'): 1.0, ('ADJ', 'pointless'): 1.0, ('NOUN', 'enterprises'): 1.0, ('NOUN', 'needs'): 2.0, ('VERB', 'collaborate'): 2.0, ('ADV', 'beautifully'): 1.0, ('ADJ', 'fertile'): 1.0, ('VERB', 'monitor'): 6.0, ('NOUN', 'finances'): 1.0, ('NOUN', 'relief'): 1.0, ('VERB', 'react'): 1.0, ('ADJ', 'forceful'): 1.0, ('ADV', 'decisively'): 1.0, ('NOUN', 'naivety'): 1.0, ('VERB', 'surrender'): 1.0, ('VERB', 'yielded'): 1.0, ('VERB', 'matter'): 1.0, ('VERB', 'matters'): 1.0, ('VERB', 'won'): 1.0, ('ADV', 'surely'): 1.0, ('ADJ', 'costly'): 2.0, ('VERB', 'expelling'): 2.0, ('VERB', 'forbidding'): 1.0, ('NOUN', 'veils'): 1.0, ('VERB', 'functioning'): 1.0, ('ADJ', 'judicial'): 1.0, ('NOUN', 'equality'): 1.0, ('NOUN', 'ideas'): 2.0, ('ADJ', 'racial'): 1.0, ('NOUN', 'defamation'): 1.0, ('NOUN', 'avoidance'): 1.0, ('VERB', 'regarding'): 5.0, ('NOUN', 'places'): 1.0, ('NOUN', 'worship'): 1.0, ('ADJ', 'inflammatory'): 1.0, ('ADV', 'carefully'): 1.0, ('NOUN', 'transition'): 2.0, ('VERB', 'paving'): 1.0, ('VERB', 'prevail'): 1.0, ('VERB', 'takes'): 1.0, ('NOUN', 'landscape'): 1.0, ('NOUN', 'victory'): 1.0, ('NOUN', 'key'): 1.0, ('ADJ', 'understandable'): 1.0, ('NOUN', 'recoil'): 1.0, ('NOUN', 'wars'): 1.0, ('NOUN', 'horrors'): 1.0, ('PROPN', 'II'): 1.0, ('VERB', 'cost'): 1.0, ('ADJ', 'additional'): 1.0, ('NOUN', 'tide'): 1.0, ('VERB', 'associated'): 3.0, ('NOUN', 'faction'): 2.0, ('PROPN', 'Egyptian'): 5.0, ('PROPN', 'Vanguards'): 4.0, ('PROPN', 'Conquest'): 4.0, ('VERB', 'seeking'): 4.0, ('VERB', 'recreate'): 2.0, ('NOUN', 'taking'): 2.0, ('PROPN', 'mecca'): 2.0, ('NOUN', 'struggle'): 2.0, ('VERB', 'acquired'): 2.0, ('NOUN', 'anthrax'): 38.0, ('NOUN', 'strain'): 10.0, ('PROPN', 'Veterinary'): 2.0, ('PROPN', 'Medical'): 3.0, ('PROPN', 'Diagnostic'): 2.0, ('PROPN', 'Lab'): 2.0, ('ADJ', 'senior'): 5.0, ('X', 'counter'): 3.0, ('ADV', 'abroad'): 3.0, ('NOUN', 'islamists'): 14.0, ('ADJ', 'available'): 4.0, ('VERB', 'requested'): 3.0, ('PROPN', 'Anthrax'): 1.0, ('NOUN', 'Means'): 1.0, ('NOUN', 'Motive'): 1.0, ('NOUN', 'Modus'): 1.0, ('NOUN', 'Operandi'): 1.0, ('NOUN', 'Opportunity'): 1.0, ('NOUN', 'Homeland'): 1.0, ('PROPN', 'Central'): 2.0, ('VERB', 'disclosed'): 1.0, ('PROPN', 'Atta'): 5.0, ('PROPN', 'Zacarias'): 7.0, ('PROPN', 'Moussaoui'): 11.0, ('NOUN', 'inquiries'): 5.0, ('NOUN', 'cropdusters'): 3.0, ('VERB', 'contemplated'): 2.0, ('VERB', 'dispersing'): 2.0, ('VERB', 'included'): 3.0, ('NOUN', 'rumor'): 1.0, ('NOUN', 'setback'): 1.0, ('PROPN', 'KSM'): 8.0, ('PROPN', 'Rawalpindi'): 2.0, ('NOUN', 'Panel'): 1.0, ('NOUN', 'experts'): 2.0, ('VERB', 'concluded'): 3.0, ('VERB', 'restrained'): 1.0, ('NOUN', 'difficulties'): 2.0, ('PROPN', '9/11'): 2.0, ('NOUN', 'Staff'): 1.0, ('ADJ', 'ambitious'): 1.0, ('NOUN', 'advances'): 1.0, ('VERB', 'produce'): 1.0, ('ADJ', 'prior'): 3.0, ('PROPN', 'Director'): 2.0, ('PROPN', 'Tenet'): 1.0, ('NUM', '9'): 1.0, ('NOUN', 'Spring'): 3.0, ('VERB', 'named'): 3.0, ('PROPN', 'Shukrijumah'): 2.0, ('PROPN', 'Jafar'): 4.0, ('PROPN', 'Pilot'): 4.0, ('NOUN', 'wave'): 6.0, ('VERB', 'casing'): 1.0, ('NOUN', 'Photographs'): 1.0, ('NOUN', 'computer'): 5.0, ('NOUN', 'disc'): 2.0, ('NOUN', 'locks'): 1.0, ('NOUN', 'passengers'): 1.0, ('NOUN', 'bulletin'): 1.0, ('NOUN', 'surveillance'): 4.0, ('VERB', 'disperse'): 2.0, ('VERB', 'building'): 1.0, ('NOUN', 'mailings'): 8.0, ('ADV', 'bureaucratically'): 1.0, ('ADJ', 'impolite'): 1.0, ('VERB', 'contest'): 1.0, ('ADJ', 'lone'): 1.0, ('NOUN', 'scientist'): 2.0, ('VERB', 'argued'): 3.0, ('NOUN', 'Fall'): 3.0, ('NOUN', 'warning'): 2.0, ('PROPN', 'Princeton'): 1.0, ('NOUN', 'islamist'): 3.0, ('NOUN', 'scholar'): 1.0, ('PROPN', 'Lewis'): 1.0, ('VERB', 'explained'): 5.0, ('VERB', 'disagree'): 1.0, ('NOUN', 'innocents'): 1.0, ('VERB', 'sanctioned'): 1.0, ('NOUN', 'notice'): 1.0, ('ADJ', 'biochemical'): 4.0, ('PROPN', 'Prophet'): 4.0, ('NOUN', 'guidance'): 3.0, ('PROPN', 'Scheuer'): 2.0, ('NOUN', 'analyst'): 1.0, ('VERB', 'warn'): 1.0, ('NOUN', 'letters'): 7.0, ('NOUN', 'branches'): 1.0, ('PROPN', 'D.C.'): 2.0, ('ADJ', 'symbolic'): 1.0, ('NOUN', 'detention'): 1.0, ('NOUN', 'sheik'): 4.0, ('PROPN', 'Abdel'): 4.0, ('PROPN', 'Trade'): 3.0, ('NUM', '1993'): 4.0, ('ADJ', 'Handwritten'): 1.0, ('NOUN', 'notes'): 1.0, ('NOUN', 'laptop'): 2.0, ('NOUN', 'capture'): 2.0, ('NOUN', '#'): 3.0, ('NOUN', 'production'): 5.0, ('NOUN', 'spray'): 1.0, ('NOUN', 'dryer'): 1.0, ('VERB', 'addressed'): 3.0, ('NOUN', 'recruitment'): 2.0, ('NOUN', 'expertise'): 2.0, ('PROPN', 'Salama'): 1.0, ('PROPN', 'Mabruk'): 2.0, ('NOUN', 'disk'): 1.0, ('VERB', 'confiscated'): 1.0, ('PROPN', 'Azerbaijan'): 1.0, ('NOUN', 'risk'): 1.0, ('VERB', 'underestimate'): 1.0, ('NOUN', 'research'): 3.0, ('NOUN', 'stages'): 1.0, ('NUM', '9/11'): 7.0, ('ADJ', '5th'): 1.0, ('PROPN', 'Capitol'): 1.0, ('NOUN', 'Others'): 1.0, ('NOUN', 'airliners'): 1.0, ('VERB', 'directed'): 2.0, ('ADJ', 'West'): 1.0, ('NOUN', 'Coast'): 1.0, ('NUM', '31'): 1.0, ('VERB', 'indicating'): 3.0, ('NOUN', 'crop'): 5.0, ('NOUN', 'dusting'): 3.0, ('VERB', 'last'): 1.0, ('PROPN', 'Ramzi'): 2.0, ('PROPN', 'Binalshibh'): 1.0, ('NUM', '14,000'): 1.0, ('NOUN', 'conspirator'): 1.0, ('NOUN', 'mission'): 2.0, ('NOUN', 'dusters'): 2.0, ('NOUN', 'biochemist'): 2.0, ('ADJ', 'Malaysian'): 3.0, ('PROPN', 'Yazid'): 6.0, ('PROPN', 'Sufaat'): 14.0, ('ADJ', 'sharpest'): 1.0, ('NOUN', 'shed'): 1.0, ('NOUN', 'superiors'): 1.0, ('ADJ', 'unreliable'): 1.0, ('NOUN', 'filing'): 1.0, ('PROPN', 'Jew'): 1.0, ('NOUN', 'sympathizer'): 1.0, ('PROPN', 'Hambali'): 5.0, ('VERB', 'supervised'): 1.0, ('PROPN', 'Hindi'): 1.0, ('PROPN', 'Britani'): 1.0, ('VERB', 'case'): 1.0, ('PROPN', 'NYC'): 2.0, ('VERB', 'studied'): 1.0, ('ADJ', 'virulent'): 1.0, ('PROPN', 'Ames'): 8.0, ('NOUN', 'guide'): 1.0, ('PROPN', 'Dr.'): 9.0, ('NOUN', 'quest'): 1.0, ('VERB', 'weaponize'): 3.0, ('NOUN', 'confidante'): 2.0, ('NOUN', 'prison'): 1.0, ('NOUN', 'Emails'): 1.0, ('PROPN', 'Atef'): 2.0, ('PROPN', 'USAMRIID'): 1.0, ('PROPN', 'koran'): 1.0, ('VERB', 'instructed'): 1.0, ('NOUN', 'jihadist'): 1.0, ('NOUN', 'crusader'): 1.0, ('NOUN', 'enemies'): 1.0, ('PROPN', 'OBL'): 2.0, ('PROPN', 'Wall'): 1.0, ('PROPN', 'Street'): 1.0, ('PROPN', 'Journal'): 1.0, ('VERB', 'contains'): 1.0, ('NOUN', 'memo'): 2.0, ('VERB', 'seek'): 1.0, ('NOUN', 'talent'): 1.0, ('ADJ', 'beneficial'): 1.0, ('ADJ', 'easy'): 8.0, ('NOUN', 'specialists'): 2.0, ('ADV', 'greatly'): 1.0, ('NOUN', 'stage'): 1.0, ('PROPN', 'Christmas'): 1.0, ('NOUN', 'audiotape'): 2.0, ('ADJ', 'orange'): 1.0, ('NOUN', 'alert'): 1.0, ('ADV', 'since'): 1.0, ('ADJ', 'yellow'): 1.0, ('NOUN', 'brigades'): 2.0, ('NOUN', 'banner'): 1.0, ('NOUN', 'paradise'): 2.0, ('ADV', 'somewhere'): 1.0, ('VERB', 'spotted'): 3.0, ('ADV', 'Wherever'): 1.0, ('ADJ', 'traceable'): 1.0, ('NOUN', 'profilers'): 1.0, ('NOUN', 'draft'): 8.0, ('PROPN', 'Sheik'): 2.0, ('PROPN', 'Bojinka'): 3.0, ('VERB', 'threatened'): 2.0, ('VERB', 'dates'): 1.0, ('NOUN', 'Use'): 1.0, ('NOUN', 'detentions'): 1.0, ('NOUN', 'planning'): 2.0, ('PROPN', 'Patrick'): 3.0, ('ADJ', 'Retired'): 1.0, ('PROPN', 'Assistant'): 1.0, ('PROPN', 'Secretary'): 1.0, ('PROPN', 'Analysis'): 1.0, ('PROPN', 'Homeland'): 1.0, ('VERB', 'testified'): 2.0, ('NOUN', 'interrogations'): 1.0, ('ADJ', 'nonconventional'): 1.0, ('ADV', 'notably'): 1.0, ('PROPN', 'WTC'): 5.0, ('NOUN', 'briefing'): 1.0, ('NOUN', 'steps'): 1.0, ('ADJ', 'raw'): 1.0, ('NOUN', 'seed'): 1.0, ('NOUN', 'product'): 2.0, ('ADJ', 'unweaponized'): 1.0, ('PROPN', 'Moro'): 3.0, ('PROPN', 'MILF'): 1.0, ('ADJ', 'half'): 2.0, ('VERB', 'obtaining'): 2.0, ('VERB', 'intended'): 2.0, ('NOUN', 'shura'): 1.0, ('VERB', 'sworn'): 1.0, ('ADJ', 'lengthy'): 1.0, ('NOUN', 'confession'): 1.0, ('NOUN', 'attorney'): 2.0, ('VERB', 'resort'): 1.0, ('VERB', 'possessed'): 1.0, ('NOUN', 'extradition'): 3.0, ('VERB', 'faced'): 1.0, ('NOUN', 'associate'): 2.0, ('NOUN', 'recipe'): 1.0, ('ADJ', 'unprocessed'): 1.0, ('NOUN', 'weight'): 1.0, ('NOUN', 'accounts'): 1.0, ('NOUN', 'distribution'): 1.0, ('NOUN', 'transfers'): 1.0, ('ADV', 'Significantly'): 1.0, ('ADJ', 'individual'): 2.0, ('ADJ', 'quarter'): 1.0, ('NOUN', 'century'): 1.0, ('ADJ', 'retired'): 1.0, ('SCONJ', 'upon'): 1.0, ('VERB', 'contacted'): 2.0, ('ADV', 'necessarily'): 2.0, ('NOUN', 'copy'): 2.0, ('PROPN', 'Ft.'): 1.0, ('PROPN', 'Detrick'): 1.0, ('PROPN', 'Senator'): 2.0, ('PROPN', 'Leahy'): 7.0, ('ADJ', 'Congressional'): 1.0, ('NOUN', 'hearing'): 1.0, ('VERB', 'collected'): 2.0, ('PROPN', 'MSNBC'): 1.0, ('VERB', 'relying'): 1.0, ('NOUN', 'spokesperson'): 1.0, ('VERB', 'narrowed'): 1.0, ('NOUN', 'pool'): 1.0, ('NOUN', 'labs'): 1.0, ('NOUN', 'match'): 1.0, ('ADV', 'overseas'): 1.0, ('VERB', 'evidence'): 1.0, ('NOUN', 'techniques'): 1.0, ('ADJ', 'efficient'): 2.0, ('NOUN', 'bioweaponeer'): 2.0, ('PROPN', 'Kenneth'): 2.0, ('PROPN', 'Alibek'): 3.0, ('ADJ', 'optimal'): 1.0, ('NOUN', 'method'): 1.0, ('ADJ', 'electrostatic'): 2.0, ('NOUN', 'dominance'): 1.0, ('NOUN', 'spores'): 1.0, ('NUM', 'trillion'): 4.0, ('NOUN', 'spore'): 4.0, ('NOUN', 'concentration'): 3.0, ('NOUN', 'clumps'): 1.0, ('NUM', '40'): 1.0, ('NOUN', 'microns'): 2.0, ('NOUN', 'Spores'): 1.0, ('ADJ', 'inhalable'): 1.0, ('VERB', 'point'): 2.0, ('ADJ', 'extraordinary'): 1.0, ('ADJ', 'simpler'): 1.0, ('VERB', 'achieve'): 2.0, ('NOUN', 'grams'): 1.0, ('ADJ', 'industrial'): 1.0, ('NOUN', 'processing'): 1.0, ('VERB', 'sponsored'): 2.0, ('NOUN', 'lab'): 3.0, ('ADJ', 'mistaken'): 2.0, ('NOUN', 'theories'): 1.0, ('NOUN', 'sponsorship'): 1.0, ('VERB', 'indicated'): 1.0, ('PROPN', 'Dugway'): 1.0, ('VERB', 'undermines'): 1.0, ('NOUN', 'liberals'): 2.0, ('ADJ', 'focused'): 1.0, ('PROPN', 'Steve'): 1.0, ('PROPN', 'Hatfill'): 8.0, ('VERB', 'object'): 1.0, ('NOUN', 'biodefense'): 2.0, ('VERB', 'view'): 2.0, ('ADJ', 'useful'): 3.0, ('PROPN', 'USDA'): 1.0, ('NOUN', 'employee'): 1.0, ('PROPN', 'Johnelle'): 1.0, ('PROPN', 'Bryant'): 1.0, ('ADJ', 'sensational'): 1.0, ('VERB', 'purchasing'): 2.0, ('VERB', 'retrofitting'): 1.0, ('NOUN', 'cropduster'): 1.0, ('NOUN', 'interrogators'): 1.0, ('VERB', 'related'): 1.0, ('NOUN', 'manufacturing'): 1.0, ('NOUN', 'details'): 1.0, ('ADJ', 'aerial'): 1.0, ('NOUN', 'dispersal'): 1.0, ('ADJ', 'consistent'): 1.0, ('NOUN', 'opportunity'): 4.0, ('VERB', 'met'): 2.0, ('NOUN', 'plotters'): 1.0, ('NOUN', 'hijackers'): 1.0, ('PROPN', 'Jemaah'): 1.0, ('PROPN', 'Islamiah'): 1.0, ('PROPN', 'JI'): 2.0, ('NOUN', 'company'): 6.0, ('PROPN', 'Green'): 2.0, ('PROPN', 'Laboratory'): 2.0, ('PROPN', 'Medicine'): 2.0, ('NOUN', 'items'): 3.0, ('NOUN', 'Green'): 2.0, ('VERB', 'symbolizes'): 2.0, ('ADJ', 'holy'): 1.0, ('NOUN', 'manual'): 2.0, ('VERB', 'stayed'): 2.0, ('NOUN', 'condominium'): 1.0, ('NOUN', 'lessons'): 1.0, ('NOUN', 'marketing'): 1.0, ('NOUN', 'representative'): 1.0, ('PROPN', 'Infocus'): 1.0, ('PROPN', 'Technologies'): 1.0, ('NUM', '35,000'): 1.0, ('VERB', 'traveled'): 2.0, ('PROPN', 'Brigade'): 1.0, ('VERB', 'realizing'): 1.0, ('VERB', 'perceiving'): 1.0, ('ADJ', 'Filipino'): 1.0, ('NOUN', 'fellow'): 1.0, ('VERB', 'bragging'): 1.0, ('VERB', 'manipulate'): 1.0, ('VERB', 'thwarted'): 1.0, ('ADJ', 'logistical'): 1.0, ('ADJ', 'Various'): 1.0, ('NOUN', 'doctors'): 2.0, ('PROPN', 'Qadoos'): 4.0, ('NOUN', 'bacteriologist'): 2.0, ('PROPN', 'Aafia'): 10.0, ('PROPN', 'Siddiqui'): 5.0, ('NOUN', 'PhD'): 2.0, ('PROPN', 'Karachi'): 4.0, ('NOUN', 'Microbiologist'): 1.0, ('VERB', 'harboring'): 1.0, ('NOUN', 'fugitives'): 1.0, ('ADJ', 'cardiac'): 1.0, ('VERB', 'granted'): 1.0, ('ADJ', 'pre-arrest'): 1.0, ('NOUN', 'bail'): 1.0, ('CONJ', 'Yet'): 1.0, ('NOUN', 'stipend'): 1.0, ('ADV', 'officially'): 1.0, ('ADJ', 'low'): 2.0, ('NOUN', 'IQ'): 1.0, ('PROPN', 'Brandeis'): 1.0, ('NOUN', 'neurology'): 1.0, ('VERB', 'nabbed'): 1.0, ('ADV', 'Understandably'): 1.0, ('PROPN', 'Amerithrax'): 2.0, ('ADJ', 'confidential'): 1.0, ('ADV', 'rarely'): 1.0, ('VERB', 'grant'): 1.0, ('NOUN', 'manhunt'): 1.0, ('PROPN', 'Van'): 1.0, ('PROPN', 'Harp'): 1.0, ('ADJ', 'classified'): 1.0, ('VERB', 'releasing'): 1.0, ('VERB', 'separate'): 1.0, ('PROPN', 'Attorney'): 2.0, ('PROPN', 'Ashcroft'): 2.0, ('PROPN', 'Mueller'): 1.0, ('PRON', 'Her'): 1.0, ('PROPN', 'Ismat'): 3.0, ('ADV', 'last'): 1.0, ('NOUN', 'grandchildren'): 1.0, ('NOUN', 'minicab'): 1.0, ('DET', 'and'): 1.0, ('NOUN', 'uncle'): 2.0, ('VERB', 'detained'): 2.0, ('PROPN', 'airport'): 1.0, ('PROPN', 'Chemical'): 2.0, ('PROPN', 'Wire'): 2.0, ('PROPN', 'Group'): 11.0, ('NOUN', 'lawyer'): 1.0, ('VERB', 'advises'): 2.0, ('NOUN', 'airfare'): 1.0, ('NOUN', 'tickets'): 1.0, ('VERB', 'heard'): 1.0, ('ADJ', 'lost'): 1.0, ('NOUN', 'translation'): 1.0, ('NOUN', 'phrase'): 1.0, ('PROPN', 'english'): 1.0, ('NOUN', 'chemicals'): 1.0, ('VERB', 'appear'): 1.0, ('NOUN', 'pursuit'): 1.0, ('PROPN', 'Floridian'): 1.0, ('PROPN', 'Adnan'): 2.0, ('VERB', 'assisting'): 1.0, ('NOUN', 'nickname'): 1.0, ('PROPN', 'DOJ'): 1.0, ('ADJ', 'commercial'): 1.0, ('NOUN', 'Summer'): 1.0, ('PROPN', 'Hamilton'): 1.0, ('PROPN', 'Maati'): 1.0, ('ADJ', 'faulty'): 1.0, ('NOUN', 'analysis'): 1.0, ('NOUN', 'preconceptions'): 1.0, ('NOUN', 'testimony'): 1.0, ('NOUN', 'millennium'): 1.0, ('PROPN', 'Ressam'): 1.0, ('NOUN', 'hijacker'): 1.0, ('PROPN', 'Alhaznawi'): 1.0, ('ADJ', 'cutaneous'): 1.0, ('VERB', 'bumping'): 1.0, ('NOUN', 'suitcase'): 1.0, ('VERB', 'relating'): 2.0, ('PROPN', 'Prague'): 1.0, ('NOUN', 'conclusion'): 3.0, ('VERB', 'obtained'): 2.0, ('PROPN', 'Ken'): 4.0, ('VERB', 'assisted'): 1.0, ('ADJ', 'mobile'): 2.0, ('NOUN', 'biolab'): 1.0, ('VERB', 'foisted'): 1.0, ('PROPN', 'Curveball'): 1.0, ('VERB', 'divined'): 1.0, ('NOUN', 'design'): 1.0, ('NOUN', 'entourage'): 2.0, ('NOUN', 'birthday'): 3.0, ('PROPN', 'Mukhabarat'): 1.0, ('VERB', 'rejected'): 1.0, ('NOUN', 'suggestion'): 2.0, ('VERB', 'preferring'): 1.0, ('VERB', 'pursue'): 1.0, ('NOUN', 'concept'): 2.0, ('NOUN', 'scientists'): 1.0, ('NOUN', 'code'): 4.0, ('PROPN', 'Charlie'): 1.0, ('PROPN', 'Alpha'): 1.0, ('PROPN', 'Kay'): 4.0, ('NOUN', 'survey'): 1.0, ('NOUN', 'hunt'): 1.0, ('PROPN', 'WMD'): 1.0, ('NOUN', 'innovations'): 1.0, ('NOUN', 'milling'): 1.0, ('NOUN', 'drying'): 1.0, ('ADJ', 'confused'): 1.0, ('NOUN', 'motive'): 2.0, ('PROPN', 'Senators'): 2.0, ('PROPN', 'Daschle'): 3.0, ('VERB', 'tending'): 1.0, ('ADV', 'simplistically'): 1.0, ('VERB', 'receive'): 2.0, ('NOUN', 'appropriations'): 5.0, ('ADV', 'pursuant'): 2.0, ('VERB', 'prevented'): 1.0, ('VERB', 'achieving'): 1.0, ('VERB', 'interferes'): 1.0, ('VERB', 'expressed'): 1.0, ('NOUN', 'sentiment'): 1.0, ('NOUN', 'exchange'): 1.0, ('NOUN', 'plotter'): 2.0, ('PROPN', 'Yousef'): 2.0, ('NOUN', 'Chairman'): 1.0, ('PROPN', 'Judiciary'): 1.0, ('VERB', 'overseeing'): 1.0, ('PROPN', 'Appropriations'): 1.0, ('PROPN', 'Subcommittee'): 1.0, ('NOUN', 'blanket'): 1.0, ('NOUN', 'waiver'): 1.0, ('VERB', 'lift'): 1.0, ('VERB', 'pursuing'): 1.0, ('NOUN', 'imprisonment'): 2.0, ('PROPN', 'Mubarak'): 1.0, ('ADJ', 'foremost'): 1.0, ('NOUN', 'height'): 1.0, ('NOUN', 'sentence'): 1.0, ('ADJ', 'Albanian'): 1.0, ('NOUN', 'returnees'): 1.0, ('NOUN', 'retrial'): 1.0, ('NOUN', 'debt'): 1.0, ('NOUN', 'loan'): 1.0, ('NOUN', 'guarantees'): 1.0, ('DET', 'Those'): 1.0, ('VERB', 'pale'): 1.0, ('PROPN', 'Knights'): 1.0, ('PROPN', 'Banner'): 1.0, ('ADJ', 'secular'): 1.0, ('ADV', 'somehow'): 1.0, ('NOUN', 'creation'): 2.0, ('PROPN', 'Anwar'): 2.0, ('PROPN', 'Sadat'): 2.0, ('PROPN', 'Camp'): 2.0, ('NOUN', 'treaty'): 1.0, ('VERB', 'underscore'): 1.0, ('ADJ', 'liberal'): 2.0, ('NOUN', 'democrats'): 2.0, ('SCONJ', 'beyond'): 1.0, ('NOUN', 'profile'): 1.0, ('VERB', 'overlook'): 1.0, ('VERB', 'uses'): 1.0, ('NOUN', 'domestic'): 1.0, ('ADJ', 'educated'): 1.0, ('PROPN', 'AMI'): 3.0, ('NOUN', 'publisher'): 1.0, ('PROPN', 'Enquirer'): 2.0, ('ADJ', 'goofy'): 1.0, ('PROPN', 'Jennifer'): 1.0, ('PROPN', 'Lopez'): 1.0, ('VERB', 'enclosing'): 1.0, ('VERB', 'proposing'): 1.0, ('PROPN', 'Disease'): 1.0, ('PROPN', 'Control'): 1.0, ('NOUN', 'employees'): 4.0, ('PROPN', 'Leonard'): 1.0, ('VERB', 'containing'): 2.0, ('PROPN', 'Sun'): 4.0, ('NOUN', 'assistant'): 1.0, ('PROPN', 'Bobby'): 1.0, ('PROPN', 'Bender'): 1.0, ('VERB', 'recalls'): 1.0, ('NOUN', 'modus'): 1.0, ('NOUN', 'operandi'): 1.0, ('NOUN', 'signature'): 1.0, ('NOUN', 'offices'): 1.0, ('PROPN', 'Gamaa'): 1.0, ('PROPN', 'Islamiya'): 1.0, ('NOUN', 'treatment'): 1.0, ('VERB', 'imprisoned'): 4.0, ('NOUN', 'casualty'): 1.0, ('ADJ', 'outstanding'): 1.0, ('NOUN', 'reward'): 2.0, ('NOUN', 'rewards'): 1.0, ('NOUN', 'explanation'): 2.0, ('NOUN', 'ten'): 1.0, ('VERB', 'mailed'): 1.0, ('NOUN', 'dates'): 1.0, ('VERB', 'Sound'): 1.0, ('PROPN', 'Leavenworth'): 1.0, ('NOUN', 'defendant'): 1.0, ('NOUN', 'Parole'): 1.0, ('NOUN', 'Officer'): 1.0, ('NOUN', 'mid-February'): 1.0, ('NOUN', 'home'): 1.0, ('ADP', 'Along'): 1.0, ('ADJ', 'considerable'): 2.0, ('NOUN', 'jail'): 1.0, ('NOUN', 'tension'): 1.0, ('ADJ', 'controversial'): 1.0, ('PROPN', 'Feith'): 1.0, ('VERB', 'summarized'): 1.0, ('VERB', 'purported'): 1.0, ('NOUN', 'assistance'): 1.0, ('PROPN', 'Hayat'): 1.0, ('VERB', 'likes'): 1.0, ('ADJ', 'deadly'): 1.0, ('NOUN', 'missive'): 1.0, ('PROPN', 'Brian'): 1.0, ('PROPN', 'Jenkins'): 1.0, ('NOUN', 'sender'): 2.0, ('VERB', 'purporting'): 1.0, ('ADJ', 'islamist'): 1.0, ('NOUN', 'cyanide'): 1.0, ('PROPN', 'Zealand'): 1.0, ('NOUN', 'ingredients'): 1.0, ('NOUN', 'nerve'): 1.0, ('NOUN', 'chapter'): 1.0, ('PROPN', 'Poisonous'): 1.0, ('PROPN', 'Letter'): 1.0, ('ADJ', 'Federal'): 1.0, ('NOUN', 'Eagle'): 1.0, ('NOUN', 'stamp'): 1.0, ('NOUN', 'blue'): 1.0, ('NOUN', 'green'): 1.0, ('NOUN', 'hearts'): 2.0, ('ADJ', 'green'): 4.0, ('NOUN', 'birds'): 2.0, ('ADJ', 'very'): 1.0, ('NOUN', 'codes'): 1.0, ('NOUN', 'masterminds'): 1.0, ('PROPN', 'Jenny'): 1.0, ('VERB', 'representing'): 1.0, ('NOUN', 'symbolism'): 2.0, ('ADJ', 'Green'): 2.0, ('NOUN', 'Birds'): 2.0, ('NOUN', 'video'): 1.0, ('PROPN', '19'): 1.0, ('PROPN', 'Martyrs'): 1.0, ('NOUN', 'FAQ'): 1.0, ('PROPN', 'Azzam'): 1.0, ('PROPN', 'Publications'): 1.0, ('NOUN', 'Hearts'): 1.0, ('VERB', 'refers'): 2.0, ('NOUN', 'mailer'): 1.0, ('PROPN', 'Greendale'): 3.0, ('NOUN', 'Senators'): 1.0, ('ADJ', 'revealing'): 1.0, ('NOUN', 'correspondence'): 1.0, ('NOUN', 'color'): 1.0, ('NOUN', 'perp'): 1.0, ('ADJ', 'cute'): 2.0, ('VERB', 'naming'): 1.0, ('NOUN', 'Dale'): 1.0, ('NOUN', 'river'): 2.0, ('X', 'i.e.'): 2.0, ('PROPN', 'Cairo'): 1.0, ('VERB', 'announcing'): 1.0, ('NOUN', 'merger'): 2.0, ('PROPN', 'Darunta'): 1.0, ('NOUN', 'complex'): 1.0, ('NOUN', 'recruits'): 1.0, ('VERB', 'washed'): 1.0, ('PROPN', 'Hadith'): 1.0, ('PROPN', 'Messenger'): 1.0, ('PROPN', 'Allah'): 1.0, ('ADV', 'wherever'): 1.0, ('VERB', 'please'): 1.0, ('NOUN', 'Paradise'): 1.0, ('ADV', 'seldom'): 1.0, ('VERB', 'relates'): 1.0, ('NOUN', 'sleepers'): 1.0, ('NOUN', 'spadework'): 1.0, ('NOUN', 'fundraising'): 1.0, ('ADJ', 'charitable'): 1.0, ('NOUN', 'causes'): 1.0, ('NOUN', 'alias'): 1.0, ('NOUN', 'sergeant'): 1.0, ('DET', 'What'): 1.0, ('DET', 'Whatever'): 1.0, ('NOUN', 'persuasion'): 1.0, ('VERB', 'deserve'): 1.0, ('VERB', 'lack'): 1.0, ('ADJ', 'sufficient'): 1.0, ('ADV', 'second'): 1.0, ('NOUN', 'Media'): 1.0, ('NOUN', 'approximation'): 1.0, ('ADV', 'Second'): 1.0, ('NOUN', 'hindsight'): 1.0, ('PROPN', 'Efrem'): 1.0, ('PROPN', 'Zimbalist'): 1.0, ('PROPN', 'Jr.'): 1.0, ('VERB', 'striking'): 1.0, ('ADJ', 'appropriate'): 1.0, ('NOUN', 'balance'): 2.0, ('ADJ', 'due'): 1.0, ('VERB', 'exhausted'): 1.0, ('VERB', 'accusing'): 1.0, ('PROPN', 'Stephen'): 1.0, ('ADJ', 'dubious'): 1.0, ('NOUN', 'suspicion'): 2.0, ('VERB', 'founded'): 1.0, ('NOUN', 'premises'): 1.0, ('ADJ', 'reliable'): 2.0, ('NOUN', 'guilt'): 1.0, ('NOUN', 'fixation'): 1.0, ('VERB', 'rumored'): 1.0, ('VERB', 'stemmed'): 1.0, ('NOUN', 'Senator'): 1.0, ('NOUN', 'careers'): 1.0, ('VERB', 'hung'): 1.0, ('NOUN', 'staff'): 1.0, ('NOUN', 'predisposition'): 1.0, ('NOUN', 'winger'): 1.0, ('NUM', '7'): 1.0, ('NUM', '8'): 3.0, ('ADJ', 'pending'): 1.0, ('NOUN', 'libel'): 1.0, ('ADJ', 'uncertain'): 1.0, ('NOUN', 'suit'): 1.0, ('NOUN', 'columnist'): 1.0, ('PROPN', 'Nicholas'): 1.0, ('PROPN', 'Kristof'): 1.0, ('VERB', 'agreed'): 1.0, ('VERB', 'delay'): 1.0, ('NOUN', 'proceeding'): 1.0, ('VERB', 'frustrated'): 1.0, ('ADJ', 'apparent'): 1.0, ('NOUN', 'compromise'): 1.0, ('VERB', 'permit'): 1.0, ('NOUN', 'discovery'): 1.0, ('VERB', 'proceed'): 1.0, ('VERB', 'file'): 1.0, ('NOUN', 'Answer'): 1.0, ('NOUN', 'Complaint'): 1.0, ('NOUN', 'Theory'): 1.0, ('ADV', 'ironically'): 1.0, ('ADJ', 'coincidental'): 1.0, ('VERB', 'adding'): 1.0, ('ADJ', 'regrettable'): 1.0, ('NOUN', 'leaks'): 1.0, ('VERB', 'leak'): 1.0, ('NOUN', 'enthusiasm'): 1.0, ('VERB', 'dropped'): 1.0, ('VERB', 'proves'): 1.0, ('PROPN', 'Berry'): 1.0, ('NOUN', 'residences'): 1.0, ('VERB', 'prove'): 1.0, ('NOUN', 'gasp'): 1.0, ('NOUN', 'insider'): 1.0, ('NOUN', 'searches'): 1.0, ('VERB', 'excluding'): 1.0, ('PROPN', 'Maureen'): 2.0, ('PROPN', 'Andre'): 2.0, ('NOUN', 'curves'): 6.0, ('NOUN', 'Board'): 2.0, ('VERB', 'forward'): 1.0, ('VERB', 'request'): 2.0, ('NOUN', 'FX'): 3.0, ('NOUN', 'inflation'): 6.0, ('NOUN', 'CPI'): 2.0, ('NOUN', 'PPI'): 1.0, ('NOUN', 'valuation'): 1.0, ('VERB', 'determine'): 2.0, ('NOUN', 'index'): 1.0, ('ADV', 'Typically'): 1.0, ('NOUN', 'savings'): 1.0, ('VERB', 'eliminating'): 1.0, ('ADJ', 'nonessential'): 1.0, ('NOUN', 'specifics'): 1.0, ('PROPN', 'Bolivia'): 1.0, ('PROPN', 'Jamaica'): 1.0, ('PROPN', 'Guatemala'): 1.0, ('PROPN', 'Columbia'): 1.0, ('PROPN', 'Puerto'): 1.0, ('PROPN', 'Rico'): 1.0, ('NOUN', 'data'): 4.0, ('PROPN', 'Panama'): 1.0, ('PROPN', 'Brazil'): 1.0, ('PROPN', 'Euro'): 1.0, ('NOUN', 'Underwriting'): 1.0, ('NOUN', 'IV'): 1.0, ('PROPN', 'Cindy'): 2.0, ('VERB', 'revalue'): 1.0, ('PROPN', 'Philip'): 3.0, ('VERB', 'decline'): 1.0, ('NOUN', 'regrets'): 1.0, ('NOUN', 'commitments'): 1.0, ('PROPN', 'Kaminski'): 1.0, ('ADJ', 'Dear'): 1.0, ('VERB', 'inviting'): 1.0, ('NOUN', 'congress'): 2.0, ('PROPN', 'Risk'): 2.0, ('PROPN', 'Australia'): 1.0, ('PROPN', 'Sydney'): 1.0, ('VERB', 'aiming'): 1.0, ('NOUN', 'programme'): 1.0, ('AUX', 'am'): 4.0, ('PROPN', 'Hong'): 1.0, ('PROPN', 'Kong'): 1.0, ('NOUN', 'Tel'): 1.0, ('NUM', '+852'): 1.0, ('NUM', '2545'): 1.0, ('NUM', '2710'): 1.0, ('ADJ', 'Kind'): 1.0, ('PROPN', 'Annesley'): 1.0, ('NOUN', 'Conference'): 1.0, ('NOUN', 'Producer'): 1.0, ('PROPN', 'Waters'): 1.0, ('NUM', '+44'): 2.0, ('NUM', '7484'): 2.0, ('NUM', '9866'): 1.0, ('NUM', '9800'): 1.0, ('X', 'www.risk-conferences.com/risk2001aus'): 1.0, ('PROPN', 'Beth'): 2.0, ('NOUN', 'reservations'): 2.0, ('ADJ', 'Round'): 1.0, ('NOUN', 'Table'): 1.0, ('NUM', '5/18'): 2.0, ('PROPN', 'Sullivan'): 2.0, ('PROPN', 'Steak'): 2.0, ('NUM', '6:30'): 2.0, ('ADV', 'ling'): 2.0, ('ADJ', 'overdue'): 2.0, ('PROPN', 'Paula'): 3.0, ('AUX', 'Will'): 1.0, ('VERB', 'drive'): 1.0, ('PROPN', 'Round'): 1.0, ('PROPN', 'Table'): 1.0, ('ADV', 'FYI'): 2.0, ('PROPN', 'Anne'): 3.0, ('PROPN', 'Sarah'): 2.0, ('NOUN', 'voice'): 1.0, ('NOUN', 'mail'): 1.0, ('PROPN', 'Roberts'): 1.0, ('VERB', 'hire'): 1.0, ('PROPN', 'Sara'): 4.0, ('PROPN', 'Woody'): 1.0, ('NOUN', 'MBA'): 1.0, ('NOUN', 'grad'): 1.0, ('PROPN', 'Rice'): 1.0, ('VERB', 'compared'): 1.0, ('NOUN', 'title'): 2.0, ('ADJ', 'admin'): 1.0, ('NOUN', 'coordinator'): 1.0, ('ADV', 'honestly'): 1.0, ('ADJ', 'Admin'): 1.0, ('NOUN', \"coordinator's\"): 1.0, ('ADJ', 'administrative'): 1.0, ('VERB', 'compare'): 1.0, ('NOUN', 'duties'): 1.0, ('ADJ', 'Sr.'): 1.0, ('NOUN', 'Spec.'): 1.0, ('PROPN', 'Parkhill'): 1.0, ('PROPN', 'Sevil'): 1.0, ('NOUN', 'scope'): 1.0, ('NOUN', 'responsibilities'): 1.0, ('ADJ', 'broad'): 1.0, ('VERB', 'place'): 1.0, ('NOUN', 'specialist'): 2.0, ('NOUN', 'salary'): 1.0, ('NOUN', 'range'): 3.0, ('NUM', '33'): 1.0, ('NUM', '66'): 2.0, ('NUM', 'K'): 2.0, ('VERB', 'advise'): 1.0, ('PROPN', 'Clayton'): 2.0, ('VERB', 'appreciate'): 1.0, ('ADJ', 'Good'): 1.0, ('PROPN', 'Martin'): 1.0, ('NOUN', 'box'): 2.0, ('ADV', 'wonderfully'): 1.0, ('NOUN', 'keys'): 1.0, ('ADJ', 'nice'): 2.0, ('PROPN', 'Enron'): 9.0, ('VERB', 'save'): 1.0, ('NOUN', 'refund'): 1.0, ('NOUN', 'server'): 1.0, ('PROPN', 'Iris'): 2.0, ('NOUN', 'Congratulations'): 1.0, ('NOUN', 'file'): 2.0, ('NOUN', 'paperwork'): 2.0, ('ADJ', 'unique'): 1.0, ('NOUN', 'opportunities'): 2.0, ('NOUN', 'advisors'): 1.0, ('VERB', 'consisted'): 1.0, ('ADJ', 'mutual'): 1.0, ('NOUN', 'fund'): 1.0, ('VERB', 'invest'): 1.0, ('PROPN', 'Home'): 1.0, ('PROPN', 'Depot'): 1.0, ('PROPN', 'Coke'): 1.0, ('ADJ', 'biased'): 1.0, ('ADJ', 'valuable'): 1.0, ('NOUN', 'equity'): 1.0, ('NOUN', 'hedge'): 1.0, ('NOUN', 'funds'): 1.0, ('NOUN', 'growth'): 1.0, ('NOUN', 'correlation'): 1.0, ('NOUN', 'S@P'): 1.0, ('NOUN', 'diversification'): 1.0, ('NOUN', 'dinner'): 3.0, ('NOUN', 'markets'): 2.0, ('VERB', 'close'): 1.0, ('PROPN', 'Gapinski'): 2.0, ('NOUN', 'Account'): 2.0, ('NOUN', 'Vice'): 2.0, ('NOUN', 'President'): 2.0, ('PROPN', 'Emery'): 2.0, ('PROPN', 'Financial'): 2.0, ('PROPN', 'PaineWebber'): 14.0, ('PROPN', 'Inc.'): 3.0, ('NUM', '713-654-0365'): 2.0, ('NUM', '800-553-3119'): 2.0, ('NOUN', 'x365'): 2.0, ('NOUN', 'Fax'): 2.0, ('NUM', '713-654-1281'): 2.0, ('NOUN', 'Cell'): 4.0, ('NUM', '281-435-0295'): 2.0, ('VERB', 'Appreciate'): 1.0, ('NOUN', 'long'): 1.0, ('PROPN', 'ENE'): 4.0, ('ADJ', 'bound'): 1.0, ('VERB', 'forgo'): 1.0, ('SYM', '%'): 2.0, ('NOUN', 'premium'): 1.0, ('VERB', 'lighten'): 1.0, ('NOUN', 'stategy'): 1.0, ('VERB', 'approved'): 1.0, ('VERB', 'sell'): 2.0, ('VERB', 'hedge'): 1.0, ('ADJ', 'unexercised'): 1.0, ('ADJ', 'vested'): 1.0, ('NOUN', 'pullback'): 1.0, ('ADP', 'back'): 1.0, ('NOUN', 'stock'): 1.0, ('NOUN', 'bounce'): 1.0, ('ADP', '@'): 2.0, ('NUM', '75'): 2.0, ('NUM', '73'): 1.0, ('VERB', 'Call'): 1.0, ('NOUN', 'Notice'): 5.0, ('VERB', 'Regarding'): 6.0, ('NOUN', 'Entry'): 3.0, ('VERB', 'transmit'): 3.0, ('NOUN', 'orders'): 6.0, ('NOUN', 'instructions'): 9.0, ('X', 's'): 3.0, ('VERB', 'transmitted'): 3.0, ('NOUN', 'Privacy'): 3.0, ('NOUN', 'Confidentiality'): 3.0, ('VERB', 'reserves'): 3.0, ('VERB', 'review'): 5.0, ('NOUN', 'content'): 3.0, ('NOUN', 'communications'): 3.0, ('PROPN', 'Notice'): 1.0, ('VERB', 'cares'): 1.0, ('PUNCT', '??????'): 1.0, ('VERB', 'note'): 1.0, ('X', 'Sean.Cooper@ElPaso.com'): 1.0, ('VERB', 'rescheduled'): 1.0, ('PRON', 'Your'): 3.0, ('NOUN', 'buddy'): 1.0, ('PROPN', 'Beau'): 1.0, ('VERB', 'invited'): 2.0, ('PROPN', 'thursday'): 2.0, ('PROPN', 'NYMEX'): 2.0, ('NOUN', 'cocktail'): 2.0, ('NOUN', 'hour'): 2.0, ('INTJ', 'oh'): 2.0, ('NOUN', 'agenda'): 2.0, ('PROPN', \"Tuesday's\"): 2.0, ('NUM', '3:30'): 2.0, ('PROPN', 'Houston'): 2.0, ('ADV', 'completely'): 2.0, ('NOUN', 'update'): 2.0, ('VERB', 'move'): 1.0, ('ADJ', 'weekly'): 1.0, ('PROPN', \"Monday's\"): 1.0, ('PUNCT', '???????????????'): 1.0, ('NOUN', 'employment'): 2.0, ('VERB', 'respond'): 1.0, ('ADV', 'below'): 7.0, ('PROPN', 'Lavorato'): 1.0, ('VERB', 'attached'): 3.0, ('NOUN', 'allocation'): 1.0, ('ADP', 'amongst'): 1.0, ('NOUN', 'categories'): 1.0, ('PROPN', 'Real'): 1.0, ('PROPN', 'Time'): 1.0, ('PROPN', 'Traders'): 1.0, ('NOUN', 'presentation'): 1.0, ('NOUN', 'tomorrow'): 1.0, ('NOUN', 'x'): 1.0, ('NOUN', 'comparisons'): 1.0, ('VERB', 'lose'): 1.0, ('NUM', '65'): 2.0, ('NUM', 'k'): 5.0, ('VERB', 'hiring'): 1.0, ('NOUN', 'k'): 3.0, ('NOUN', 'math'): 1.0, ('NOUN', 'stretch'): 1.0, ('ADJ', 'hourly'): 1.0, ('ADV', 'ASAP'): 1.0, ('VERB', 'reviewing'): 2.0, ('NOUN', 'renewal'): 1.0, ('PROPN', 'Mark'): 2.0, ('PROPN', 'Laurent'): 1.0, ('VERB', 'communicated'): 1.0, ('VERB', 'ask'): 1.0, ('ADJ', 'Current'): 1.0, ('NOUN', 'Salary'): 3.0, ('NUM', '47,500'): 2.0, ('NOUN', 'Job'): 1.0, ('NOUN', 'Group'): 1.0, ('NOUN', 'Specialist'): 1.0, ('NOUN', 'YE'): 1.0, ('NOUN', 'PRC'): 1.0, ('NOUN', 'Rating'): 1.0, ('ADJ', 'Satisfactory'): 1.0, ('NOUN', 'Original'): 1.0, ('NOUN', 'Proposition'): 1.0, ('NOUN', 'Base'): 2.0, ('NOUN', 'Year'): 2.0, ('NOUN', 'Agreement'): 7.0, ('VERB', 'Revised'): 1.0, ('NOUN', 'Proposal'): 1.0, ('NUM', '55'): 2.0, ('NOUN', 'Business'): 1.0, ('NOUN', 'Case'): 1.0, ('ADJ', 'Dean'): 1.0, ('ADV', 'approximately'): 1.0, ('NOUN', 'consideration'): 1.0, ('VERB', 'rated'): 1.0, ('ADJ', 'satisfactory'): 1.0, ('ADJ', 'specialist'): 1.0, ('ADJ', 'salary'): 1.0, ('NOUN', 'room'): 2.0, ('VERB', 'progress'): 1.0, ('ADV', 'Secondly'): 1.0, ('X', 'out'): 1.0, ('VERB', 'price'): 1.0, ('NOUN', 'performers'): 2.0, ('ADV', 'respectively'): 1.0, ('X', 'eg.'): 1.0, ('PROPN', 'Thomas'): 2.0, ('PROPN', 'Jason'): 2.0, ('PROPN', 'Choate'): 2.0, ('PROPN', 'Todd'): 3.0, ('PROPN', 'DeCook'): 2.0, ('PROPN', 'Makkai'): 2.0, ('PROPN', 'Salary'): 1.0, ('PROPN', 'Listing'): 1.0, ('PROPN', 'Job'): 1.0, ('PROPN', 'Maria'): 1.0, ('PROPN', 'Valdes'): 1.0, ('NUM', '62,500'): 1.0, ('NUM', '55,008'): 1.0, ('NUM', '60,008'): 1.0, ('NUM', '42,008'): 1.0, ('PROPN', 'Oxley'): 1.0, ('NOUN', 'Transmission'): 1.0, ('NOUN', 'Expansion'): 1.0, ('PROPN', 'Systems'): 1.0, ('PROPN', 'Transition'): 1.0, ('PROPN', 'Conference'): 3.0, ('PROPN', 'Feb.'): 1.0, ('PROPN', 'Florida'): 1.0, ('NOUN', 'OVERVIEW'): 1.0, ('ADJ', 'regulatory'): 2.0, ('NOUN', 'challenges'): 2.0, ('ADJ', 'electric'): 1.0, ('NOUN', 'transmission'): 3.0, ('NOUN', 'capacity'): 1.0, ('NOUN', 'demands'): 1.0, ('ADJ', 'competitive'): 1.0, ('PROPN', 'FERC'): 2.0, ('PROPN', 'RTO'): 1.0, ('NOUN', 'initiatives'): 1.0, ('NOUN', 'parameters'): 1.0, ('NOUN', 'solutions'): 1.0, ('NOUN', 'technology'): 1.0, ('NOUN', 'models'): 1.0, ('VERB', 'analyze'): 1.0, ('NOUN', 'proposals'): 1.0, ('NOUN', 'pricing'): 4.0, ('NOUN', 'alternatives'): 1.0, ('ADJ', 'traditional'): 1.0, ('NOUN', 'methods'): 1.0, ('ADJ', 'legislative'): 1.0, ('VERB', 'ensuring'): 1.0, ('NOUN', 'grid'): 1.0, ('NOUN', 'brochure'): 1.0, ('ADJ', 'associated'): 1.0, ('PROPN', 'Workshops'): 1.0, ('VERB', 'clicking'): 1.0, ('NOUN', 'link'): 1.0, ('PUNCT', '<'): 2.0, ('X', 'http://www.euci.com/pdf/trans_expn.pdf'): 2.0, ('PUNCT', '>'): 2.0, ('PUNCT', '<<'): 1.0, ('PUNCT', '>>'): 1.0, ('NOUN', 'Electricity'): 1.0, ('PROPN', 'Market'): 1.0, ('PROPN', 'Design'): 1.0, ('PROPN', 'Georgia'): 1.0, ('NUM', '969'): 2.0, ('PROPN', 'Essie'): 9.0, ('PROPN', 'Sonya'): 1.0, ('NUM', '07/30/2001'): 1.0, ('NUM', '05:17'): 1.0, ('VERB', 'Embedded'): 1.0, ('NOUN', 'Picture'): 1.0, ('NOUN', 'Device'): 1.0, ('ADJ', 'Independent'): 1.0, ('NOUN', 'Bitmap'): 1.0, ('PRON', 'WHO'): 1.0, ('PRON', 'WHAT'): 1.0, ('ADJ', 'Happy'): 1.0, ('NOUN', 'Hour'): 1.0, ('PROPN', 'Suarez'): 1.0, ('ADV', 'WHEN'): 1.0, ('NOUN', 'Today'): 2.0, ('NOUN', 'pm'): 1.0, ('ADV', 'WHERE'): 1.0, ('PROPN', 'Porch'): 1.0, ('NUM', '217'): 1.0, ('PROPN', 'Gray'): 1.0, ('PROPN', 'St.'): 2.0, ('NUM', '713'): 2.0, ('NUM', '571-9571'): 1.0, ('ADV', 'WHY'): 1.0, ('PROPN', 'EBS'): 5.0, ('PART', 'NOT'): 1.0, ('PROPN', 'Leon'): 5.0, ('VERB', 'xferring'): 2.0, ('NOUN', 'Co.'): 2.0, ('NUM', '1691'): 4.0, ('NOUN', 'Regards'): 2.0, ('PROPN', 'Vicsandra'): 2.0, ('NOUN', 'decisions'): 2.0, ('ADJ', 'helpful'): 2.0, ('PROPN', 'Patty'): 4.0, ('ADJ', 'Corporate'): 2.0, ('NOUN', 'Tax'): 3.0, ('NOUN', 'x35172'): 2.0, ('PROPN', 'EB'): 2.0, ('NUM', '1774'): 2.0, ('ADJ', 'red'): 2.0, ('NOUN', 'companies'): 4.0, ('VERB', 'fit'): 2.0, ('NOUN', 'indication'): 2.0, ('NOUN', 'rank'): 3.0, ('NOUN', 'entities'): 1.0, ('VERB', 'assigned'): 8.0, ('NOUN', '18T'): 1.0, ('PROPN', 'EI'): 1.0, ('PROPN', 'Operations'): 1.0, ('PROPN', 'LLC'): 4.0, ('NOUN', 'entity'): 6.0, ('PROPN', 'TIS'): 1.0, ('PROPN', 'SAP'): 1.0, ('PROPN', 'Hyperion'): 1.0, ('NOUN', 'income'): 2.0, ('NOUN', 'liabilities'): 2.0, ('ADJ', 'corporate'): 1.0, ('NOUN', 'sheet'): 1.0, ('ADJ', 'inactive'): 1.0, ('NOUN', '86M'): 1.0, ('PROPN', 'Net'): 1.0, ('PROPN', 'Works'): 1.0, ('PROPN', 'MTM'): 2.0, ('ADP', 'Per'): 1.0, ('NOUN', 'financials'): 1.0, ('NUM', 'M'): 1.0, ('NOUN', 'expenses'): 1.0, ('AUX', 'get'): 1.0, ('NOUN', 'reorg'): 1.0, ('NOUN', '80Y'): 1.0, ('PROPN', 'Broadband'): 1.0, ('PROPN', 'Acquisition'): 1.0, ('PROPN', 'Inc'): 1.0, ('NOUN', 'acquistion'): 1.0, ('PROPN', 'WarpSpeed'): 2.0, ('PROPN', 'Communications'): 1.0, ('NOUN', '83N'): 3.0, ('VERB', 'dissolved'): 1.0, ('NOUN', 'completion'): 1.0, ('NOUN', 'Company'): 2.0, ('NOUN', 'amount'): 1.0, ('NOUN', 'I/S'): 1.0, ('NUM', '1579'): 1.0, ('PROPN', 'Co.'): 1.0, ('NOUN', 'Division'): 1.0, ('NOUN', '17H'): 2.0, ('NOUN', 'Networks'): 2.0, ('NOUN', 'Services'): 3.0, ('NOUN', 'Holding'): 1.0, ('ADJ', 'Administrative'): 1.0, ('NOUN', 'Companies'): 1.0, ('ADJ', 'Same'): 2.0, ('NOUN', 'Network'): 1.0, ('PROPN', 'EPI'): 2.0, ('VERB', 'Set'): 2.0, ('VERB', 'centralize'): 2.0, ('VERB', 'Broke'): 2.0, ('NUM', '1179'): 2.0, ('NOUN', 'Commodity'): 1.0, ('NOUN', 'Trade'): 1.0, ('PROPN', 'Richards'): 1.0, ('PROPN', 'Mary'): 1.0, ('PROPN', 'Fischer'): 1.0, ('NUM', '1307'): 3.0, ('PROPN', 'EBIC'): 1.0, ('PROPN', 'Apache'): 1.0, ('VERB', 'Rolls'): 1.0, ('PROPN', 'Cherokee'): 2.0, ('PROPN', 'Finance'): 2.0, ('PROPN', 'VOF'): 2.0, ('NOUN', 'CFC'): 1.0, ('PROPN', 'Glen'): 2.0, ('PROPN', 'Walloch'): 2.0, ('PROPN', 'Walker'): 2.0, ('NUM', '1689'): 1.0, ('PROPN', 'Ventures'): 1.0, ('PROPN', 'Branom'): 1.0, ('NOUN', 'Analyst'): 3.0, ('NUM', '345-8702'): 1.0, ('X', 'leon.branom@enron.com'): 1.0, ('PROPN', 'Virginia'): 1.0, ('INTJ', 'Hello'): 1.0, ('PROPN', 'Cross'): 1.0, ('PROPN', 'Signac'): 1.0, ('ADJ', 'impressionist'): 3.0, ('NOUN', 'lithograph'): 1.0, ('NOUN', 'gallery'): 2.0, ('PROPN', 'Weston'): 1.0, ('PROPN', 'Lichtenstein'): 1.0, ('VERB', 'hanging'): 1.0, ('ADJ', 'magnificent'): 1.0, ('PROPN', 'Appel'): 1.0, ('ADV', 'Actually'): 3.0, ('VERB', 'own'): 1.0, ('VERB', 'depending'): 1.0, ('VERB', 'listed'): 1.0, ('ADP', 'vs.'): 1.0, ('NOUN', 'favorites'): 1.0, ('VERB', 'purchase'): 1.0, ('VERB', 'numbered'): 1.0, ('NOUN', 'artist'): 1.0, ('NOUN', 'buying'): 1.0, ('ADV', 'outrageously'): 1.0, ('ADJ', 'expensive'): 1.0, ('PROPN', 'Rothko'): 2.0, ('PROPN', 'Kline'): 2.0, ('PROPN', 'Bonnard'): 1.0, ('ADJ', 'colorful'): 1.0, ('PROPN', 'Lautrec'): 1.0, ('PROPN', 'Suerat'): 1.0, ('NOUN', 'lithos'): 1.0, ('PROPN', 'Braque'): 1.0, ('PROPN', 'Arp'): 1.0, ('PROPN', 'Rouault'): 1.0, ('PROPN', 'Modrian'): 1.0, ('PROPN', 'Motherwell'): 1.0, ('VERB', 're-looking'): 1.0, ('PROPN', 'Picasso'): 1.0, ('ADJ', 'cubist'): 1.0, ('NOUN', '1920s'): 1.0, ('NOUN', 'technique'): 1.0, ('NOUN', 'wa'): 1.0, ('NOUN', 'pronunciation'): 1.0, ('ADP', 'Vs'): 1.0, ('ADJ', 'proper'): 1.0, ('NOUN', 'spelling'): 1.0, ('NOUN', 'sale'): 1.0, ('NOUN', 'galleries'): 1.0, ('PROPN', 'SF'): 1.0, ('PROPN', 'Matisse'): 1.0, ('ADJ', 'back'): 1.0, ('NOUN', 'burner'): 1.0, ('PROPN', 'mike'): 1.0, ('PROPN', 'Huskers'): 2.0, ('VERB', 'drool'): 2.0, ('PROPN', 'Sooners'): 3.0, ('X', 'Ken'): 3.0, ('X', 'Rice@ENRON'): 3.0, ('X', 'COMMUNICATIONS'): 3.0, ('NUM', '01/19/2001'): 3.0, ('NUM', '09:14'): 1.0, ('NOUN', 'AM'): 4.0, ('X', 'Mike'): 1.0, ('X', 'McConnell@ECT'): 1.0, ('NUM', '01/19/01'): 1.0, ('NUM', '07:55'): 1.0, ('PROPN', 'm'): 2.0, ('NUM', '07:24'): 2.0, ('VERB', 'postpone'): 2.0, ('NOUN', 'Presentation'): 2.0, ('PROPN', 'sunday'): 2.0, ('VERB', 'cancel'): 2.0, ('NOUN', 'PS'): 2.0, ('NOUN', 'bowl'): 4.0, ('NOUN', 'games'): 2.0, ('X', 'galleryfurniture.com'): 2.0, ('PROPN', 'Shreveport'): 2.0, ('PUNCT', '))'): 2.0, ('VERB', 'realized'): 2.0, ('ADJ', 'sick'): 2.0, ('NOUN', 'college'): 2.0, ('NOUN', 'fan'): 2.0, ('ADJ', 'intial'): 1.0, ('PROPN', 'word'): 1.0, ('NOUN', 'document'): 1.0, ('NOUN', 'changes'): 1.0, ('NOUN', 'additions'): 1.0, ('NOUN', 'wording'): 1.0, ('NOUN', 'comment'): 2.0, ('NOUN', 'calendars'): 1.0, ('ADV', 'Too'): 1.0, ('PROPN', 'Compaq'): 2.0, ('PROPN', 'San'): 3.0, ('PROPN', 'Antonio'): 2.0, ('INTJ', 'wow'): 1.0, ('NOUN', 'let'): 1.0, ('NOUN', 'down'): 1.0, ('NOUN', 'restaurants'): 1.0, ('PROPN', 'Ric'): 1.0, ('PROPN', 'Menger'): 1.0, ('X', 'Kay'): 1.0, ('X', 'Mann@ENRON'): 1.0, ('NUM', '09/20/2000'): 1.0, ('NUM', '04:18'): 1.0, ('PROPN', 'Francis'): 1.0, ('VERB', 'TRY'): 1.0, ('NUM', '445'): 1.0, ('PROPN', 'ENA'): 1.0, ('NOUN', 'orientation'): 1.0, ('NOUN', 'am'): 1.0, ('NUM', '430'): 1.0, ('NUM', '27'): 1.0, ('NOUN', 'vacation'): 2.0, ('PROPN', 'Francisco'): 1.0, ('PROPN', 'Enterprise'): 1.0, ('NOUN', 'spouses'): 1.0, ('VERB', 'Figures'): 1.0, ('ADV', 'fyi'): 1.0, ('NOUN', 'review'): 1.0, ('VERB', 'wrap'): 2.0, ('NOUN', 'LOI'): 5.0, ('VERB', 'licensed'): 2.0, ('NOUN', 'Fuel'): 2.0, ('NOUN', 'Energy'): 2.0, ('NOUN', 'equipment'): 2.0, ('VERB', 'intend'): 2.0, ('VERB', 'reference'): 3.0, ('NOUN', 'manufacturer'): 4.0, ('ADJ', 'generic'): 2.0, ('VERB', 'attach'): 2.0, ('NOUN', 'Development'): 6.0, ('VERB', 'condition'): 3.0, ('VERB', 'agreeing'): 2.0, ('ADV', 'substantially'): 2.0, ('NOUN', 'optionality'): 2.0, ('NOUN', 'bug'): 2.0, ('VERB', 'bite'): 2.0, ('NOUN', 'backside'): 2.0, ('NUM', 'ONE'): 3.0, ('NOUN', 'EPC'): 5.0, ('NOUN', 'contract'): 5.0, ('NOUN', 'confidentiality'): 2.0, ('PROPN', 'Bart'): 2.0, ('ADJ', 'handy'): 2.0, ('VERB', 'speed'): 2.0, ('VERB', 'Forget'): 1.0, ('NOUN', 'contracts'): 3.0, ('NOUN', 'project'): 7.0, ('NUM', '170'): 1.0, ('NUM', 'm'): 1.0, ('VERB', 'paid'): 2.0, ('VERB', 'expecting'): 1.0, ('ADJ', 'attributable'): 1.0, ('PROPN', 'CRRA'): 4.0, ('PROPN', 'FuelCell'): 1.0, ('ADJ', 'meaningful'): 1.0, ('VERB', 'listing'): 1.0, ('PROPN', 'ONSI'): 1.0, ('ADV', 'Hence'): 1.0, ('NOUN', 'attachment'): 1.0, ('VERB', 'mark'): 1.0, ('NOUN', 'assumption'): 1.0, ('VERB', 'Based'): 1.0, ('VERB', 'reacts'): 1.0, ('ADJ', 'ultimate'): 1.0, ('NOUN', 'Performance'): 1.0, ('NOUN', 'tests'): 1.0, ('ADV', 'Remarkably'): 1.0, ('VERB', 'Sent'): 1.0, ('X', 'Ben'): 1.0, ('X', 'Jacoby@ECT'): 1.0, ('PROPN', 'Mom'): 1.0, ('NOUN', 'tommorow'): 1.0, ('PROPN', 'Chris'): 2.0, ('VERB', 'Look'): 1.0, ('NOUN', 'beers'): 1.0, ('DET', '**OFFSET**'): -0.1, ('INTJ', '**OFFSET**'): -0.1, ('CONJ', '**OFFSET**'): -0.1, ('NUM', '**OFFSET**'): -0.1, ('X', '**OFFSET**'): -0.1, ('PRON', '**OFFSET**'): -0.1, ('ADP', '**OFFSET**'): -0.1, ('AUX', '**OFFSET**'): -0.1, ('ADJ', '**OFFSET**'): -0.1, ('NOUN', '**OFFSET**'): 0.1, ('PART', '**OFFSET**'): -0.1, ('SYM', '**OFFSET**'): -0.1, ('PUNCT', '**OFFSET**'): -0.1, ('SCONJ', '**OFFSET**'): -0.1, ('PROPN', '**OFFSET**'): -0.1, ('VERB', '**OFFSET**'): -0.1, ('ADV', '**OFFSET**'): -0.1})\n"
     ]
    }
   ],
   "source": [
    "reload(most_common);\n",
    "\n",
    "# tests\n",
    "! nosetests tests/test_classifier.py:test_mcc_tagger_output\n",
    "! nosetests tests/test_classifier.py:test_mcc_tagger_accuracy\n",
    "\n",
    "\n",
    "theta_mc = most_common.get_most_common_word_weights(TRAIN_FILE)\n",
    "print(theta_mc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger_mc = tagger_base.make_classifier_tagger(theta_mc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['PRON', 'AUX', 'AUX', 'NOUN']\n"
     ]
    }
   ],
   "source": [
    "tags = tagger_mc(['They','can','can','fish'],all_tags)\n",
    "print (tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['DET', 'ADJ', 'NOUN', 'DET', 'PROPN', 'PUNCT']\n"
     ]
    }
   ],
   "source": [
    "tags = tagger_mc(['The','old','man','the','boat','.'],all_tags)\n",
    "print (tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- Now let's run your tagger on the dev data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8111242915513378\n"
     ]
    }
   ],
   "source": [
    "confusion = tagger_base.eval_tagger(tagger_mc,'most-common.preds',all_tags=all_tags)\n",
    "print (scorer.accuracy(confusion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Naive Bayes as a tagger.\n",
    "    \n",
    "- You can use your Naive Bayes classifier form pset1 to set the weights for the classifier tagger. We have added in a helper function `naive_bayes.get_nb_weights(..)`. Make sure to retain this function, when you copy your code.\n",
    "- If you don't think you got it right in pset1, you are free to change it now. If you got it right, then just examine the performance of naive bayes as tagger in the following blocks. There is no test or deliverable for this part.\n",
    "- Note that, for text classification, we had a bag of words feature vector and label for each document. For POS tagging, in order to estimate the weights for the classifier tagger, we will consider each token to be its own document. The following helper code converts the dataset to token level bag-of-words feature vector and labels.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(naive_bayes);\n",
    "nb_weights = naive_bayes.get_nb_weights(TRAIN_FILE, .01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- This gives weights for each tag-word pair that represent $\\log P(word \\mid tag)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# obtaining vocab of words\n",
    "vocab = set([word for tag,word in nb_weights.keys() if word is not constants.OFFSET])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6912\n"
     ]
    }
   ],
   "source": [
    "print (len(vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9999999999999096\n",
      "1.000000000000072\n",
      "0.9999999999998604\n"
     ]
    }
   ],
   "source": [
    "print (sum(np.exp(nb_weights[('ADJ',word)]) for word in vocab))\n",
    "print (sum(np.exp(nb_weights[('NOUN',word)]) for word in vocab))\n",
    "print (sum(np.exp(nb_weights[('PUNCT',word)]) for word in vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- We have zero weights for OOV terms -- think about how this affects the classification here!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n"
     ]
    }
   ],
   "source": [
    "print (nb_weights[('ADJ','baaaaaaaaad')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-2.1314729713701412\n",
      "-3.0977985544826994\n",
      "-2.8092821056787987\n"
     ]
    }
   ],
   "source": [
    "print (nb_weights[('VERB',constants.OFFSET)])\n",
    "print (nb_weights[('ADV',constants.OFFSET)])\n",
    "print (nb_weights[('PRON',constants.OFFSET)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Offsets should correspond to log-probabilities $\\log P(y)$ such that $\\sum_y P(y) = 1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9999999999999998"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(np.exp(nb_weights[(tag,constants.OFFSET)]) for tag in all_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now let us look at the accuracy of our naive_bayes tagger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8104652695400026\n"
     ]
    }
   ],
   "source": [
    "confusion = tagger_base.eval_tagger(tagger_base.make_classifier_tagger(nb_weights),'nb-simple.preds')\n",
    "dev_acc = scorer.accuracy(confusion)\n",
    "print (dev_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Just as good as the heuristic tagger from above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Viterbi Algorithm\n",
    "\n",
    "In this section you will implement the Viterbi algorithm in **PyTorch**. To get warmed up, let's work out an example by hand. For simplicity, there are only two tags, **N**OUN and **V**ERB. Here are the parameters:\n",
    "\n",
    "| | Value |\n",
    "| ------------- |:-------------:|\n",
    "| $\\log P_E(\\cdot|N)$ | they: -1, can: -3, fish: -3 |\n",
    "| $\\log P_E(\\cdot|V)$ | they: -11, can: -2, fish: -4 |\n",
    "| $\\log P_T(\\cdot|N)$ | N: -5, V: -2, END: -2 |\n",
    "| $\\log P_T(\\cdot|V)$ | N: -1, V: -3, END: -3 |\n",
    "| $\\log P_T(\\cdot|\\text{START})$ | N :-1, V :-2 |\n",
    "\n",
    "where $P_E(\\cdot|\\cdot)$ is the emission probability and $P_T(\\cdot|\\cdot)$ is the transition probability.\n",
    " \n",
    "- In class we discussed the sentence *They can fish*. \n",
    "- Now work out a more complicated example: \"*They can can fish*\", where the second \"*can*\" refers to the verb of putting things into cans.\n",
    " \n",
    "** Deliverable 3.1 ** Work out the trellis by hand, and fill in the table in the file **```text-answers.md```** (*0.5 points*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing Viterbi ##\n",
    "\n",
    "Here are some predefined weights, corresponding to the weights from the problem 3.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "START_TAG = constants.START_TAG\n",
    "END_TAG = constants.END_TAG\n",
    "UNK = constants.UNK\n",
    "\n",
    "nb_weights={('NOUN','they'):-1,\\\n",
    "            ('NOUN','can'):-3,\\\n",
    "            ('NOUN','fish'):-3,\\\n",
    "            ('VERB','they'):-11,\\\n",
    "            ('VERB','can'):-2,\\\n",
    "            ('VERB','fish'):-4,}\n",
    "hmm_trans_weights={('NOUN','NOUN'):-5,\\\n",
    "                   ('VERB','NOUN'):-2,\\\n",
    "                   (END_TAG,'NOUN'):-2,\\\n",
    "                   ('NOUN','VERB'):-1,\\\n",
    "                   ('VERB','VERB'):-3,\\\n",
    "                   (END_TAG,'VERB'):-3,\\\n",
    "                   ('NOUN',START_TAG):-1,\\\n",
    "                   ('VERB',START_TAG):-2}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Deliverable 3.2 ** Complete the ```hmm.compute_weights_variables(...)``` function in `hmm.py` file (*0.5 points*) The function should basically convert the weights to respective pytorch variables .\n",
    "\n",
    "- **Inputs** :\n",
    "    - `nb_weights`: emission_weights: dictionary of weights\n",
    "    - `hmm_trans_weights`: dictionary of weights\n",
    "    - `vocab`: list of all the words\n",
    "    - `word_to_ix`: a dictionary that maps each word in the vocab to a unique index. **Does not have the OFFSET_TOKEN.**\n",
    "    - `tag_to_ix`: a dictionary that maps each tag (including the `START_TAG` and the `END_TAG`) to a unique index.  \n",
    "\n",
    "- **Outputs** : returns two torch Variables\n",
    "    - `emission_probs`: torch Variable of a matrix of size `Vocab x Tagset_size`: \n",
    "        such that for a specific weight say `(word1, tag1):value` would result in\n",
    "        `emission_probs[word_to_ix[word1]][tag_to_ix[tag1]]=value`, else a zero. Also, make sure to set weights such that `START_TAG` and `END_TAG` cannot generate any word. **Make sure to ignore the OFFSET weights that might be present in the nb_weights. Consider the words only in your word_to_ix.**\n",
    "        \n",
    "    - `tag_transition_probs`: torch Variable of a matrix of size `Tagset_size x Tagset_size`: \n",
    "        such that for a specific feature say `(tag1, tag2):value` \n",
    "        where tag1 is my succeeding tag, tag2 is my current tag. \n",
    "        This would result in `tag_transition_probs[tag_to_ix[tag1]][tag_to_ix[tag2]]=value`. \n",
    "        Also ensure to set the other weights such that there are no illegal transitions \n",
    "        (like from some tag to START_TAG -or- from END_TAG to some other tag)  \n",
    "\n",
    "- **Tests**: ```test_viterbi.py: test_compute_hmm_weights_variables()```\n",
    "\n",
    "Hint: Use `-np.inf` as weights for illegal transitions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Below, observe that we are calculating `tag_to_ix, ix_to_tag, word_to_ix`. These are useful to access a particular emission score for a particular token and a tag. Look through the variables: tag_transition_probs and emission_probs below and it should be clear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.001s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "reload(hmm);\n",
    "! nosetests tests/test_viterbi.py:test_compute_hmm_weights_variables\n",
    "\n",
    "word_to_ix={'they':0, 'can':1, 'fish':2, UNK:3}\n",
    "tag_to_ix = {START_TAG:0, 'NOUN':1, 'VERB':2, END_TAG:3}\n",
    "ix_to_tag = {v:k for k,v in tag_to_ix.items()}\n",
    "all_tags = [START_TAG, 'NOUN', 'VERB', END_TAG]\n",
    "words = ['they', 'can', 'fish']\n",
    "vocab = ['they','can','fish',UNK]\n",
    "# note that we are also including an UNK token: this will be helpful later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "emission_probs, tag_transition_probs = hmm.compute_weights_variables(nb_weights, hmm_trans_weights, vocab, \n",
    "                                                                     word_to_ix, tag_to_ix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      "-inf -inf -inf -inf\n",
      " -1  -5  -1 -inf\n",
      " -2  -2  -3 -inf\n",
      "-inf  -2  -3 -inf\n",
      "[torch.FloatTensor of size 4x4]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print (tag_transition_probs)\n",
    "# tag_transition_probs[0] corresponds to scores for START_TAG from START_TAG, NOUN, VERB, END_TAG\n",
    "# tag_transition_probs[1] corresponds to scores for NOUN from START_TAG, NOUN, VERB, END_TAG\n",
    "# tag_transition_probs[2] corresponds to scores for VERB from START_TAG, NOUN, VERB, END_TAG\n",
    "# tag_transition_probs[3] corresponds to scores for END_TAG from START_TAG, NOUN, VERB, END_TAG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      "-inf  -1 -11 -inf\n",
      "-inf  -3  -2 -inf\n",
      "-inf  -3  -4 -inf\n",
      "-inf   0   0 -inf\n",
      "[torch.FloatTensor of size 4x4]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print (emission_probs)\n",
    "# emission_probs[0] corresponds to scores for the token 'they' for START_TAG, NOUN, VERB, END_TAG\n",
    "# emission_probs[1] corresponds to scores for the token 'can' for START_TAG, NOUN, VERB, END_TAG\n",
    "# emission_probs[2] corresponds to scores for the token 'fish' for START_TAG, NOUN, VERB, END_TAG\n",
    "# emission_probs[2] corresponds to scores for the token 'UNK' for START_TAG, NOUN, VERB, END_TAG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- we will be using these emission scores as inputs for each token in the input in the following function: ```viterbi_step()```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 3.3** The Viterbi recurrence. (*0.5 point*)\n",
    "\n",
    "Implement `viterbi_step` in `gtnlplib/viterbi.py`. This is the method that will compute the best path score and corresponding back pointer for a particular token in the sentence for all possible tags, which you will later call from the main viterbi routine. You will also be using this later for the `Bi-LSTM_CRF` model.\n",
    "\n",
    "### Inputs\n",
    "- `all_tags`: list of all tags: includes both the `START_TAG` and the `END_TAG`\n",
    "- `tag_to_ix`: a dictionary that maps each tag (including the `START_TAG` and the `END_TAG`) to a unique index: this is useful to access the respective tag transition scores from the tag_transition_probs variable.\n",
    "- `cur_tag_scores`: pytorch Variable that contains the local emission score for each tag for the current token in the sentence.\n",
    "    - `cur_tag_scores` size is : `[ len(all_tags) ] `\n",
    "- `transition_scores`: pytorch Variable that contains the `tag_transition_scores`. \n",
    "    - `transition_scores` size is : `[ len(all_tags) x len(all_tags) ]` \n",
    "- `prev_scores`: pytorch Variable that contains the scores for each tag for the previous token in the sentence.\n",
    "    - `prev_scores` size is : `[ 1 x len(all_tags) ] `\n",
    "\n",
    "### Outputs\n",
    "- `viterbivars`: a pytorch Variable that contains the global scores for each tag for the current token in the sentence\n",
    "- `bptrs`: a list of idx that contains the best_previous_tag for each tag for the current token in the sentence\n",
    "\n",
    "### Tests\n",
    "- ```test_viterbi.py: test_viterbi_step_init()```  \n",
    "\n",
    "There are a lot of inputs, but the code itself will not be very complex. Make sure you understand what each input represents before starting to write a solution.\n",
    "\n",
    "**Do not convert the pytorch variables into numpy. You will be using this function in BiLSTM-CRF model later on and you need the computation graph to be intact in order to backpropagate.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Consider the sentence: `'they can can fish'`\n",
    "- Let us observe the viterbi scores at each of the tokens 'they', 'can', 'can', 'fish'.\n",
    "- We will walk through this example and all along: these scores should match with the scores you obtained when you worked it out by hand.\n",
    "- **Please note the dimensions of the tensors below**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.003s\n",
      "\n",
      "OK\n",
      "Variable containing:\n",
      "  0 -inf -inf -inf\n",
      "[torch.FloatTensor of size 1x4]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "reload(viterbi);\n",
    "# print(emission_probs[0], 'hallo', torch.max(emission_probs[0]))\n",
    "! nosetests tests/test_viterbi.py:test_viterbi_step_init\n",
    "\n",
    "initial_vec = np.full((1, len(all_tags)),-np.inf) \n",
    "initial_vec[tag_to_ix[START_TAG]][0] = 0 #setting all the score to START_TAG\n",
    "prev_scores = viterbi.get_torch_variable(initial_vec)\n",
    "# these are the previous scores for each_tag: START_TAG, NOUN, VERB, END_TAG\n",
    "print (prev_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The block above says that the only possible previous tag at $m=1$ is `START_TAG`\n",
    "- Now let us look at the tag scores for the first token 'they'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['--START--', 'NOUN', 'VERB', '--END--']\n"
     ]
    }
   ],
   "source": [
    "# make sure both START_TAG and END_TAG is included in all_tags\n",
    "print (all_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Carefully observe all the inputs to the `viterbi_step(..)` function here.\n",
    "    - `all_tags`: is the list of all possible tags here\n",
    "    - `tag_to_ix`: a mapping from tags to unique ids: this is useful to access the respective tag transition scores from the `tag_transition_probs` variable.\n",
    "    - `cur_tag_scores`: observe that from previous section: `emission_probs` indicates the emission scores for each tag for each word they: since we will be tagging the word 'they' right now in our example: we will be using ```emission_probs[0]```: note that '0' is the id for our word 'they'.Thus, we send in `emission_probs[0]` as our cur_tag_scores\n",
    "    - `tag_transition_probs`: tag transition probabilities\n",
    "    - `prev_scores`: prev_scores obtained: we have initially calculated these scores above such that the `START_TAG` has all the score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "from --START-- to --START-- : [-inf]\n",
      "from --START-- to NOUN : [-2.]\n",
      "from --START-- to VERB : [-13.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "reload(viterbi);\n",
    "viterbivars, bptrs = viterbi.viterbi_step(all_tags, tag_to_ix, \n",
    "                                          emission_probs[0], \n",
    "                                          tag_transition_probs,\n",
    "                                          prev_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The following are the scores obtained for each tag for the word token 'they' and the backpointer refers to that particular previous tag which resulted in that score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tag:  --START--  score:  [-inf]  back-pointer-tag:  --START--\n",
      "tag:  NOUN  score:  [-2.]  back-pointer-tag:  --START--\n",
      "tag:  VERB  score:  [-13.]  back-pointer-tag:  --START--\n",
      "tag:  --END--  score:  [-inf]  back-pointer-tag:  --START--\n"
     ]
    }
   ],
   "source": [
    "scores = viterbivars\n",
    "for k,v in tag_to_ix.items():\n",
    "    print ('tag: ',k, ' score: ',scores[v].data.numpy(), ' back-pointer-tag: ', \n",
    "           ix_to_tag[bptrs[v]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `NOUN` has the highest score for the current tag, and its backpointer is to `START_TAG`\n",
    "- Now, let us look at the scores for the tags for the second token 'can'. Send in `emission_probs[1]` as our `current_tag_scores`, and update `prev_scores` to be the scores obtained for $m=1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "from --START-- to --START-- : [-inf]\n",
      "from NOUN to NOUN : [-10.]\n",
      "from NOUN to VERB : [-6.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prev_scores = viterbi.get_torch_variable([-np.inf, -2, -13, -np.inf])\n",
    "viterbivars, bptrs = viterbi.viterbi_step(all_tags, tag_to_ix,\n",
    "                                          emission_probs[1],\n",
    "                                          tag_transition_probs,\n",
    "                                          prev_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The following are the scores obtained for each tag for the word token 'can' and its respective back_pointer tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tag:  --START--  score:  [-inf]  back-pointer-tag:  --START--\n",
      "tag:  NOUN  score:  [-10.]  back-pointer-tag:  NOUN\n",
      "tag:  VERB  score:  [-6.]  back-pointer-tag:  NOUN\n",
      "tag:  --END--  score:  [-inf]  back-pointer-tag:  --START--\n"
     ]
    }
   ],
   "source": [
    "scores = viterbivars\n",
    "for k,v in tag_to_ix.items():\n",
    "    print ('tag: ',k, ' score: ',scores[v].data.numpy(), ' back-pointer-tag: ',\n",
    "           ix_to_tag[bptrs[(all_tags).index(k)]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now, Below, let us look at the scores for the tags for the third token 'can'. So, now we send in `emission_probs[1]` as our `current_tag_scores` and we update `prev_scores` to be the scores obtained for the previous token 'can'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "from --START-- to --START-- : [-inf]\n",
      "from VERB to NOUN : [-10.]\n",
      "from VERB to VERB : [-11.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prev_scores = viterbi.get_torch_variable([-np.inf, -10, -6, -4]) \n",
    "viterbivars, bptrs = viterbi.viterbi_step(all_tags, tag_to_ix,\n",
    "                                          emission_probs[1],\n",
    "                                          tag_transition_probs,\n",
    "                                          prev_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tag:  --START--  score:  [-inf]  back-pointer-tag:  --START--\n",
      "tag:  NOUN  score:  [-10.]  back-pointer-tag:  VERB\n",
      "tag:  VERB  score:  [-11.]  back-pointer-tag:  VERB\n",
      "tag:  --END--  score:  [-inf]  back-pointer-tag:  --START--\n"
     ]
    }
   ],
   "source": [
    "scores = viterbivars\n",
    "for k,v in tag_to_ix.items():\n",
    "    print ('tag: ',k, ' score: ',scores[v].data.numpy(), ' back-pointer-tag: ',\n",
    "           ix_to_tag[bptrs[(all_tags).index(k)]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - Now, let us look at the scores for the tags for the last token 'fish', So, now we send in `emission_probs[2]` as our `current_tag_scores` and we update `prev_scores` to be the scores obtained above for the previous token 'can'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "from --START-- to --START-- : [-inf]\n",
      "from VERB to NOUN : [-15.]\n",
      "from NOUN to VERB : [-16.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prev_scores = viterbi.get_torch_variable([-np.inf, -10, -11, -np.inf])\n",
    "viterbivars, bptrs = viterbi.viterbi_step(all_tags, tag_to_ix,\n",
    "                                           emission_probs[2],\n",
    "                                           tag_transition_probs,\n",
    "                                           prev_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tag:  --START--  score:  [-inf]  back-pointer-tag:  --START--\n",
      "tag:  NOUN  score:  [-15.]  back-pointer-tag:  VERB\n",
      "tag:  VERB  score:  [-16.]  back-pointer-tag:  NOUN\n",
      "tag:  --END--  score:  [-inf]  back-pointer-tag:  --START--\n"
     ]
    }
   ],
   "source": [
    "scores = viterbivars\n",
    "for k,v in tag_to_ix.items():\n",
    "    print ('tag: ',k, ' score: ',scores[v].data.numpy(), ' back-pointer-tag: ',\n",
    "           ix_to_tag[bptrs[(all_tags).index(k)]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 3.4** Build the Viterbi trellis. (*0.5 points*)\n",
    "\n",
    "This function should compute the `best_path` and the `path_score`. This function takes in the `emission_scores` for each particular token in the sentence, the `tag_transitions_weight` and returns the best set of tags for that particular sequence. Use `viterbi_step` to implement `build_trellis` in `viterbi.py` in Pytorch. \n",
    "\n",
    "This function should take:\n",
    "- **Inputs**:\n",
    "    - `all_tags`: a list of all tags: includes START_TAG and END_TAG\n",
    "    - `tag_to_ix`: a dictionary that maps each tag to a unique id.\n",
    "    - `cur_tag_scores`: a list of pytorch Variables where each contains the local emission score for each tag for that particular token in the sentence, len(cur_tag_scores) will be equal to len(words): each pytorch variables size would be equal to len(all_tags) indicating the score for each_tag.\n",
    "    - `transition_scores`: pytorch Variable (a matrix) that contains the tag_transition_scores\n",
    "\n",
    "- **Outputs**:\n",
    "    - `path_score`: the score for the best_path\n",
    "    - `best_path`: the actual best_path, which is the list of tags for each token: exclude the `START_TAG` and `END_TAG` here.   \n",
    "    \n",
    "- **Tests**: ```test_viterbi.py: test_trellis_score(), test_build_trellis()```\n",
    "First, make sure to pass the ```test_trellis_score()``` test and then move on to the ```test_build_trellis()``` test.\n",
    "\n",
    "** Note that for the input cur_tag_scores: we are sending in a list of pytorch variables: one for each token in the sentence to be tagged **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.007s\n",
      "\n",
      "OK\n",
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.009s\n",
      "\n",
      "OK\n",
      "['--START--', 'NOUN', 'VERB', '--END--']\n"
     ]
    }
   ],
   "source": [
    "# make sure START_TAG and END_TAG are in all_tags\n",
    "! nosetests tests/test_viterbi.py:test_trellis_score\n",
    "\n",
    "! nosetests tests/test_viterbi.py:test_build_trellis\n",
    "\n",
    "print (all_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- consider the same sentence as above: 'they can can fish'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['they', 'can', 'can', 'fish']\n",
      "{'they': 0, 'can': 1, 'fish': 2, '<UNK>': 3}\n"
     ]
    }
   ],
   "source": [
    "words = 'they can can fish'.split()\n",
    "print (words)\n",
    "print (word_to_ix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Below, we create `cur_tag_scores` using the `emission_probs` for each word in the sentence `'they can can fish'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "#preparing cur_tag_scores for the above sentence 'they can can fish'\n",
    "cur_tag_scores=[emission_probs[0],emission_probs[1],emission_probs[1],emission_probs[2]];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Complete the code to obtain the correct path_score initially and then use the backpointers to obtain the best_path. \n",
    "- Observe the inputs we are sending in for our example: 'they can can fish'\n",
    "    - `all_tags`: list of all tags including the `START_TAG` and `END_TAG`\n",
    "    - `tag_to_ix`: a mapping from tags to their unique ids\n",
    "    - `cur_tag_scores`: a list of pytorch variables: where each one is the score for each tag for a particular token. We send in these scores for each token in the sentence.\n",
    "    - `tag_transition_probs`: tag transition probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(viterbi);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "from --START-- to --START-- : [-inf]\n",
      "from --START-- to NOUN : [-2.]\n",
      "from --START-- to VERB : [-13.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n",
      "from --START-- to --START-- : [-inf]\n",
      "from NOUN to NOUN : [-10.]\n",
      "from NOUN to VERB : [-6.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n",
      "from --START-- to --START-- : [-inf]\n",
      "from VERB to NOUN : [-10.]\n",
      "from VERB to VERB : [-11.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n",
      "from --START-- to --START-- : [-inf]\n",
      "from VERB to NOUN : [-15.]\n",
      "from NOUN to VERB : [-16.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "path_score, best_path = viterbi.build_trellis(all_tags, \n",
    "                                                  tag_to_ix, \n",
    "                                                  cur_tag_scores, \n",
    "                                                  tag_transition_probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['NOUN', 'VERB', 'VERB', 'NOUN']\n",
      "-17.0\n"
     ]
    }
   ],
   "source": [
    "print (best_path)\n",
    "print (path_score.data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "from --START-- to --START-- : [-inf]\n",
      "from --START-- to NOUN : [-2.]\n",
      "from --START-- to VERB : [-13.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n",
      "from --START-- to --START-- : [-inf]\n",
      "from NOUN to NOUN : [-10.]\n",
      "from NOUN to VERB : [-6.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n",
      "from --START-- to --START-- : [-inf]\n",
      "from VERB to NOUN : [-10.]\n",
      "from VERB to VERB : [-11.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n",
      "from --START-- to --START-- : [-inf]\n",
      "from VERB to NOUN : [-15.]\n",
      "from NOUN to VERB : [-14.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n",
      "from --START-- to --START-- : [-inf]\n",
      "from VERB to NOUN : [-18.]\n",
      "from NOUN to VERB : [-19.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n",
      "from --START-- to --START-- : [-inf]\n",
      "from VERB to NOUN : [-23.]\n",
      "from NOUN to VERB : [-24.]\n",
      "from --START-- to --END-- : [-inf]\n",
      "\n",
      "\n",
      "['NOUN', 'VERB', 'VERB', 'NOUN', 'VERB', 'NOUN'] [-25.]\n"
     ]
    }
   ],
   "source": [
    "sentence = ['they','can','can','can','can','fish']\n",
    "cur_tag_scores = [emission_probs[word_to_ix[w]] for w in sentence]\n",
    "\n",
    "path_score, best_path = viterbi.build_trellis(all_tags, \n",
    "                                                  tag_to_ix, \n",
    "                                                  cur_tag_scores, \n",
    "                                                  tag_transition_probs)\n",
    "print (best_path, path_score.data.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Hidden Markov Model: Estimation\n",
    "\n",
    "You will now implement the estimatation for a hidden Markov model.\n",
    "\n",
    "We'll start with the tag transitions.\n",
    "\n",
    "**Deliverable 4.1** (*0.5 points* for 4650, *0.25 points* for 7650) Complete the function `most_common.get_tag_trans_counts()`.  This function should get the tag transition counts from the each tag to all possible tags. Don't forget to add the transitions from the `START_TAG` and the transitions from the `END_TAG`.\n",
    "\n",
    "You should use the `preproc.conll_seq_generator()` function.  \n",
    "\n",
    "- **Inputs**: `trainfile`, name of file containing training data\n",
    "- **Outputs**: a dictionary where keys are current tags and values are counters of succeeding tags.\n",
    "- **Tests**: ```test_hmm_trans_counts.py: test_tag_trans_counts()```  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.179s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "reload(most_common);\n",
    "! nosetests tests/test_hmm_trans_counts.py:test_tag_trans_counts\n",
    "tag_trans_counts = most_common.get_tag_trans_counts(TRAIN_FILE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function returns a dict of counters, where the keys are tags.\n",
    "\n",
    "Each counter is the frequency of tags following a given tag, e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'NOUN': 1753, 'ADJ': 866, 'PROPN': 562, 'VERB': 65, 'PUNCT': 61, 'ADV': 59, 'NUM': 55, 'DET': 29, 'ADP': 29, 'PRON': 10, 'AUX': 7, 'X': 7, 'SYM': 2})\n",
      "Counter({'PRON': 422, 'PROPN': 327, 'DET': 284, 'ADV': 172, 'ADP': 142, 'NOUN': 108, 'PUNCT': 80, 'CONJ': 79, 'SCONJ': 75, 'ADJ': 67, 'VERB': 64, 'X': 51, 'NUM': 42, 'AUX': 42, 'INTJ': 31, 'SYM': 9, 'PART': 5})\n"
     ]
    }
   ],
   "source": [
    "print (tag_trans_counts['DET'])\n",
    "print (tag_trans_counts[START_TAG])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 4.2** Estimate transition log-probabilities for an HMM. (*0.5 points* for 4650, *0.25 points* for 7650)\n",
    "\n",
    "Implement `compute_transition_weights` in `hmm.py`. This function should return a dictionary of weights such that ```weights[(tag2,tag1)]``` = indicates the weights for transitions from `tag1` $\\rightarrow$ `tag2`. These weights will be used later for the Viterbi Tagger.\n",
    "\n",
    "### Inputs\n",
    "- Transition counts (generated from `get_tag_trans_counts`)\n",
    "- Smoothing\n",
    "\n",
    "### Outputs\n",
    "- Defaultdict with weights for transition features, in the form $[(y_m,y_{m-1})]$\n",
    "\n",
    "### Tests\n",
    "```test_hmm_trans.py: test_hmm_trans_weights_sum_to_one(), test_hmm_trans_weights_exact_vals() ```  \n",
    "\n",
    "Hints: \n",
    "\n",
    "- Don't forget to assign smoothed probabilities to transitions which do not appear in the counts. \n",
    "- Do not assign probabilities for transitions to the `START_TAG`, which can only come first. This will also affect your computation of the denominator, since you are not smoothing the probability of transitions to the `START_TAG`.\n",
    "- Don't forget to assign probabilities to transitions to the `END_TAG`; this too will affect your denominator.\n",
    "- As always, probabilities should sum to one (this time conditioned on the previous tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.176s\n",
      "\n",
      "OK\n",
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.187s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "reload(hmm);\n",
    "sum(tag_trans_counts['NOUN'].values())\n",
    "hmm_trans_weights = hmm.compute_transition_weights(tag_trans_counts,.001)\n",
    "! nosetests tests/test_hmm_trans.py:test_hmm_trans_weights_sum_to_one\n",
    "! nosetests tests/test_hmm_trans.py:test_hmm_trans_weights_exact_vals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [],
   "source": [
    "hmm_trans_weights = hmm.compute_transition_weights(tag_trans_counts,.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "108 -2.9187709731609703\n",
      "64 -3.4420127512639795\n",
      "65 -3.9875482805835087\n",
      "0 -15.069706213958325\n",
      "1753 -0.6928664796044606\n",
      "0 -inf\n",
      "-inf\n"
     ]
    }
   ],
   "source": [
    "print (tag_trans_counts[START_TAG]['NOUN'], hmm_trans_weights[('NOUN',START_TAG)])\n",
    "print (tag_trans_counts[START_TAG]['VERB'], hmm_trans_weights[('VERB',START_TAG)])\n",
    "print (tag_trans_counts['DET']['VERB'], hmm_trans_weights[('VERB','DET')])\n",
    "print (tag_trans_counts['DET']['INTJ'], hmm_trans_weights[('INTJ','DET')])\n",
    "print (tag_trans_counts['DET']['NOUN'], hmm_trans_weights[('NOUN','DET')])\n",
    "print (tag_trans_counts['VERB'][START_TAG], hmm_trans_weights[(START_TAG,'VERB')])\n",
    "# print (tag_trans_counts[END_TAG]['VERB']) # will throw key error\n",
    "print (hmm_trans_weights[('VERB',END_TAG)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These log-probabilities should normalize to when summing over $y_m$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9999999999999998\n",
      "1.0\n",
      "0.9999999999999999\n"
     ]
    }
   ],
   "source": [
    "#calculating all tags here, we also add END_TAG here.\n",
    "all_tags = sorted(list(tag_trans_counts.keys()) + [END_TAG])\n",
    "print (sum(np.exp(hmm_trans_weights[(tag,'NOUN')]) for tag in all_tags))\n",
    "print (sum(np.exp(hmm_trans_weights[(tag,'SYM')]) for tag in all_tags))\n",
    "print (sum(np.exp(hmm_trans_weights[(tag,'ADJ')]) for tag in all_tags))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Now let us compute the weight variables for the whole dataset**\n",
    "- So, we recalculate them below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [],
   "source": [
    "#recalculating nb_weights for the whole dataset\n",
    "nb_weights = naive_bayes.get_nb_weights(TRAIN_FILE, .01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'--END--': 0, '--START--': 1, 'ADJ': 2, 'ADP': 3, 'ADV': 4, 'AUX': 5, 'CONJ': 6, 'DET': 7, 'INTJ': 8, 'NOUN': 9, 'NUM': 10, 'PART': 11, 'PRON': 12, 'PROPN': 13, 'PUNCT': 14, 'SCONJ': 15, 'SYM': 16, 'VERB': 17, 'X': 18}\n"
     ]
    }
   ],
   "source": [
    "# recalculating tag_to_ix={}\n",
    "tag_to_ix={}\n",
    "for tag in list(all_tags):\n",
    "    if tag not in tag_to_ix:\n",
    "        tag_to_ix[tag]=len(tag_to_ix)\n",
    "print (tag_to_ix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Note about OOV's**: We provide a helper function to calculate the `vocab` as shown below. We add an `UNK` token to the `vocab`. This is useful because, when we don't find a token's emission weight, we choose the `UNK` tokens weight and proceed with our tagging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6913\n"
     ]
    }
   ],
   "source": [
    "# recalculating vocab for the whole dataset now. # we also add an UNK token to the vocab here\n",
    "reload(most_common);\n",
    "vocab, word_to_ix = most_common.get_word_to_ix(TRAIN_FILE) #obtains all the words in the file\n",
    "print (len(vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [],
   "source": [
    "emission_probs, tag_transition_probs = hmm.compute_weights_variables(nb_weights, hmm_trans_weights, \n",
    "                                                                     vocab, word_to_ix, tag_to_ix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 4.3** (*0.5 points*)\n",
    "\n",
    "We can now combine `Viterbi` and the `HMM` weights to compute the tag sequence for the example sentence. Make sure your implementation passes the test for this deliverable, and **explain (in `text-answers.md`) whether you think these predicted tags are correct**, based on your understanding of the universal part-of-speech tag set.\n",
    "\n",
    "- **Tests**: ```test_hmm.py: test_hmm_on_example_sentence()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['--END--', '--START--', 'ADJ', 'ADP', 'ADV', 'AUX', 'CONJ', 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']\n",
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.007s\n",
      "\n",
      "OK\n",
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.007s\n",
      "\n",
      "OK\n",
      ".\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.974s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "# make sure all_tags has END_TAG\n",
    "print (all_tags)\n",
    "! nosetests tests/test_viterbi.py:test_trellis_score\n",
    "! nosetests tests/test_viterbi.py:test_build_trellis\n",
    "! nosetests tests/test_hmm.py:test_hmm_on_example_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "from --END-- to --END-- : [-inf]\n",
      "from --END-- to --START-- : [-inf]\n",
      "from --START-- to ADJ : [-15.932702]\n",
      "from --START-- to ADP : [-15.555761]\n",
      "from --START-- to ADV : [-14.590198]\n",
      "from --START-- to AUX : [-15.812539]\n",
      "from --START-- to CONJ : [-15.035642]\n",
      "from --START-- to DET : [-14.738578]\n",
      "from --START-- to INTJ : [-13.601336]\n",
      "from --START-- to NOUN : [-16.326632]\n",
      "from --START-- to NUM : [-15.0324135]\n",
      "from --START-- to PART : [-17.599619]\n",
      "from --START-- to PRON : [-4.5463486]\n",
      "from --START-- to PROPN : [-14.726809]\n",
      "from --START-- to PUNCT : [-16.288916]\n",
      "from --START-- to SCONJ : [-14.659508]\n",
      "from --START-- to SYM : [-14.829995]\n",
      "from --START-- to VERB : [-16.521912]\n",
      "from --START-- to X : [-13.398914]\n",
      "\n",
      "\n",
      "from --END-- to --END-- : [-inf]\n",
      "from --END-- to --START-- : [-inf]\n",
      "from PRON to ADJ : [-20.129038]\n",
      "from PRON to ADP : [-20.72856]\n",
      "from PRON to ADV : [-19.567686]\n",
      "from PRON to AUX : [-9.181789]\n",
      "from PRON to CONJ : [-20.636398]\n",
      "from PRON to DET : [-21.504446]\n",
      "from INTJ to INTJ : [-26.367805]\n",
      "from PRON to NOUN : [-20.04963]\n",
      "from PRON to NUM : [-21.706017]\n",
      "from PRON to PART : [-20.440554]\n",
      "from PRON to PRON : [-20.510601]\n",
      "from PRON to PROPN : [-21.912365]\n",
      "from PRON to PUNCT : [-20.415176]\n",
      "from PRON to SCONJ : [-20.814417]\n",
      "from PRON to SYM : [-21.06152]\n",
      "from PRON to VERB : [-18.60736]\n",
      "from X to X : [-25.42138]\n",
      "\n",
      "\n",
      "from --END-- to --END-- : [-inf]\n",
      "from --END-- to --START-- : [-inf]\n",
      "from AUX to ADJ : [-27.917799]\n",
      "from AUX to ADP : [-28.697285]\n",
      "from AUX to ADV : [-23.48713]\n",
      "from AUX to AUX : [-14.643444]\n",
      "from AUX to CONJ : [-27.59079]\n",
      "from AUX to DET : [-26.701864]\n",
      "from AUX to INTJ : [-32.822445]\n",
      "from AUX to NOUN : [-28.27847]\n",
      "from AUX to NUM : [-27.648443]\n",
      "from AUX to PART : [-22.9012]\n",
      "from AUX to PRON : [-24.870853]\n",
      "from AUX to PROPN : [-27.450054]\n",
      "from AUX to PUNCT : [-27.0653]\n",
      "from AUX to SCONJ : [-27.855347]\n",
      "from AUX to SYM : [-32.814423]\n",
      "from AUX to VERB : [-22.76332]\n",
      "from AUX to X : [-33.11785]\n",
      "\n",
      "\n",
      "from --END-- to --END-- : [-inf]\n",
      "from --END-- to --START-- : [-inf]\n",
      "from AUX to ADJ : [-33.379456]\n",
      "from AUX to ADP : [-34.15894]\n",
      "from AUX to ADV : [-28.948784]\n",
      "from AUX to AUX : [-29.000866]\n",
      "from AUX to CONJ : [-33.052444]\n",
      "from AUX to DET : [-32.16352]\n",
      "from AUX to INTJ : [-38.2841]\n",
      "from AUX to NOUN : [-29.125004]\n",
      "from AUX to NUM : [-33.1101]\n",
      "from AUX to PART : [-28.362854]\n",
      "from AUX to PRON : [-30.332508]\n",
      "from AUX to PROPN : [-32.91171]\n",
      "from AUX to PUNCT : [-32.526955]\n",
      "from AUX to SCONJ : [-33.317]\n",
      "from AUX to SYM : [-38.276077]\n",
      "from AUX to VERB : [-28.224976]\n",
      "from AUX to X : [-38.579506]\n",
      "\n",
      "\n",
      "from --END-- to --END-- : [-inf]\n",
      "from --END-- to --START-- : [-inf]\n",
      "from VERB to ADJ : [-43.23774]\n",
      "from VERB to ADP : [-42.728928]\n",
      "from VERB to ADV : [-42.90307]\n",
      "from AUX to AUX : [-43.358288]\n",
      "from NOUN to CONJ : [-43.722504]\n",
      "from VERB to DET : [-42.631275]\n",
      "from VERB to INTJ : [-45.425976]\n",
      "from PART to NOUN : [-43.705063]\n",
      "from VERB to NUM : [-43.884064]\n",
      "from VERB to PART : [-42.637527]\n",
      "from VERB to PRON : [-42.989685]\n",
      "from VERB to PROPN : [-44.239742]\n",
      "from NOUN to PUNCT : [-31.480871]\n",
      "from VERB to SCONJ : [-42.580635]\n",
      "from ADV to SYM : [-44.77014]\n",
      "from PART to VERB : [-41.897945]\n",
      "from ADV to X : [-45.478867]\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(Variable containing:\n",
       " -32.4456\n",
       " [torch.FloatTensor of size 1], ['PRON', 'AUX', 'AUX', 'NOUN', 'PUNCT'])"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reload(viterbi);\n",
    "viterbi.build_trellis(all_tags,\n",
    "                      tag_to_ix,\n",
    "                      [emission_probs[word_to_ix[w]] for w in ['they', 'can', 'can', 'fish','.']], \n",
    "                      tag_transition_probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 4.4** (*0.5 points*)\n",
    "\n",
    "- Run your HMM tagger on the dev data and test data, using the code blocks below.\n",
    "- **Tests**: ```test_hmm.py: test_hmm_dev_accuracy(), test_hmm_test_accuracy()```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Observe that, based on our definition of the viterbi function, we need to send in two sets of important scores to the `build_trellis()` function\n",
    "    - cur_tag_scores: a list of emission scores for each tag for each token in the sentence\n",
    "    - tag_transition_probs: tag transition scores\n",
    "- When using the `HMM` with `Viterbi` Tagger: we have calculated the `cur_tag_scores` using the `naive_bayes_weights` and `tag_transition_probs` in 4.1.\n",
    "- As I have already mentioned above, for `cur_tag_scores`, we are sending in a list of pytorch variables: one for each token in the sentence to be tagged\n",
    "- Below, in the tagger that we create: we first calculate the set of `cur_tag_scores` for the words in the sentence and then send them in. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(viterbi);\n",
    "# print(cur_tag_scores, tag_to_ix, tag_transition_probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Observe the way `cur_tag_scores` is computed in the loop below: \n",
    "    - For each particular word in a sentence: we assign the respective emission scores if it is present in our `vocab`, else we assign the emission_scores of an `UNK` token.\n",
    "    - This is repeated everywhere from now on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From/ADP the/DET AP/NOUN comes/VERB this/DET story/NOUN :/PUNCT \n",
      "\n",
      "President/PROPN Bush/PROPN on/ADP Tuesday/PROPN nominated/PROPN two/NUM individuals/NOUN to/PART replace/VERB retiring/DET jurists/NOUN on/ADP federal/ADJ courts/NOUN in/ADP the/DET Washington/PROPN area/NOUN ./PUNCT \n",
      "\n",
      "Bush/PROPN nominated/PROPN Jennifer/PROPN M./PART Anderson/VERB for/ADP a/DET 15/NUM -/PUNCT year/NOUN term/NOUN as/ADP associate/NOUN judge/NOUN of/ADP the/DET Superior/ADJ Court/NOUN of/ADP the/DET District/PROPN of/ADP Columbia/PROPN ,/PUNCT replacing/PROPN Steffen/PROPN W./PROPN Graae/PROPN ./PUNCT \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# this is just for fun\n",
    "# print(cur_tag_scores)\n",
    "\n",
    "for i,(words,_) in enumerate(preproc.conll_seq_generator(DEV_FILE)):\n",
    "    cur_tag_scores = [emission_probs[word_to_ix[w]] \n",
    "                      if w in word_to_ix else emission_probs[word_to_ix[UNK]] for w in words]\n",
    "    \n",
    "    pred_tags = viterbi.build_trellis(all_tags,\n",
    "                                      tag_to_ix,\n",
    "                                      cur_tag_scores,\n",
    "                                      tag_transition_probs)[1]\n",
    "    for word,pred_tag in zip(words,pred_tags):\n",
    "        print (\"%s/%s\"%(word,pred_tag),end=\" \")\n",
    "    print ('\\n')\n",
    "    if i >= 2: break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger = lambda words, all_tags : viterbi.build_trellis(all_tags, \n",
    "                                                        tag_to_ix,\n",
    "                                                            [emission_probs[word_to_ix[w]] \n",
    "                                                             if w in word_to_ix \n",
    "                                                             else emission_probs[word_to_ix[UNK]] \n",
    "                                                             for w in words],\n",
    "                                                            tag_transition_probs)[1]\n",
    "confusion = tagger_base.eval_tagger(tagger,'hmm-dev-en.preds', all_tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8409120864636879\n"
     ]
    }
   ],
   "source": [
    "print (scorer.accuracy(confusion))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger_base.apply_tagger(tagger,'hmm-te-en.preds',all_tags, testfile=TEST_FILE_HIDDEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'data/en-ud-test.conllu'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-187-25eadfdb0d50>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# you don't have en-ud-test.conllu, so you can't run this\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mte_confusion\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscorer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_confusion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mTEST_FILE\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'hmm-te-en.preds'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mscorer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maccuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mte_confusion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Dev/cs4650/psets/ps2/gtnlplib/scorer.py\u001b[0m in \u001b[0;36mget_confusion\u001b[0;34m(keyfilename, responsefilename)\u001b[0m\n\u001b[1;32m     24\u001b[0m     \"\"\"\n\u001b[1;32m     25\u001b[0m     \u001b[0mcounts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdefaultdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m     \u001b[0;32mwith\u001b[0m \u001b[0mcodecs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeyfilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'utf8'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mkeyfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     27\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponsefilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mresfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mkey_line\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkeyfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Dev/Lib/miniconda/lib/python3.6/codecs.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(filename, mode, encoding, errors, buffering)\u001b[0m\n\u001b[1;32m    893\u001b[0m         \u001b[0;31m# Force opening of the file in binary mode\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m         \u001b[0mmode\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'b'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 895\u001b[0;31m     \u001b[0mfile\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbuiltins\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffering\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    896\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mencoding\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'data/en-ud-test.conllu'"
     ]
    }
   ],
   "source": [
    "# you don't have en-ud-test.conllu, so you can't run this\n",
    "te_confusion = scorer.get_confusion(TEST_FILE,'hmm-te-en.preds')\n",
    "print (scorer.accuracy(te_confusion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part-of-Speech Tagging in Norwegian\n",
    "**Deliverable 4.5** (*0.5 points*)\n",
    "- Now, let us do part of speech tagging for data in Norwegian language using the Viterbi Tagger.\n",
    "- **Tests**: ```test_hmm.py: test_nr_hmm_dev_accuracy(), test_nr_hmm_test_accuracy()```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- First, we calculate the `nb_weights`/emission weights for the norwegian language in a similar way as we did for the english language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [],
   "source": [
    "#recalculating nb_weights for the whole dataset\n",
    "nb_weights_nr = naive_bayes.get_nb_weights(NR_TRAIN_FILE, .01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now, we calculate the `tag_transition_weights` for the norwegian language"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_trans_counts_nr = most_common.get_tag_trans_counts(NR_TRAIN_FILE)\n",
    "hmm_trans_weights_nr = hmm.compute_transition_weights(tag_trans_counts_nr,.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now, we obtain the `vocab`, `word_to_ix` and `tag_to_ix` below for the norwegian language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'--END--': 0, '--START--': 1, 'ADJ': 2, 'ADP': 3, 'ADV': 4, 'AUX': 5, 'CCONJ': 6, 'DET': 7, 'INTJ': 8, 'NOUN': 9, 'NUM': 10, 'PART': 11, 'PRON': 12, 'PROPN': 13, 'PUNCT': 14, 'SCONJ': 15, 'SYM': 16, 'VERB': 17, 'X': 18}\n"
     ]
    }
   ],
   "source": [
    "#Using helper functions to obtain vocab, word_to_ix, tag_to_ix\n",
    "all_tags_nr = sorted(list(tag_trans_counts_nr.keys()) + [END_TAG])\n",
    "vocab_nr, word_to_ix_nr = most_common.get_word_to_ix(NR_TRAIN_FILE) #obtains all the words in the vocab\n",
    "tag_to_ix_nr={}\n",
    "for tag in list(all_tags_nr):\n",
    "    tag_to_ix_nr[tag]=len(tag_to_ix_nr)\n",
    "print (tag_to_ix_nr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now, we convert these weights into pytorch variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [],
   "source": [
    "emission_probs_nr, tag_transition_probs_nr = hmm.compute_weights_variables(nb_weights_nr, hmm_trans_weights_nr, \n",
    "                                                                           vocab_nr, word_to_ix_nr, tag_to_ix_nr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now, we construct a viterbi tagger for the norwegian language using these weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger = lambda words, all_tags : viterbi.build_trellis(all_tags_nr, \n",
    "                                                           tag_to_ix_nr,\n",
    "                                                            [emission_probs_nr[word_to_ix_nr[w]] \n",
    "                                                             if w in word_to_ix_nr \n",
    "                                                             else emission_probs_nr[word_to_ix_nr[UNK]] \n",
    "                                                             for w in words],\n",
    "                                                            tag_transition_probs_nr)[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion = tagger_base.eval_tagger(tagger,'hmm-dev-nr.preds', all_tags_nr,\n",
    "                                    trainfile=NR_TRAIN_FILE,\n",
    "                                    testfile=NR_DEV_FILE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8615031554790591\n"
     ]
    }
   ],
   "source": [
    "print (scorer.accuracy(confusion))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger_base.apply_tagger(tagger,'hmm-te-nr.preds',all_tags_nr, \n",
    "                         trainfile=NR_TRAIN_FILE, testfile=NR_TEST_FILE_HIDDEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'data/no_bokmaal-ud-test.conllu'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-196-d6b8a1e28724>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# you don't have no_bokmaal-ud-test.conllu, so you can't run this\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mte_confusion\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscorer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_confusion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mNR_TEST_FILE\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'hmm-te-nr.preds'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mscorer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maccuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mte_confusion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Dev/cs4650/psets/ps2/gtnlplib/scorer.py\u001b[0m in \u001b[0;36mget_confusion\u001b[0;34m(keyfilename, responsefilename)\u001b[0m\n\u001b[1;32m     24\u001b[0m     \"\"\"\n\u001b[1;32m     25\u001b[0m     \u001b[0mcounts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdefaultdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m     \u001b[0;32mwith\u001b[0m \u001b[0mcodecs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeyfilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'utf8'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mkeyfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     27\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponsefilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mresfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mkey_line\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkeyfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Dev/Lib/miniconda/lib/python3.6/codecs.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(filename, mode, encoding, errors, buffering)\u001b[0m\n\u001b[1;32m    893\u001b[0m         \u001b[0;31m# Force opening of the file in binary mode\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m         \u001b[0mmode\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'b'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 895\u001b[0;31m     \u001b[0mfile\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbuiltins\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffering\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    896\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mencoding\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'data/no_bokmaal-ud-test.conllu'"
     ]
    }
   ],
   "source": [
    "# you don't have no_bokmaal-ud-test.conllu, so you can't run this\n",
    "te_confusion = scorer.get_confusion(NR_TEST_FILE,'hmm-te-nr.preds')\n",
    "print (scorer.accuracy(te_confusion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. BiLSTM model for Part of Speech Tagging\n",
    "=========================\n",
    "\n",
    "A `BiLSTM` model for part-of-speech tagging takes as input the word embeddings of the tokens in the sentence, and passes them through an `LSTM`. For each token, the hidden state is used as input to a network that computes a score for each tag. A softmax layer then converts these scores to probabilities. This model should be trained end-to-end with the cross-entropy loss function.\n",
    "\n",
    "We will be building this `BiLSTM` model as a class using pytorch. Your implementation will include three functions:\n",
    "\n",
    "- `BiLSTM.__init__()`: define all the necessary model parameters\n",
    "    1. The word-embedding matrix, which maps the words to vectors\n",
    "    2. A BiLSTM Neural Network, which takes the word embeddings for the words as inputs and produces a hidden state for each token.\n",
    "    3. A one layer feedforward Neural Network, which projects the hidden state to a vector of scores for each tag\n",
    "- `forward()`: pass the input through the model, obtaining probability distributions over tags\n",
    "    1. Convert all the words to their word-vectors from the word-embedding matrix\n",
    "    2. Pass these word-vectors through a BiLSTM to obtain hidden states for the tokens\n",
    "    3. Pass these hidden states through the feedforward neural network to obtain the probability distributions of tags for each token.\n",
    "- `cross_entopy_loss()`, the training objective\n",
    "\n",
    "The description below provides additional help for each of these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "words in the vocabulary:  6900\n",
      "6899\n"
     ]
    }
   ],
   "source": [
    "# recalculating vocab: obtains the most common 6900 words from the file\n",
    "vocab, word_to_ix = most_common.get_word_to_ix(TRAIN_FILE, 6900)\n",
    "print ('words in the vocabulary: ',len(word_to_ix))\n",
    "print (word_to_ix[UNK])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- updating `tag_to_ix` and `all_tags` to remove `START_TAG` and `END_TAG`: these labels are not necessary in tagging with `BiLSTM`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [],
   "source": [
    "if START_TAG in all_tags:\n",
    "    all_tags.remove(START_TAG)\n",
    "if END_TAG in all_tags:\n",
    "    all_tags.remove(END_TAG)\n",
    "tag_to_ix={}\n",
    "for tag in all_tags:\n",
    "    if tag not in tag_to_ix:\n",
    "        tag_to_ix[tag] = len(tag_to_ix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Take a look at the helper functions `preproc.load_data(...)` and `bilstm.prepare_sequence(...)` \n",
    "    - `preproc.load_data(...)`: loads the data into a list of lists\n",
    "    - `bilstm.prepare_sequence(...)` given a sequence of words/tags and the `to_ix` dictionary that maps them to its unique indices: it returns a sequence of its unique indices.\n",
    "- The function `prepare_sequence()` will be used a lot from now on to convert the input to a `torch.LongTensor` and then send it to the model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Loading Train data for english:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  17  759   14  114  451   20    0  278    5   33  901 2020    2]\n",
      "[ 5  0  3  3 15  1  5  7  1  5  7  7 12]\n"
     ]
    }
   ],
   "source": [
    "reload(preproc);\n",
    "X_tr, Y_tr = preproc.load_data(TRAIN_FILE)\n",
    "\n",
    "print (bilstm.prepare_sequence(X_tr[5],word_to_ix).data.numpy())\n",
    "print (bilstm.prepare_sequence(Y_tr[5],tag_to_ix).data.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Loading Dev data for english:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_dv, Y_dv = preproc.load_data(DEV_FILE)\n",
    "# loading dev data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Loading Test data for english:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.511s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "# X_te, Y_te = preproc.load_data(TEST_FILE_HIDDEN)\n",
    "# loading test data\n",
    "! nosetests tests/test_bilstm.py:test_dlmodel_init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 5.1**: (*0.5 points*)\n",
    "- Complete the `__init__()` function in the Class `bilstm.BiLSTM` that defines the model parameters:\n",
    "    - **Inputs**:\n",
    "        - `vocab_size`: vocab size of the model\n",
    "        - `tag_to_ix`: tag_to_ix: a dictionary that maps the tags to its unique id\n",
    "        - `embedding_dim`: embedding dimension for the words\n",
    "        - `hidden_dim`: hidden dimension for the `Bi-LSTM` model\n",
    "        - `embeddings`: embedding matrix of size: vocab_size x embedding_dim (this is to initialize embeddings with pretrained embeddings).\n",
    "    - The function does the following:\n",
    "        - Create an embedding matrix using torch.nn.Embedding of the size vocab \n",
    "            - [check this pytorch_doc_for_embedding](http://pytorch.org/docs/0.3.0/nn.html?highlight=embedding#torch.nn.Embedding)\n",
    "        - Create a Bi-LSTM model with just one layer, and hidden dimension = hidden_dim \n",
    "            - [check this pytorch_doc_for_LSTM](http://pytorch.org/docs/0.3.0/nn.html?highlight=lstm#torch.nn.LSTM)\n",
    "        - Also add a FullyConnected Layer, that would project the hidden state onto the tag space. \n",
    "            - [check this pytorch_doc_for_Linear](http://pytorch.org/docs/0.3.0/nn.html?highlight=linear#torch.nn.Linear)\n",
    "    - Make sure to name the parameters as follows: The unit tests will check for these variables\n",
    "        - `self.word_embeds`\n",
    "        - `self.lstm`\n",
    "        - `self.hidden2tag`\n",
    "    - **Tests**: ```test_bilstm.py: test_dlmodel_init()```\n",
    "    - All you need to do here is to define the model parameters here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Below, you can find brief example of how these components are created in Torch. In these examples, the dimensions are arbitrary; you will need to determine the correct dimensions for your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this basically defines a matrix of embeddings where the vocab_size=10 and the embedding_dim=10 \n",
    "word_embeds = nn.Embedding(num_embeddings=10, embedding_dim=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the following line is how you define an LSTM such that the input_size of the vector=10, \n",
    "# hidden state of each LSTM (forward and backward) =20, num of layers=1, \n",
    "# bidirectional=True indicating both forward and backward LSTM will be included.\n",
    "lstm = nn.LSTM(input_size=10, hidden_size = 20, num_layers=1, bidirectional=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the following line is the way to define a Fully Connected Layer with input_dim=40, output_dim=10 and bias=True.\n",
    "hidden2tag = nn.Linear(in_features=40, out_features=10, bias=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Once, you have defined the parameters of the model: check if you have done it right using the unit test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(bilstm);\n",
    "torch.manual_seed(765);\n",
    "embedding_dim=30\n",
    "hidden_dim=30\n",
    "model = bilstm.BiLSTM(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BiLSTM(\n",
       "  (word_embeds): Embedding(6900, 30)\n",
       "  (lstm): LSTM(30, 15, bidirectional=True)\n",
       "  (hidden2tag): Linear(in_features=30, out_features=17)\n",
       ")"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 5.2** (*0.5 points*)\n",
    "- Complete the `bilstm.BiLSTM.forward()` function in the `bilstm.BiLSTM` class to obtain the scores for each tag for each of the words in a sentence\n",
    "- **Input**:\n",
    "    - sentence: a sequence of ids for each word in the sentence\n",
    "- The function does the following:\n",
    "    - Obtains the embeddings for the input sequence\n",
    "    - passes them through an `LSTM` to get the respective hidden states; use the hidden state initialized in the function.\n",
    "    - projects them onto the tag-space using the FC layer\n",
    "- Make sure to reshape the embeddings of the words before sending them to the `BiLSTM`. The axes semantics are: `seq_len, mini_batch, embedding_dim`.\n",
    "- You can use the .view() method to reshape a tensor. You might need to use this, as the neural network components expect their inputs to have a certain shape. [check the pytorch doc on view](http://pytorch.org/docs/0.3.0/tensors.html?highlight=view#torch.Tensor.view)\n",
    "- **Tests**: ```test_bilstm.py: test_dlmodel_forward()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting the random seed\n",
    "reload(bilstm);\n",
    "# ! nosetests tests/test_bilstm.py:test_dlmodel_forward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [],
   "source": [
    "#creating your model\n",
    "torch.manual_seed(765);\n",
    "model = bilstm.BiLSTM(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preparing your first sentence to be an input\n",
    "words= X_tr[5]\n",
    "tags = Y_tr[5]\n",
    "sentence = bilstm.prepare_sequence(words, word_to_ix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- this calls the `forward()` function on the model, which returns the tag_scores for each tag for each particular token in the sentence\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 0.1702\n",
      "-0.1568\n",
      " 0.2289\n",
      " 0.0965\n",
      "-0.0905\n",
      "[torch.FloatTensor of size 5]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lstm_feats = model(sentence)\n",
    "print (lstm_feats[0][0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- we provide the `predict()` function that returns the set of tags obtained for the specific input by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ADV', 'NUM', 'ADJ']\n"
     ]
    }
   ],
   "source": [
    "tags = model.predict(sentence)\n",
    "print (tags[0:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Train the model for now\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1: Dev Accuracy: 0.5463292473968631\n",
      "Epoch 3: Dev Accuracy: 0.6546724660603664\n",
      "Epoch 5: Dev Accuracy: 0.6737841043890866\n",
      "Epoch 7: Dev Accuracy: 0.6985633320152893\n",
      "Epoch 9: Dev Accuracy: 0.7249242124686964\n"
     ]
    }
   ],
   "source": [
    "reload(bilstm);\n",
    "torch.manual_seed(765);\n",
    "loss = torch.nn.CrossEntropyLoss()\n",
    "model, losses, accuracies = bilstm.train_model(loss, model, X_tr,Y_tr, word_to_ix, tag_to_ix, \n",
    "                                        X_dv, Y_dv, num_its=10, status_frequency=2, \n",
    "                                        optim_args = {'lr':0.1,'momentum':0}, param_file = 'best.params')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtQAAACdCAYAAACU5e99AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xd81tX5//HXlUX2XpAEEiAMQYbgFheKs9rWbafVam1tHa22dlo7frbWWlttrau1rdXW+cVK3QOVOhgKsmRDICHskIRAxvX7474TAgaJZHyS3O/n45HHfX8m10fgeHE+1znH3B0RERERETkwUUEHICIiIiLSmymhFhERERHpACXUIiIiIiIdoIRaRERERKQDlFCLiIiIiHSAEmoRERERkQ5QQi0iIiIi0gFKqEVEREREOkAJtYiIiIhIB8QEHcAnlZ2d7cXFxUGHISJyQGbNmrXR3XOCjqO7qM0Wkd6svW12r0uoi4uLmTlzZtBhiIgcEDNbFXQM3Ulttoj0Zu1ts1XyISIiIiLSARGTULt70CGIiIiISB/U60o+Pil357bnP2R7XT0/PXt00OGIiIiISBfa2dDIyo21LKnczpL11SytrKY4O5HrTxnRZb9mn0+ozYy6+kYe/N8qJo/M49hhETMWSERERKTPqqtvZPmGmpbEeUnldpZUVrNqUy2NTaHKBDMYlJlI/7T4Lo2lzyfUAN85ZTivLK7ku4/P5dlrjiUtITbokERERESkHWp3NbCssqYlYQ71Om9n9eZawnkz0VHGoKxESnOTOX10f0rzkhmam8yQnGTiY6O7PMaISKjjY6O57fxxnPOnGdz89AJuO39s0CGJiIiISCvVOxtYWlnNkvXNiXPos2zLjpZzYqONkuwkRg1I4+xxBZTmJVOam0JxdiL9Yro+cd6XiEioAcYVpfP144fwh5eXcsqoPKaMyg86JBEREZGIs21HPUtbyjRCP0vXb2fdtrqWc+JiohicncQhAzO4YGJRuMc5hUFZicRG97w5NSImoQb45omlvLiwku8/OY+JxZlkJsUFHZKIiIhIn9LQ2ERFVR3rttaxdmst67bWUbZlB6s317BkfTWV23e2nBsfG8XQ3GQOH5zF0NxkSnOTKc1LoSgjgZgemDjvS0Ql1HExUfz2/LGcdecb/OipD7jrc4cEHZKIiIhIr1Kzs4G1W3eEfrbsYF34+7rwdkVVXUttc7OspDgKMxM5dlhOOGkOlWoUpCcQFWXBPEgniqiEGmBk/1SuOWkYtz63mFPfX8enxg4IOiQRERGRHqGpydlYszPUu9wqWS5r9X3bjvo9romJMvqnxzMgLYEjhmRRkJ5AQXoCA9ITKMhIYEBaAglxwdU3d4eIS6gBrjh2MC8sWM+P/u8DDi/JJDe1a6dSEREREekJdjY0Ur61jnVbd1DWqld53bbmzzp2NTTtcU1yv5hQkpyRwIRBGS2JckF6PAXpieSk9CO6D/Qyd0REJtQx0VHcdv5YTr/jdW58Yh73fWkiZpH9B0FERET6pk3VO3l+wXqmzStnxrJNLXM0Q2ie5tyUfgxIT2B0QRqnjMpv6VUuyAj1Mmu64f2LyIQaYEhOMjecOoKf/WcBj84q4/yJRUGHJCLSbmb2G+Av7j4/6FhEpOfZWL2T5+ZXMG1eOW8t30xjk1Oclchlx5QwNDeZgowECtMTyUvrF+h0c31FxCbUAJccVczz8yu4+ekFHDUki8KMxKBDEhFpr0XAPWYWA/wFeNjdtwUck4gEaMP2nTw7v4Jpc8t5e8UmmhwGZydx5XFDOP3g/ozsn6I38l0kohPqqCjjN+eN5ZTfTeeGx+byj0sP7xMjTUWk73P3+4D7zGw4cAkw18zeBO5191eCjU5Eukvl9jqe/aCCZ+aW887KzbjDkJwkrjphKKeP6c/wPCXR3SGiE2qAosxEfnjGQXz/yXn84+1VfPHI4qBDEhFpFzOLBkaEfzYC7wPXmdkV7n5hoMGJSJdZX1XHf+eVM+2DCt4NJ9Glucl868RSzhjTn9LcZCXR3SziE2qAiw4r4tn5Ffy/aYuYVJpDSXZS0CGJiHwsM/stcBbwEvBLd38nfOhXZrY4uMhEpCtUbKtj2rxy/vtBOTNXbcEdhuelcM3kYZx+cD6leSlBhxjRlFADZsavzxnDlNtf4zuPvs+/rzgy4qd/EZEe7wPgh+5e28axw7o7GBHpfOu27uC/H4QGFs5atQWAEfkpXHvSME4/uD9Dc5MDjlCaKaEOy0+L56dnj+Laf73Pfa8v54rjhgQdkojIx9kCtMxlZWbpwPHu/tT+Biea2anAHUA0cJ+737LX8duBE8KbiUCuu6eHj30J+GH42M/d/cHOeBgRCVm7dQf/nVfOM/PKmbN6KxBalO47U4Zx2sH9GZKjJLonUkLdyqfHFfDsBxXc9vyHnDAil2F6fSIiPddP3P3J5g1332pmPwGe+riLwnXXdwEnA2XAu2Y21d0XtLrXta3O/yYwPvw9E/gJMBFwYFb42i2d91gikWfN5lr++0E5z8yr4P01oSR61IBUrj9lOKcf3F+lqL2AEupWzIxffOZgptw+nev+/R5Pfv1oYqOjgg5LRKQtbTVO7WnTDwOWuvtyADN7BDgbWLCP8y8ilEQDnAK84O6bw9e+AJwKPPwJ4hYRQkn0M/PKmTavnLlloZdKBxek8d1TR3Da6HyKlUT3Kkqo95Kd3I9ffmY0X/vHbO56ZSnXnDQs6JBERNoyMzww8S5CvcXfBGa147oCYE2r7TLg8LZONLNBQAnw8sdcW9DGdZcDlwMMHDiwHSGJ9F119Y2s3FTD0spqllXWsGxDNYsrtrN4/XYAxhamceNpIzhtdH8GZmk9jN5KCXUbTh3dn0+PG8CdLy9l8og8Di5MCzokEZG9fRP4EfAvwIDngW+047q2Rlx7G/sALgQec/fGT3Ktu98D3AMwceLEfd1bpE/ZUrOLZRuqQ4nzhmqWbQgl0Wu21OLhvwVmUJiRwJCcZM6ZUMBpo/tTlKkkui9QQr0PPz1rNP9bvolvP/oeU686hvhYLcspIj2Hu9cA3zuAS8uAolbbhcC6fZx7IXsm6WXA8Xtd++oBxCDSKzU1OWu37mDphmqWNSfOlTUs3VDN5ppdLef1i4licE4yYwrT+Mz4AobmJjMkJ5nBOUnKJ/ooJdT7kJYYyy3njOGSv7zL7S9+yI2njQw6JBGRFmaWA9wAjALim/e7+4n7ufRdoNTMSoC1hJLmi9u4/3AgA/hfq93PAb80s4zw9hTgxgN9BpGeqq6+keUbaj7S47x8QzU7G5pazstMimNoTjKnjMpjSE4yQ3KTGZqTzID0BE2/G2GUUH+ME4bnctFhRdwzfTlTDspjwqDMoEMSEWn2EKFyjzOBrwFfAjbs7yJ3bzCzqwglx9HAA+4+38xuBma6+9TwqRcBj7i7t7p2s5n9jFBSDnBz8wBFkd7G3dlcs6ulNCOUNId+yrbs2KNMoygjkaG5yRwzNIshOckMzU1mcE4ymUlxwT6E9BjWqq3s3BubPUCooa9099FtHD8e+D9gRXjXE+5+8/7uO3HiRJ85c2Znhvqxqnc2cOrvphMTZUy7ehKJcfo3iIgcODOb5e4TO+k+E8xsrruPCe97zd2P63iUnae722yRj1O5vY4ZSzfxxtKNvLl0I+Xb6lqOxcdGMTg7uaU8Y0huEkNzkynOUplGJGtvm92V2eFfgTuBv33MOa+7+5ldGEOHJfeL4dfnjuHie9/m188u5qazRgUdkogIQH34s9zMziBUB10YYDwiPU71zgbeWbGJN5Zs4s2lG1tm1khPjOXoIdmMH5jekkAXpCcQpTINOUBdllC7+3QzK+6q+3eno4Zk8+WjivnrjJVMOSiPo4ZmBx2SiMjPzSwN+DbwByAVuPbjLxHp2+obm3h/zdaWHug5q7fS0OT0i4nisJJMPnNIAccMzeag/qlKnqVTBV2/cKSZvU+oZ+U77j6/rZN6wpym3z11BK99uIHrH5vLs9dMIiU+dv8XiYh0gfBqh6Xu/h9gG7uXCReJKO7Okspq3lgSSqDfWr6Jml2NmMGYgjQuP3YwxwzN5pBBGSrbkC4VZEI9Gxjk7tVmdjqh5XJL2zqxJ8xpmhAXzW/OG8t5d8/g5/9ZyK/OHRNEGCIiuHujmZ0F3B50LCLdrXzbDt5cGirheGPpRjZs3wlASXZSSw/0EYOzSE/UgEHpPoEl1O5e1er7NDP7o5llu/vGoGLanwmDMrjiuCH86dVlnDI6jxNH5AUdkohErhlmdiehmT5qmne6++zgQhLpfFV19by1bHcCvWxD6I97VlIcRw/N5pih2Rw1NIvCDC2QIsEJLKE2s3xgvbu7mR0GRAGbgoqnva45qZSXF1by3cfn8fw1GWRoyhwRCcZR4c/WsyM5sL95qEV6tJ0NjcxZvbUlgX5/zVaaHBJiozl8cCYXHTaQo4dmMzwvRXXQ0mN0WUJtZg8TWlEr28zKgJ8AsQDufjdwLnClmTUAO4ALvavm8OtE/WKiue38sXz6rjf5ydT5/P6i8UGHJCIRyN1VNy19QlOTs6hie0sC/c6KzeyobyQ6yhhbmMZVJwzl6KHZjB+YQVxMVNDhirSpK2f5uGg/x+8kNK1erzO6II1vTS7lty98yCmj8jljTP+gQxKRCGNmP25rf3vm8xcJQkNjE2VbdrBiUw0rNtSwclMNKzbWsGBdFZvCy3YPzU3mgkOLOHpoNocPziRVEwBILxH0LB+91pXHD+HFhev54VPzOKwkk5yUfkGHJCKRpabV93hCC2ktDCgWESDU21xRVceKjTUtPyvDn6s319LQtPtFdEq/GIqzkzhueA5HD8nm6KHZ5KfFBxi9yIFTQn2AYqOjuO28sZzxhze48Yl53PvFCZiplktEuoe739Z628x+A0zdx+kincbd2Vi9K9TDvKHmIz3OOxuaWs6Nj42iOCuJ4fkpnDI6n5KsJEpykijOSiI7OU7/35Q+Qwl1B5TmpXD9lOH8YtpCnpi9lnMmaJEyEQlMIjA46CCk79hWWx9KljdWs2Jj7R69zdU7G1rOi4kyBmYlUpKVxDFDsynOTmJwdhLF2Unkp8Zr4KBEBCXUHfSVY0p4fkEFNz09nyOHZDEgPSHokEQkApjZPEKzegBEAznsOeOHyH41NDaxclMtiyu2t0qcq1m5qZbN4bpmADMozEigOCuJcw4poDg7iZLwT0F6AjHRGiwokU0JdQdFRxm/OW8sp/7udb77+Fz+9pXD9ApLRLrDma2+NxCahrRhXyeLbK7ZxaKKKhaWb2dReRWLKrbz4frte5Ro5KfGU5ydyCmj8ijJDpVmDM5JoigzkX4xWmlQZF+UUHeCQVlJfP+MkfzoqQ946O3VfP6IQUGHJCJ9X39gvrtvBzCzZDMb5e5vBxyXBGxXQxPLN1azqHw7CyuqWFS+nUUVVayv2tlyTlZSHCP7p/LFIwcxIj+V4fkpDM5JIjFOaYHIgdDfnE7y+cMH8vz8Cn45bSGTSrMZlJUUdEgi0rf9CTik1XZtG/ukD3N3NlTvbEmYF5ZvZ2F5Fcs2VFPfGKoGiouOYmhuMkcPzWZkfioj+qcwIj9VM1OJdDIl1J3EzPjVOWM45fbpXP/oXB65/AgNxBCRrmStF8Ny9yYzU5veR9XVN7K0spqF4VKNReGe502t6pzzU+MZ0T+F44fnMrJ/CiP7p1KSnUSs6ptFupwa3040ID2BH3/qIK5/bC4PvLmCyyZpwL2IdJnlZvYtQr3SAF8HlgcYj3QCd6d8W93uWueKUK/zio01NIbncI6PjWJ4Xgonjcxr6XEekZ9CRlJcwNGLRC4l1J3s3AmFPDe/gl8/t5jjh+cwNDcl6JBEpG/6GvB74IeEZvt4Cbi8PRea2anAHYRmB7nP3W9p45zzgZvC937f3S8O728E5oVPW+3uZ3XsMQRgxcYafvHMAt5ZsZmqut1jSwszEhiRn8ppo/NDiXP/FIqzkojWG1CRHkUJdSczM3752YOZcvt0vv3v93n8yqM0nZCIdDp3rwQu/KTXmVk0cBdwMlAGvGtmU919QatzSoEbgaPdfYuZ5ba6xQ53H9ex6KVZY5PzwBsr+M3zi+kXE8WZYwcwsn8qI/NTGJafoqW3RXqJdmV6Zna1maVayP1mNtvMpnR1cL1Vbko8P//0aN4v28bdry0LOhwR6YPM7EEzS2+1nWFmD7Tj0sOApe6+3N13AY8AZ+91zleBu9x9C7Qk79LJlqzfzjl/msEvpi1kUmkOL1x3HL/8zMF84YhBTCzOVDIt0ou0t+v0K+5eBUwhtHjAJcBHXhHKbmeOGcCZY/pzx0tLeGv5pqDDEZG+Z4y7b23eCCe/49txXQGwptV2WXhfa8OAYWb2ppm9FS4RaRZvZjPD+z/d1i9gZpeHz5m5YcOG9j1NBKlvbOKuV5Zyxu/fYNWmGn5/0Xju/eIE8lLjgw5NRA5Qe0s+mou1Tgf+4u7vm1Yv2a+fnT2aOau3cuE9bzHloDy+PWU4w/NVUy0inSLKzDKae5HNLJP2telttd2+13YMUAocDxQCr5vZ6HACP9Dd15nZYOBlM5vn7nu8inP3e4B7ACZOnLj3vSPa/HXbuOGxucxfV8UZY/rz07NGkZ2sKexEerv2JtSzzOx5oAS40cxSgKb9XBPxMpLieO7aY3ngjRXcO305p94xnbPHDuCak4ZRnK15qkWkQ24DZpjZY+Ht84BftOO6MqCo1XYhsK6Nc95y93pghZktJpRgv+vu6wDcfbmZvUqoV1y1bfuxq6GJO19ewh9fXUZ6Yhx3f/4QTh3dP+iwRKSTtDehvhQYByx399pwT8glXRdW35HcL4ZvTS7li0cO4u7XlvPXGSt4em45508s5JsnljIgPSHoEEWkF3L3v5nZLOAEQr3On209sPBjvAuUmlkJsJbQwMaL9zrnKeAi4K9mlk2oBGS5mWUAte6+M7z/aODXnfNEfdf7a7Zyw2NzWbx+O58dX8CPzjxIU9yJ9DHtTaiPBN5z9xoz+zyhlbju6Lqw+p70xDi+d9oIvnJMMX98ZRkPvb2Kx2ev5fOHD+LrJwzRKz8R+cTcfb6ZbQDiAcxsoLuv3s81DWZ2FfAcoWnzHgjf52ZgprtPDR+bYmYLgEbgenffZGZHAX82syZCY3BuaWcSH5Hq6hu5/cUPuXf6cnJT4nngyxM5cURe0GGJSBewVgtt7fsks7nAWGAM8HfgfkK9Icd1bXgfNXHiRJ85c2Z3/7KdrmxLLb9/aQmPzSojPjaaS44u5vJJQ0hL1Khukb7MzGa5+8ROuM9ZhMo+BgCVwCBgobuP6ui9O1NfabM/qZkrN3PD43NZvqGGiw4r4sbTR2rWDpFeqL1tdntn+WgIL3F7NnCHu98BaHRdBxRmJPLrc8fy4nXHMXlkHne9soxJv36Zu15ZSs3Ohv3fQEQi3c+AI4AP3b0EmAy8GWxIUrurgZ8+PZ/z/vw/dtY38Y9LD+f/fXaMkmmRPq69CfV2M7sR+ALwTHhhALUOnWBwTjJ/uGg80741icNKMrn1ucUcd+sr3P/GCurqG4MOT0R6rnp330Roto8od3+F0FgXCciMZRs59Xev85c3V/KFIwbx/LXHckxpdtBhiUg3aG8N9QWEBq18xd0rzGwgcGvXhRV5DhqQyn1fOpTZq7fwm+cW87P/LOC+15fzrcmlnDuhkFittigie9pqZsnAdOAhM6sE9HorANvr6rnlv4t46O3VFGcl8q/Lj+DwwVlBhyUi3ahdNdQAZpYHHBrefCeolbMipR5vxtKN3Pr8Yuas3sqgrESuO3kYnxozgKgoTf8t0pt1Yg11ErCD0JvGzwFpwEPhXuseo6+32a8uruT7T8yjvKqOS48u4dtThpMQFx10WCLSSdrbZrerh9rMzifUI/0qoemZ/mBm17v7Yx97oRywo4Zm88SQLF5eVMmtzy3m6kfe44+vLOO6KcOYclAeWldHJLK5e034axPwYJCxRKJttfX87JkFPDarjKG5yTx+5VEcMjAj6LBEJCDtLfn4AXBoc6+0meUALwJKqLuQmTF5ZB4nDM/lmXnl3P7Ch1zx91mMLUzj21OGM6k0W4m1iEg3e2HBen7w5Dw21eziGycM4ZsnlhIfq15pkUjW3oQ6aq8Sj020f0CjdFBUlPGpsQM4bXQ+T8xZyx0vLuGLD7zD4SWZXH/KcCYWZwYdoohIn7e5Zhc3TZ3P1PfXMSI/hfu/dCgHF6YFHZaI9ADtTaifNbPngIfD2xcA07omJNmXmOgozp9YxNnjBvDIO2v4w8tLOffu/3H88By+M2U4owvUsItECjO7OjyF6cfuk45zd6bNq+DH//cBVXX1XHvSMK48fghxMepXEpGQTzIo8RxCy8waMN3dn+zKwPalrw9w+SRqdzXw4IxV3P3aMrbtqOf0g/O57uRhDM3VFOEiPVUnDkqc7e6H7LVvjruP7+i9O1Nvb7Mrt9fx46fm8+z8Cg4uSOPW88YwIj816LBEpJt06qBEAHd/HHi8Q1FJp0qMi+HK44fwuSMGct/rK7j/9eU8+0EFnx5fwLUnDaMoMzHoEEWkk5nZRYSmMS0xs6mtDqUQKseTTuDuPPXeWn769AJqdzXy3VNH8NVJJcRoClMRacPHJtRmth1oqwvbAHd3/TO9B0iNj+W6k4fx5aOKufu1ZTw4YyVT31vHBYcWcc6EQsYWphOt6fZE+ooZQDmQTWjp8WbbgbmBRNTHlG/bwQ+e/ICXF1VyyMB0fn3uWIbmJgcdloj0YB+bULu7agd6kcykOL5/+ki+cnQJd76yhH+9u4aH3l5NVlIcJ4zI5aSRuRxTmkNyv3a/mBCRHsbdVwGrgCPNbBBQ6u4vmlkCkEAosZYD9J+567jx8XnUNzXxozMP4stHFatDQkT2S5lVH5SfFs/PP30w35kynNc+3MBLCyt5fn4Fj80qIy46isMHZ3LSyDwmj8ylMENlISK9kZl9FbgcyASGAIXA3cDkIOPqzWYs3cg1j7zHmMI0fnv+OIqzk4IOSUR6iS5LqM3sAeBMoNLdR7dx3IA7gNOBWuDL7j67q+KJROmJcZw9roCzxxVQ39jErFVbeGnhel5aWMlPps7nJ1PnMzwvhckjc5k8Mo9xRSoNEelFvgEcBrwN4O5LzCw32JB6r+UbqrnyodmUZCfx4FcOIyU+NuiQRKQX6coe6r8CdwJ/28fx04DS8M/hwJ/Cn9IFYqOjOGJwFkcMzuIHZxzE8g3VvLyokhcXrufP05fzx1eXkZUUx/HDQ6Uhk4apNESkh9vp7ruaF3cysxjaHvMi+7Gttp7LHpxJdJTxwJcPVTItIp9Yl2VM7j7dzIo/5pSzgb95aN6+t8ws3cz6u3t5V8Ukuw3OSWZwTjKXTRrMttp6XluygZcWrufFhet5fHYZsdHGEYOzmDwi1HutGUNEepzXzOz7QIKZnQx8HXg64Jh6nfrGJr7+z1mUbdnBQ189XG2diByQILsgC4A1rbbLwvs+klCb2eWEagUZOHBgtwQXSdISYzlr7ADOGjuAhsYmZq7a0tJ7fdPTC7jp6QUMy0tm8sg8ThqZy7iiDJWGiATve8ClwDzgCkKLbd0XaES9jLtz09T5vLl0E785byyHatVZETlAQSbUbWVkbb6udPd7gHsgtEhAVwYV6WJalYZ8//SRrNhY01J3fe/05fzp1WVkJsVx/PAcThqZx6TSbL0eFQmAuzcB9wL3mlkmUOjtXKnLzE4lNIYlGrjP3W9p45zzgZsItcvvu/vF4f1fAn4YPu3n7v5gR58lKA/OWMlDb6/ma8cN4dwJhUGHIyK9WJAJdRlQ1Gq7EFgXUCyyDyXZSVw2aXCoNGRHPdM/3NCSYD8xey2x0cbhJVlMHpnLSSoNEek2ZvYqcBahdvw9YIOZvebu1+3numjgLuBkQu3wu2Y21d0XtDqnFLgRONrdtzQPdgwn7j8BJhJKtGeFr93S6Q/YxV5dXMnN/1nAlIPyuOGU4UGHIyK9XJAJ9VTgKjN7hNBgxG2qn+7Z0hJi+dTYAXwqXBoyq1VpyE+fXsBPn15AaW6oNOSE4TmMLUonPjY66LBF+qo0d68ys8uAv7j7T8ysPQu7HAYsdfflAOE2+GxgQatzvgrc1Zwou3tleP8pwAvuvjl87QvAqcDDnfJE3WTJ+u18859zGJ6fyu0XjCNKJWwi0kFdOW3ew8DxQLaZlRHq1YgFcPe7CdX7nQ4sJTRt3iVdFYt0vpjoKA4fnMXhg7O48fSRrNxYw0uLKnlp4Xrue305d7+2jJgoY2T/VA4ZmM74gRmMH5jOwMxEmmclEJEOiTGz/sD5wA8+wXVtjV/Ze4alYQBm9iahspCb3P3ZfVxb8AnjDtTmml1c+uBM+sVGc/+XJpKk2YxEpBN05SwfF+3nuBOaR1X6gOLsJC49poRLjylh24563lmxmffWbGH2qq08OquMB/+3Cgit5ji+KJ3x4SR7TGGaarBFDszNwHPAG+7+rpkNBpa047r2jF+JITSl6fGEyvFeN7PR7by2xw4k39XQxNf+PouKqjr+dfkRDEhPCDokEekj9E9z6XRpCbGcfFAeJx+UB0Bjk/Ph+u3MWb2VOau3MGfNVl5aFHqDbAbDclM4ZFA644tCvdhDcpL1ClZkP9z9UeDRVtvLgXPacWl7xq+UAW+5ez2wwswWE0qwywgl2a2vfbWN2HrcQHJ35wdPzuOdlZv5/UXjGT8wI+iQRKQPUUItXS46XPoxsn8qFx8e6q3atqOe99dsZc7qrcxevYVp8yp4+J3Qm+SUfjGMG5ge7snOYFxROhlJcUE+gkhf8i5QamYlwFrgQuDivc55CrgI+KuZZRMqAVkOLAN+aWbN2egUQoMXe7x7pi/n0VllXD25lLPGDgg6HBHpY5RQSyDSEmI5dlgOxw7LAaCpyVmxqWZ3L/bqrdz5ylKawn1bJdlJLWUi44vSGZGfQkx0VIBPINI7uXuDmV1FqFwkGnjA3eeb2c3ATHefGj42xcwWAI3A9e6+CcDMfkYoKQe4uXmAYk/2woL13PLsIs4Y05+rJ5cGHY6I9EHWzmlLe4yJEyf6zJkzgw5DukGxZrRjAAAQo0lEQVTNzgbmrd3WkmTPXr2VjdU7AUiIjebgwrRQkl2UwSED08lNjQ84YpH9M7NZ7j6xE+4T7e6NnRFTVwq6zV6wropz755BaW4y/7riSM08JCKfSHvbbPVQS4+V1C+mZZEZCNVAlm3ZwZw1u3uxH3hjBfWNywEoSE9g3MB0xhWmM2pAKgcNSCU9UaUi0metMLNngX8BL7d3UZdIUrm9jssefJfU+Fju/eJEJdMi0mWUUEuvYWYUZSZSlJnYUgNZV9/IgvKqPUpFnpm7ezrzgvQERg1IZdSAtNBnQSr5qfGauk/6guHApwjNlnS/mf0HeMTd3wg2rJ6hrr6RK/4+iy219Tz6tSP1BktEupQSaunV4mOjOWRgBocMzABKANhUvZMF5VXMX9f8s40XFq6nuf8uMymupQe7OdEuyUrSzCLSq7j7DuDfwL/DgwTvAF4jVBcd0dyd7z4+lzmrt3L35w9hdEFa0CGJSB+nhFr6nKzkfkwqzWFSaU7LvpqdDSyqCCfYa6uYX76Nv7yxkl2NTQAkxkUzsn9quDc7lGiX5iXTLybicxPpwczsOOAC4DRCAwXPDzainuHOl5fyf++t4/pThnPq6P5BhyMiEUAJtUSEpH4xTBiUyYRBmS37djU0sbSymvnrtjF/XRUL1lXxxOy1/C28CE1MlFGal7JHkj2yf4oWopEewcxWAO8R6qW+3t1rAg6pR5g2r5zbXviQz44v4OvHDwk6HBGJEEqoJWLFxURxULj047zwvqYmZ/Xm2pZSkfnrqnh18QYem1XWcl1xViKjBqSFS0ZCiXZOSr9gHkIi2Vh3rwo6iJ5kbtlWrvv3e0wYlMH/O+dgjZUQkW6jhFqklagoozg7ieLsJM4Ys/tVcWVV3R5J9ry123hm3u7Bj7kp/Rg1IJXSvBRKspNafnJT+ul/6tJV8s3sSSDP3Ueb2RjgLHf/edCBBaFiWx1f/dtMspL68ecvTFC5loh0KyXUIu2QmxpPbmo8J4zIbdlXVVfPglYDHxesq+LNZZvY1dDUck5SXDTF4eR6cHYSJTlJFGclMTg7mbRElY5Ih9wLXA/8GcDd55rZP4GIS6h37Grksr+9S3VdA49//Siyk/XGSES6lxJqkQOUGh+7xzzZECoZKa+qY8WGGlZsrGb5xhpWbKxh3tptTJtX3rLyI4RmG2ndm938U5yVREKcetdkvxLd/Z293oA0BBVMUJqanOv+/R7z11Vx/5cmMiI/NeiQRCQCKaEW6URRUUZBegIF6QkcU5q9x7FdDU2s3lzLynCSHUq2q3l9yZ412gAD0uJberNLspMYnJNESXYyhRkJxGrJdQnZaGZDAAcws3OB8o+/pO+5/cUP+e8HFfzwjJGcOCIv6HBEJEIpoRbpJnExUQzNTWZobvJHjtXsbGBFONFunXA//f46qup2dzrGRBkDMxNbykiaS0mKs5PIT43XXNqR5RvAPcAIM1sLrAA+H2xI3eupOWv5w8tLufDQIi49piTocEQkgimhFukBkvrFMLog7SMLULg7W2rrQ+UjG2pYuSmcbG+oYcayjdTV767Xjo0O9Y4XZSZSmJFAYUZiy/eijESyk+M0QLIPcfflwElmlgREufv2oGPqTrNWbeGGx+dyxOBMbj57tP5si0iglFCL9GBmRmZSHJlJe86hDaHa0Yqqupae7TVbainbsoOyzbU8t66KzTW79jg/PjYqlGS3JNvhz/D3tIRYJSW9gJldt4/9ALj7b7s1oACUbanlir/PZEBaPH/63ATiYlQGJSLBUkIt0ktFRRkD0hMYkJ7A0UOzP3K8ZmdDKMHeUsuazaFke82WWtZs3sGsVVv2KCUBSO4XE+rNbtWr3bxdlJlIcj81Fz1ESvhzOHAoMDW8/SlgeiARdaPqnQ1c9uBMdjY08cjlh5KRFBd0SCIiSqhF+qqkfjEMz09heH5Km8e37agPJ9uhpLtsyw7WbK5l9aZa3ly6kdpdjXucn54Yu2eSHe7pLsxIID8tXitIdhN3/ymAmT0PHNJc6mFmNwGPBhhal2tscq5+eA5LKqv56yWHtjkeQUQkCEqoRSJUWkIsaQlpjBqQ9pFjzbXbazbXtpSSNPdyL16/nZcWVe4x3zaEerjz0+LpnxZPfmo8+WnxrbYT6J8WT3qiyko60UCgdV3PLqA4mFC6x6+eXcRLiyr52dmjmFSaE3Q4IiItlFCLyEfsrt2OY2xR+keONzU5G6t3tiTb66vqKN9WR8W20OeS9Rup3F63x7zbAP1iokKJdmo40U5LCH/u3peV3I9ozVbSHn8H3gmvlujAZ4AHgw2p6/z73TXcM305XzxyEF84sjjocERE9qCEWkQ+sagoa1k9csKgts9paGxiY/UuyrftaEm0K6pCSXfFtjpmrd7C+m0V7Grcs6c7JsrIa9XDvTv53p2E56b0i/j5uN39F2b2X2BSeNcl7j4nyJi6ylvLN/GDp+YxqTSbH595UNDhiIh8hBJqEekSMdFRLUnxvjQ1OZtrd7Uk2eVVdVRs29HS271wXRUvL6xkR/2e9dxmkJ3cjwHpCRS11HSHZispykhkQHpCRMz84O6zgdmf9DozOxW4A4gG7nP3W/Y6/mXgVmBteNed7n5f+FgjMC+8f7W7n3Vg0bfPqk01XPmPWQzMTOTOiw8hJsL/ISUiPZMSahEJTFSUkZ3cj+zkfh+Zg7uZu1NV1xDu5W7V272tjrVbdzBv7Taem19BfePu+hIzyE+NDw2iDCfZzQMpizITyUuNj9iyEjOLBu4CTgbKgHfNbKq7L9jr1H+5+1Vt3GKHu4/r6jgBqurqufTBmThw/5cOJS1BA19FpGdSQi0iPZqZhQdQxu5zxpLG8JzcZZtrWRMeQLlmSy1lm3fwv2WbeLJqLd6qnrt5EZw95uNulXBnJfXpRXAOA5aGF4bBzB4Bzgb2TqgD1dDYxFX/nMPKjTX847LDKc5OCjokEZF9UkItIr1edFQoQS5IT+DwNo7vbGhk3da6lkS7earANVt28Pz89WzaaxGchNjolvKR3XNzh0tKMhNJ7d1TBBYAa1ptl0Gb/9nOMbNjgQ+Ba929+Zp4M5sJNAC3uPtTe19oZpcDlwMMHDjwgIL8+TMLmf7hBn51zsEcMTjrgO4hItJdlFCLSJ/XLyaakuwkSvbRy9m8CE7rhHtNeEGct1dspnrnnovgpCXEcvXkUr5yTEl3hN/Z2up632s+Fp4GHnb3nWb2NUKzh5wYPjbQ3deZ2WDgZTOb5+7L9riZ+z3APQATJ07c+9779Y+3VvHXGSv56qQSLjj0wBJyEZHupIRaRCLexy2C4+5s21G/R5K9ZkvtPpPzXqAMKGq1XQisa32Cu29qtXkv8KtWx9aFP5eb2avAeGCPhLqjRvZP5bwJhXzvtJGdeVsRkS6jhFpE5GOYGemJcaQnxnFwYdsDJ3uZd4FSMyshNIvHhcDFrU8ws/7uXh7ePAtYGN6fAdSGe66zgaOBX3d2gBMGZTBhUEZn31ZEpMsooRYRiSDu3mBmVwHPEZo27wF3n29mNwMz3X0q8C0zO4tQnfRm4Mvhy0cCfzazJiCKUA11jxrMKCISBCXUIiIRxt2nAdP22vfjVt9vBG5s47oZwMFdHqCISC+jGfJFRERERDrA3D/xAOxAmdkGYNUBXJoNbOzkcHqDSHxuPXPk6I3PPcjdc4IOoruozf7EIvG5I/GZITKfuzc+c7va7F6XUB8oM5vp7hODjqO7ReJz65kjR6Q+dySI1N/bSHzuSHxmiMzn7svPrJIPEREREZEOUEItIiIiItIBkZRQ3xN0AAGJxOfWM0eOSH3uSBCpv7eR+NyR+MwQmc/dZ585YmqoRURERES6QiT1UIuIiIiIdLqISKjN7FQzW2xmS83se0HH09XMrMjMXjGzhWY238yuDjqm7mJm0WY2x8z+E3Qs3cXM0s3sMTNbFP49PzLomLqamV0b/rP9gZk9bGbxQccknUdtduS02RB57XYkttnQ99vtPp9Qm1k0cBdwGnAQcJGZHRRsVF2uAfi2u48EjgC+EQHP3OxqYGHQQXSzO4Bn3X0EMJY+/vxmVgB8C5jo7qMJLZ99YbBRSWdRmx1xbTZEXrsdUW02REa73ecTauAwYKm7L3f3XcAjwNkBx9Sl3L3c3WeHv28n9Je1INioup6ZFQJnAPcFHUt3MbNU4FjgfgB33+XuW4ONqlvEAAlmFgMkAusCjkc6j9rsCGmzIfLa7Qhus6GPt9uRkFAXAGtabZcRIQ0VgJkVA+OBt4ONpFv8DrgBaAo6kG40GNgA/CX8yvQ+M0sKOqiu5O5rgd8Aq4FyYJu7Px9sVNKJ1GZHTpsNkdduR1ybDZHRbkdCQm1t7IuIqU3MLBl4HLjG3auCjqcrmdmZQKW7zwo6lm4WAxwC/MndxwM1QJ+uOTWzDEI9liXAACDJzD4fbFTSidRmR0CbDRHbbkdcmw2R0W5HQkJdBhS12i6kj71maIuZxRJqmB9y9yeCjqcbHA2cZWYrCb0iPtHM/hFsSN2iDChz9+berMcINdZ92UnACnff4O71wBPAUQHHJJ1HbXZktNkQme12JLbZEAHtdiQk1O8CpWZWYmZxhIrgpwYcU5cyMyNUn7XQ3X8bdDzdwd1vdPdCdy8m9Hv8srv3qX/9tsXdK4A1ZjY8vGsysCDAkLrDauAIM0sM/1mfTAQM6okgarMjRCS22xHaZkMEtNsxQQfQ1dy9wcyuAp4jNKr0AXefH3BYXe1o4AvAPDN7L7zv++4+LcCYpOt8E3gonHwsBy4JOJ4u5e5vm9ljwGxCsyPMoQ+vvhVp1GarzY4AEdVmQ2S021opUURERESkAyKh5ENEREREpMsooRYRERER6QAl1CIiIiIiHaCEWkRERESkA5RQi4iIiIh0gBJq6dXMbEb4s9jMLu7ke3+/rV9LREQOjNps6as0bZ70CWZ2PPAddz/zE1wT7e6NH3O82t2TOyM+ERHZTW229DXqoZZezcyqw19vASaZ2Xtmdq2ZRZvZrWb2rpnNNbMrwucfb2avmNk/gXnhfU+Z2Swzm29ml4f33QIkhO/3UOtfy0JuNbMPzGyemV3Q6t6vmtljZrbIzB4KrwglIiKozZa+q8+vlCgR43u06u0IN7Lb3P1QM+sHvGlmz4fPPQwY7e4rwttfcffNZpYAvGtmj7v798zsKncf18av9VlgHDAWyA5fMz18bDwwClgHvEloBbQ3Ov9xRUR6NbXZ0qeoh1r6qinAF8PL+L4NZAGl4WPvtGqYAb5lZu8DbwFFrc7bl2OAh9290d3XA68Bh7a6d5m7NwHvAcWd8jQiIn2b2mzp1dRDLX2VAd909+f22Bmq26vZa/sk4Eh3rzWzV4H4dtx7X3a2+t6I/o6JiLSH2mzp1dRDLX3FdiCl1fZzwJVmFgtgZsPMLKmN69KALeGGeQRwRKtj9c3X72U6cEG45i8HOBZ4p1OeQkQkMqjNlj5F/xKTvmIu0BB+DfhX4A5Cr+5mhweZbAA+3cZ1zwJfM7O5wGJCrxCb3QPMNbPZ7v65VvufBI4E3gccuMHdK8KNu4iI7J/abOlTNG2eiIiIiEgHqORDRERERKQDlFCLiIiIiHSAEmoRERERkQ5QQi0iIiIi0gFKqEVEREREOkAJtYiIiIhIByihFhERERHpACXUIiIiIiId8P8BNgRAUBJftKUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111a94518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "reload(bilstm);\n",
    "bilstm.plot_results(losses, accuracies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 5.3**: (*0.5 points*)\n",
    "- As you can see from above, BiLSTM model performs worse than the Viterbi tagger.\n",
    "- Use pretrained embeddings like the following to improve your performance.\n",
    "- Check the following sources: [fastText](https://github.com/facebookresearch/fastText), [Polyglot](polyglot.readthedocs.io/en/latest/Embeddings.html), [word2vec](https://code.google.com/archive/p/word2vec/), [Glove](https://nlp.stanford.edu/projects/glove/)\n",
    "- Tune the hyperparameters such that you obtain atleast `85.00%` accuracy on the dev set.\n",
    "- You can try changing the no. of iterations, optimizers, no. of LSTM layers, the hidden dimension units, the FC layer dimensions, or use pretrained word embeddings like word2vec, fastText, polyglot, character features etc..\n",
    "- **Tests**: ```test_bilstm.py: test_bilstm_dev_accuracy(), test_bilstm_test_accuracy()```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Remember: After training a model once, everytime when you rerun the train_model(...) function, the model is retrained on top of the previous parameters. If you want to start afresh, make sure the model is reinitialized and then trained.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: pretrained embeddings\n",
    "\n",
    "- *Polyglot* provides pre-trained word embeddings for many languages\n",
    "- You can download the english polyglot embeddings [here](https://sites.google.com/site/rmyeid/projects/polyglot).\n",
    "- Use the following helper function to load in the polyglot embeddings, which you can use to initialize your parameter embeddings in your model. You need to make sure your embedding_dim matches your embeddings size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 440,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(bilstm);\n",
    "filename = 'data/polyglot-en.pkl'\n",
    "word_embeddings = bilstm.obtain_polyglot_embeddings(filename, word_to_ix)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let us have a look at these word embeddings. We can observe the cosine similarity between two word vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 441,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cosine(emb1, emb2): #function to return the cosine similarity between the embeddings\n",
    "    return emb1.dot(emb2) / (np.linalg.norm(emb1) * np.linalg.norm(emb2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 442,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.690198\n",
      "0.05310621\n",
      "0.08789002\n"
     ]
    }
   ],
   "source": [
    "diremb = word_embeddings[word_to_ix['dire']]\n",
    "catemb = word_embeddings[word_to_ix['catastrophic']]\n",
    "amazemb = word_embeddings[word_to_ix['success']]\n",
    "print(cosine(diremb,catemb)) # dire and catastrophic are similar\n",
    "print(cosine(diremb,amazemb))\n",
    "print(cosine(catemb,amazemb))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now, all we need to do is send in the word_embeddings when initializing your model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Once deleted, variables cannot be recovered. Proceed (y/[n])?  y\n",
      "Once deleted, variables cannot be recovered. Proceed (y/[n])?  y\n"
     ]
    }
   ],
   "source": [
    "reload(bilstm);\n",
    "torch.manual_seed(765);\n",
    "embedding_dim=64\n",
    "hidden_dim=30\n",
    "model = None\n",
    "loss = None\n",
    "del model\n",
    "del loss\n",
    "%reset_selective model\n",
    "%reset_selective loss\n",
    "model = bilstm.BiLSTM(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim, word_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1: Dev Accuracy: 0.7752734941347041\n",
      "Epoch 2: Dev Accuracy: 0.8104652695400026\n",
      "Epoch 3: Dev Accuracy: 0.8248319493871096\n",
      "Epoch 4: Dev Accuracy: 0.8368261499934098\n",
      "Epoch 5: Dev Accuracy: 0.8473705021747726\n",
      "Epoch 6: Dev Accuracy: 0.8530380914722552\n",
      "Epoch 7: Dev Accuracy: 0.8515882430473177\n",
      "Epoch 8: Dev Accuracy: 0.85593778832213\n",
      "Epoch 9: Dev Accuracy: 0.8631870304468169\n",
      "Epoch 10: Dev Accuracy: 0.8589692895742718\n"
     ]
    }
   ],
   "source": [
    "loss = torch.nn.CrossEntropyLoss()\n",
    "model, losses, accuracies = bilstm.train_model(loss, model, X_tr,Y_tr, word_to_ix, tag_to_ix, \n",
    "                                        X_dv, Y_dv, num_its=10, status_frequency=1, \n",
    "                                        optim_args = {'lr':0.1,'momentum':0}, param_file = 'best.params')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAACdCAYAAACKLN/OAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl4VeW59/HvnXkkgQwICZBEEFEQlQCCOA+1tg61g2OrVmtH23ra02PP6WBtzzmdrbV92zqgHWwVbfXY1lZtwQG1QEARQUFIkAQQwpwBMt7vH2snbmIgEbKzkuzf57r2tfcanrXvdUGe3HnWM5i7IyIiIiIifSsh7ABERERERIYiJdoiIiIiIjGgRFtEREREJAaUaIuIiIiIxIASbRERERGRGFCiLSIiIiISA0q0RURERERiQIm2iIiIiEgMKNEWEREREYmBpLAD6Cv5+fleUlISdhgiIodk6dKl29y9IOw4+pPqbREZrHpbZw+ZRLukpISKioqwwxAROSRm9mbYMfQ31dsiMlj1ts5W1xERERERkRhQoi0iIiIiEgMx6zpiZnOB9wNb3X1yN8cNuB04H2gErnH3ZZFjVwNfi5z6HXf/dSxi/PPyTdy/6E3uv/4kEhMsFl8hIiIiIl20tLXzcvUuFr6xjZWb9jAqJ40jCzIpK8jiyMIsRg1LI2EI5Gax7KN9H/Az4DcHOP5eYELkNRP4BTDTzEYA3wTKAQeWmtlj7r6zrwNMSjD+VbmDp1Zt4bzJR/T15UVEBhUzO4+gASQRuNvdv9vl+Fjg10Bu5Jyb3f3xyLHjgF8Bw4B2YLq77+vH8EVkAHN31mypZ+HabTy/dhuLKrfT0NxGgkFpfiaLqrZTt6+18/y05ATK8rMoK8jkyIL93zNSBs8Qw5hF6u7PmlnJQU65CPiNuzvwLzPLNbNRwOnAU+6+A8DMngLOA/7Q1zGee+wRjBmRzj0LK5Voi0hcM7NE4OfAOUANsCTSyLEq6rSvAfPc/RdmdgzwOFBiZknA74CPuvtyM8sDWvr5FkRkgNm0a29nYv382u1sq28CoCw/k0tOLObk8fnMKssjJyMZd2dbfTPrauuprG2IvNfzSs1u/rpiM+5vX3d0TlrQ8t3RAh5JwEflpBF0mBg4wvyToAiojtquiew70P4+l5hgXDu7lFv/soqXq3dx/JjcWHyNiMhgMANY6+6VAGb2AEGDSHSi7QQt1gA5wKbI53OBV9x9OYC7b++XiEVkQNnd2MKLldsjifU2Krc1AJCflcqc8XnMHp/PyePzKcpNf0dZM6MgO5WC7FROKsvb79i+ljbe3N7YmXyvq22gsraePy7bSH3T263g6cmJlHUm31Hv+VmkpyTG9uYPIMxEu7s/Ofwg+995AbMbgBsAxo4de0hBfGT6GG57ag13P1fJz6448ZCuISIyBHTXyDGzyzm3AE+a2Y1AJnB2ZP9RgJvZE0AB8IC7fz+24YpI2Pa1tLHszZ1Bq/W67ayo2UW7Q0ZKIieV5XHlSeOYMz6fo0ZmHVZLc1pyIhOPyGbiEdn77Xd3auuaWLtfK3gDL23YyV9e2bRfK3hRbnq33VCOGBbbVvAwE+0aYEzUdjFB60gNQfeR6P1Pd3cBd78TuBOgvLy822S8J1mpSVw+cyz3LKyiZmcjxcMzDuUyIiKDXW8aOS4H7nP3H5nZLOC3ZjaZ4HfJHGA6weD2f5rZUnf/5zu+pA8aSEQkHO3tzqrNezq7gyyu2kFTazuJCcYJY3K58cwJzJmQz9TiXFKSYj+xnZlROCyNwmFpzD4yf79j+1raqNrWsF83lHW1DTxUUU1Dc1vnedlpSSz7+jkkJ8Ym3jAT7ceAz0UeT84Edrv75kiLyP+Y2fDIeecCX41lIFfPLuGehVX8+oX1/Nf7jonlV4mIDFQHavyIdh3BmBnc/UUzSwPyI2WfcfdtAGb2OHAi8I5Euy8aSESkf7g7G3Y0dibWL6zbzq7GYPjFxJHZXDlzHHMm5DGjNI+s1IE1QDEtOZFJo4YxadSw/fa7O1v2NEUS73q21TfHLMmG2E7v9weClul8M6shmEkkGcDdf0kwiOZ8YC1BC8i1kWM7zOzbwJLIpW7tGBgZK0W56Zw/ZRQPLK7m82dNIDstOZZfJyIyEC0BJphZKbARuAy4oss5G4CzgPvMbBKQBtQCTwBfMbMMoBk4DbitvwIXkb6zvb6JF9YF/awXrt1Gzc69AIzKSePsSSOZMz6f2UfmUTgsLeRID42ZcUROGkfkpDF7fH7PBQ5TLGcdubyH4w589gDH5gJzYxHXgVw/p5Q/L9/EvIoarptT2p9fLSISOndvNbPPESTNicBcd19pZrcCFe7+GPAl4C4zu4mgW8k1kbp8p5n9mCBZd+Bxd/9rOHciIr21oyGY5WPt1nrWbKljUeUOVm3eAwRdKmYfmccnTy1j9vh8yvIzB9yMHoPBwGrnD9HUMbnMKBnBvc9XcfWscSTF8DGCiMhAFJkT+/Eu+74R9XkVcPIByv6OYIo/ERlA2tudjbv2dibU62rrWbe1gbW19exoaO48Ly05gRPGDOff3zORk8fnM3n0MOVCfUCJdpTrTinlk79dypOrtnD+lFFhhyMickjM7IfAve6+MuxYRKR/NLe2s357A2u3vp1Qr90azMKxt+XtwX/DM5IZX5jFuceMZHxhsArj+IIsinLTh8RKjAONEu0oZ08aybi8DO56rlKJtogMZq8Dd0YWkrkX+IO77w45JpEBr7WtnTVb6tm0ay8ZqYlkpSYFr7TgPT05MfTuE3v2tbCuM5lu6EyqN+xopK397fHFRbnpjC/MYmZpHuMLs4KkuiCTvKzUEKOPP0q0oyQmGB8/uZRvPraSpW/uZNq44T0XEhEZYNz9buBuM5tIMND8FTN7HrjL3ReEG53IwNDe7qzf3sArNbtZXrOLV2p2s3LTbva1tB+wTIJBZmoS2alJZEYl4B2vzNQkstMix6Jfae88JzUp4YBJu7uzta5pv5bpjtfWuqbO85ITjdL8TI4+Ipv3HzcqkkwPvmXKhzL9K3TxoWnF/OjJ1dyzsJJp46aFHY6IyCGJLKl+dOS1DVgO/JuZfdLdLws1OJF+5u5s3r2PV2p2sbxmN69EEuu6fcGqgmnJCUwencMVM8YxdUwO4/Iy2dvcRn1TKw1NrdRF3uv3tVLfFHnta6WhuZW6fa28tXvf2/ubWvdbKOVAEhPsHcl4ZmoSu/e2ULm1nrqoFQ+zU5M4sjCLUyYU7Nc6PXZEhvpRD3BKtLvITE3iipnjuPPZdVTvaGTMCC1gIyKDS2QGkAsJ5rH+H3dfHDn0PTNbHV5kIv1jZ0NzZyt1R3JdG2kJTkowjh6VzQVTRzO1OIfjinOZUJjVZwmru9PY3PaOBL3zc3SiHjmnI2nf3dhMZmoSHzixKEioC4I+1IXZqaF3WZFDo0S7G1fPHsfdz1Vy7/Pr+cYFWsBGRAadV4GvuXtjN8dm9HcwIrFU39TKqxt379daXb0jmPvZDI4syOKUyGqFxxXnMGnUMNKSE2MWj5mRGekiUhizb5HBQol2N0blpHPB1NE8uGQDXzxnAsO0gI2IDC47iSwQBmBmucDp7v6oBkXKYNbU2sZrm+uCpLo6SKrX1tZ3dtUoHp7O1OJcrpo5juOKc5lcNEyL0EmolGgfwHVzSnnkpY08uLiaT5xaFnY4IiLvxjfd/ZGODXffZWbfBB4NMSaRd6Wt3Vm7tZ7l1bs6u4G8/tYeWtqCrDo/K4XjinN533GjOlurNaOGDDRKtA9gclEOJ5UFC9hcc3IJyRpsICKDR3cVlup7GfDcnaVv7uTBJdX87dW3qI8MCMxOTWJKcQ7XzSkL+lWPyWV0Tpr6LcuAp4r3IK6fU8b1v6ngb6++xYVTR4cdjohIb1VEBkT+nGBJ9BuBpeGGJHJgW+v28adlG5lXUU1lbQNZqUmcP+UIZh2Zx3HFuZTmZWoxFRmUlGgfxJlHF1Kan8ndz1VywXGj9JeziAwWNwJfBx4EDHgS+GyoEYl00dLWztOra3lwSTULVm+lrd2ZUTKCz5w+nvOnHKF5oGVI0P/ig0hIMD4+p5SvP/oqFW/uZHrJiLBDEhHpkbs3ADeHHYdId9bV1jOvopo/LdtIbV0TBdmpfOKUMj5SXkxZQVbY4Yn0KSXaPfjQicECNnc/V6lEW0QGBTMrAL4CHAukdex39zNDC0riWkNTK39dsZl5S6qpeHMniQnGmUcXcmn5GE6fWKBFV2TIUqLdg/SURK6aOY6fP72WN7c3MC4vM+yQRER6cj9Bt5H3A58CrgZqQ41I4o67s2zDLh6qqObPyzfR0NxGWUEmX33v0XzgxCIKs9N6vojIIKdEuxc+Nmscv3p2Hfc+v55bLjw27HBERHqS5+73mNkX3P0Z4Bkze6anQmZ2HnA7kAjc7e7f7XJ8LPBrIDdyzs3u/niX46uAW9z9h313OzKYbKtv4pFlG3mwopq1W+vJSEnkfVNGcen0MUwbN1zjnSSuKNHuhcJhaVw4tYh5FdXcdPZR5GRo8nsRGdBaIu+bzex9wCag+GAFzCyRYJaSc4AaYImZPebuq6JO+xowz91/YWbHAI8DJVHHbwP+1je3IINJa1s7z74RDGz852tbaW13Thyby/c+OIX3HTearFSlGxKf9D+/l66bU8ofl9Xw+8Ub+PTpR4YdjojIwXzHzHKALwF3AMOAm3ooMwNY6+6VAGb2AHARQQt1B49cCyCHIIEncv7FQCXQ0Bc3IIPD+m0NzKuo5o/Latiyp4n8rBQ+PqeUD08rZsLI7LDDEwldTBPtXjyGvA04I7KZARS6e27kWBuwInJsg7tfGMtYe3LM6GGcPD6P+16o4ro5paQkaeCGiAw8kZbpCe7+F2A3b9exPSkCqqO2a4CZXc65BXjSzG4EMoGzI9+ZCfwHQWv4l3uI7wbgBoCxY8f2MjQZSPY2t/H4is3Mq6hmUdUOEgxOn1jIty4cw1mTCrXAm0iUmCXavXkM6e43RZ1/I3BC1CX2uvvxsYrvUFw/p4xr71vC4ys2c/EJRWGHIyLyDu7eZmYXEnTjeDe66zjrXbYvB+5z9x+Z2Szgt2Y2GfgWcJu71/fU/9bd7wTuBCgvL+96fRmg3J3lNbuZV1HNn1/eRF1TKyV5Gfz7eybyoWnFjBymgY0i3Ylli3ZvHkNGuxz4ZgzjOWynHVXAkQWZ3L2wkouOH60BHSIyUL1gZj8jmHmksyuHuy87SJkaYEzUdjFRXUMirgPOi1zrRTNLA/IJWr4/ZGbfJxgo2W5m+9z9Z4d9JxKqHQ3NPPLSRuYtqWb1ljrSkhM4f8ooLi0fw4zSEfo9KNKDWCbavXkMCYCZjQNKgflRu9PMrAJoBb7r7o92U65fH0EmJBjXzSnjPx9ZwaKqHZxUlhfz7xQROQSzI++3Ru1z4GDzaC8BJphZKbARuAy4oss5G4CzgPvMbBLBHN217n5KxwlmdgtQryR7cHtt8x7mLqzi/5Zvorm1naljcvnvD0zmgqmjGZamCQFEeiuWiXZvHkN2uAx42N3bovaNdfdNZlYGzDezFe6+br+LhfAI8pITi/jhk6u5+7kqJdoiMiC5e2/7ZUeXaTWzzwFPEIyrmevuK83sVqDC3R8jGFx5l5ndRFCfX+Pu6v4xRLS3O/Nf38rc56t4Yd120pMT+Uh5MVedNI6jjxjW8wVE5B1imWj35jFkh8uAz0bvcPdNkfdKM3uaoP/2uncW7V9pyYlcddI47pj/BpW19VouVkQGHDP7Rnf73f3W7vZHHX+cYMq+6H3fiPq8Cji5h2vc0utAZUBoaGrl4aU13Pt8Feu3NzIqJ43/OO9oLp8xhtyMlLDDExnUYplo9+YxJGY2ERgOvBi1bzjQ6O5NZpZPULF/P4axvisfPWkcv3x6HXOfr+I7F08JOxwRka6ip9hLI1gh8rWQYpEBqmZnI79+YT0PLKmmbl8rx4/J5Y5zJ3Le5CM0c4hIH4lZot3Lx5AQDIJ8oMvjx0nAr8ysHUgg6KN9oEGU/a4gO5WLTxjNw0tr+NI5Exmeqb/4RWTgcPcfRW+b2Q+Bxw5wusQRd2fpmzuZ+3wVf3/1LcyM904+go/PKeXEscPDDk9kyInpPNo9PYaMbN/STbkXgAHdVHzdnDLmVQQL2Hz2jPFhhyMicjAZQFnYQUh4mlvb+durm7lnYRWv1OxmWFoSnzi1jKtnlTA6Nz3s8ESGLK0MeYgmHpHNKRPyue+F9Vx/SimpSYlhhyQiAoCZreDtweeJQAH7z0AicWJnQzO/X7yB37y4ni17mijLz+TbF0/mgycWkZGiFEAk1vRTdhiuP6WMq+cu5i/LN/PBacVhhyMi0uH9UZ9bgS3u3hpWMNL/3thSx9zn1/OnZTU0tbZzyoR8vnvJcZx2VAEJCZr7WqS/KNE+DKdOyOeokVncvbCKS04s0sT9IjJQjAJWunsdgJllmdmx7r4o5LgkhtrbnWffqOWehVU898Y2UpISuOSEIq49uZSJR2SHHZ5IXFKifRjMjOvnlPGVP77Ci+u2M3t8ftghiYgA/AI4MWq7sZt9MkTsbW7jj8uC6fnW1TZQkJ3Kl845iitmjiUvKzXs8ETimhLtw3Th8aP5/hOvc9dzlUq0RWSgsOiZnNy93cxU3w8xm3fv5TcvvsnvF21g994WJhcN47ZLp/K+KaNJSdL0fCIDgSrew5SWnMhHTyrhtn+sYe3WOsYX6vGciISu0sw+T9CKDfAZoDLEeKQPvVy9i3sWVvH4is24O+ceE0zPN71kuLowigwwSrT7wFUnjeXnT6/lnoXr+d9LBvSshCISHz4F/BT4GsHsI/8Ebgg1IjksrW3t/H3lW8xdWMWyDbvISk3imtklXDO7hDEjMsIOT0QOQIl2H8jLSuWDJxbxp2U1fPnco9QnTkRC5e5bCVbjlUFuX0sb9y/awD3PVbJp9z7Gjsjgmxccw4emFZOdlhx2eCLSg1514jKzL5jZMAvcY2bLzOzcWAc3mFw3p5Sm1nbuX7Qh7FBEJM6Z2a/NLDdqe7iZzQ0zJnl3WtvaeWDxBs744dN8+y+rKB6RwZ0fncaCL5/OtSeXKskWGSR626L9cXe/3czeQ7DwwbXAvcCTMYtskBlfmM0ZEwv4zYvrueHUMtKStYCNiITmOHff1bHh7jvN7IQwA5LeaW93/rJiM7c9tYaqbQ1MHZPLjz48VYPtRQap3g5L7hhdcT5wr7svj9onEdefUsa2+mYeW74p7FBEJL4lmNnwjg0zG0EvGlbM7DwzW21ma83s5m6OjzWzBWb2kpm9YmbnR/afY2ZLzWxF5P3MPr2bOODuLHh9K++/YyGf/8NLJCcad350Go9+ZraSbJFBrLct2kvN7EmgFPiqmWUD7bELa3CafWQeRx+RzT3PVfHhacUa/S0iYfkR8IKZPRzZ/jDw3wcrYGaJwM+Bc4AaYImZPebuq6JO+xowz91/YWbHAI8DJcA24AJ332Rmk4EngKK+vKGhbHHVDn7wxOssWb+TMSPSue3SqVw4tYhEreAoMuj1NtG+DjgeqHT3xkjryLWxC2twMjOuP6WMLz+0nOfe2MapRxWEHZKIxCF3/42ZLQXOIHj6eEmXhLk7M4C17l4JYGYPABcB0eUcGBb5nANsinzfS1HnrATSzCzV3ZsO+2aGsFc37uYHT6zmmTW1FGan8u2LJ3Np+RjNgS0yhPQ20Z4FvOzuDWZ2FcHqYrfHLqzB64Kpo/je31/n7oVVSrRFJDTuvtLMaoE0CLp9uPvBRmsXAdVR2zXAzC7n3AI8aWY3ApnA2d1c54PAS0qyD2xdbT0/fnINf12xmZz0ZG5+79FcPauE9BSN7REZanr7Z/MvgEYzmwp8BXgT+E3MohrEUpMSuXrWOJ5dU8vqt+rCDkdE4pCZXWhmbwBVwDPAeuBvPRXrZp932b4cuM/diwnG7PzWzDp/j5jZscD3gE8eJLYbzKzCzCpqa2t7vJehZOOuvXzl4eWc8+NnWLB6KzeeOZ7n/uMMPnXakUqyRYao3ibarZHlfC8Cbnf32wEtgXgAV8wcR1pyAnMXVoUdiojEp28DJwFr3L0UOAt4vocyNcCYqO1iIl1DolwHzANw9xcJWsvzAcysGHgE+Ji7rzvQl7j7ne5e7u7lBQXx8dRvW30T3/rzSs74wdM8+tImrp5dwrNfOYMvnTuRYZqmT2RI623XkToz+yrwUeCUyKAZ1Q4HMCIzhQ+eWMxDS2v48nsmUpCtBWxEpF+1uPt2M0swswR3X2Bm3+uhzBJggpmVAhsJFry5oss5GwiS9vvMbBJBol0bmbP7r8BX3b2nhD5u7NnXwl3PVnLPwir2tbTxoWnFfOHsoyjKTQ87NBHpJ71t0b4UaCKYT/stgr58P+ipUC+mirrGzGrN7OXI6/qoY1eb2RuR19W9jHPAuG5OKc2t7fzuX2+GHYqIxJ9dZpYFPAvcb2a3A60HK+DurcDnCGYMeY1gdpGVZnarmV0YOe1LwCfMbDnwB+CayNPOzwHjga9H1eeFsbm1gW9vcxu/fGYdp3xvAXfMX8sZEwt56t9O4/sfmqokWyTOWFBH9uJEs5HA9Mjm4sgSvwc7PxFYQ9RUUcDl0SPfzewaoNzdP9el7AigAign6CO4FJjm7jsP9H3l5eVeUVHRq3vpL9f/egnLNuzihZvP1AI2InJQZrbU3cv76FqZwF6CxpQrCWYIud/dt/fF9fvKQKy3D0dzazsPVlRzxz/fYGtdE6dPLODL505kclFO2KGJSB/rbZ3d2yXYPwIsJpiL9SPAIjP7UA/FOqeKcvdmoGOqqN54D/CUu++IJNdPAef1suyAcd2cMnY0NPPISxvDDkVE4oi7N7h7u7u3uvuv3f2nAy3JHkra2p1HXqrh7B8/w9cffZWxIzKY98lZ3HftDCXZInGut320/wuY3tGKbWYFwD+Ahw9SpjdTRQF80MxOJWj9vsndqw9QdtAtfnBS2QiOHT2MexZWcWn5GBK0+ICIyJDh7jy1ags/enINq7fUMWnUMO69ZjqnTyzQgmUiAvS+j3ZCl64i23tRtjdTRf0ZKHH34wgS91+/i7IDfpqoYAGbUtZureeZNwZefCIicmheWLuND/y/F7jht0tpbmvnjstP4K83zuGMowuVZItIp94m2n83sycigxevIRhd/ngPZXqcKsrdt0ctanAXMK23ZSPlB/w0Ue+bMpqRw1K55zlN9Sci/cPMvtCbffLuvVy9i6vuXsQVdy9iy559fPeSKTx106lcMHW0nlqKyDv0KtF2938H7gSOA6YCd7r7f/RQrHOqKDNLIZgq6rHoE8xsVNTmhQQj3SEY9X6umQ03s+HAuZF9g05KUgJXzy5h4dptvLZ5T9jhiEh86G6mpmv6O4ihZNOuvXzytxVc/PPnWbV5D1973yQWfPl0LpsxlqRELZkuIt3rbR9t3P2PwB/fxfmtZtYxVVQiMLdjqiigwt0fAz4fmTaqFdhB5BeBu+8ws28TJOsAt7r7jt5+90Bz5Yxx3PHPtdyzsIoffnhq2OGIyBBlZpcTzH1dambRDRvZBF3+5BDsbW7j4/ctYcOORm46+yiuO6WUrNRe//oUkTh20JrCzOropm80QR9qd/dhByvv7o/TpYuJu38j6vNXga8eoOxcYO7Brj9Y5GQk85HyYn6/eANfec9ECoelhR2SiAxNLwCbCVZr/FHU/jrglVAiGuTcnf96dAWrt9Qx95rpnDExbqcHF5FDcNDnXe6e7e7Dunll95Rky/6uPbmU1nbnNy9qARsRiQ13f9Pdn3b3WcB6INndnyHolqeVUg7B/Ys28KdlG/n8mROUZIvIu6aOZf2kJD+TcyaN5HeL3mRvc1vY4YjIEGZmnyCYfvVXkV3FwKPhRTQ4vVy9i1v/vIrTjirgC2dNCDscERmElGj3o+tPKWNXYwt/XFYTdigiMrR9FjgZ2APg7m8Aao59F3Y0NPOZ3y2lIDuVn1x6vGYUEZFDokS7H00vGc5xxTnMXVhFe3t3Xd9FRPpEU2RFXgDMLInux9tIN9ranS888BLbGpr55VXTGJ6ZEnZIIjJIKdHuR2bGdXNKqdzWwILVW3suICJyaJ4xs/8E0s3sHOAhggXCpBd+8o81PPfGNm698FimFGsJdRE5dEq0+9n5U0YxOieNu7WAjYjEzs1ALbAC+CTB7E9fCzWiQeKfr23hjvlr+fC0Yi6dPqbnAiIiB6FEu58lJyZwzcklvFi5nVc37g47HBEZgty93d3vcvcPAzcAi9xdXUd6sGF7Izc9+DLHjh7Gty+erKXUReSwKdEOwaXTx5KZksh/PrKCh5fWsK2+qedCIiK9ZGZPm9kwMxsBvAzca2Y/7kW588xstZmtNbObuzk+1swWmNlLZvaKmZ0fdeyrkXKrzew9fXtHsbevpY1P/W4pAL+4chppyYkhRyQiQ4GWtgpBTnoy37jgGH705Bq+/NByzOC44lzOnFjImUcXcuzoYRrhLiKHI8fd95jZ9cC97v5NMzvogjVmlgj8HDgHqAGWmNlj7r4q6rSvAfPc/RdmdgxBl5SSyOfLgGOB0cA/zOwodx8Uc5m6O1979FVWbd7D3GvKGZuXEXZIIjJEKNEOyaXTx/KR8jGs3LSH+a9vZf7rW/nJP9dw2z/WUJCdyhkTCzjz6ELmTCjQUr8i8m4lmdko4CPAf/WyzAxgrbtXApjZA8BFQHSi7UDHYmU5wKbI54uAB9y9Cagys7WR6714WHfRT/6wuJqHl9bw+TPHc+bRI8MOR0SGEGVwITIzJhflMLkoh8+fNYHt9U08vbqW+au38rdX32JeRQ3JicaM0hGcEWntLivICjtsERn4bgWeABa6+xIzKwPe6KFMEVAdtV0DzOxyzi3Ak2Z2I5AJnB1V9l9dyhZ19yVmdgNBv3HGjh3b443E2vLqXdzy2EpOmZDPF84+KuxwRGSIUaI9gORlpfLBacV8cFoxLW3tLH1zJwsird3f+etrfOevr1GSl8EZRwdJ94zSEaQmqR+hiOzP3R8imNKvY7sS+GAPxbrrr9Z1AOXlwH3u/iMzmwX81swm97JsRyx3AncClJeXhzpAc0e5uosuAAATU0lEQVRDM5+5fxkF2an89LITSFSXPRHpY0q0B6jkxAROKsvjpLI8vnr+JKp3NLJgdZB0379oA/c+v57MlEROHp/PWZMKOWNiIYXD0sIOW0QGrxogej67Yt7uGtLhOuA8AHd/0czSgPxelh1QOhalqa1r4qFPzdKiNCISE0q0B4kxIzL42KwSPjarhL3NbbywbhvzX9/Kgte38uSqLQBMLhrGmRMLOePoQqYW52pApYi8G0uACWZWCmwkGNx4RZdzNgBnAfeZ2SQgjWC+7seA30dmNhkNTAAW91fgh+L2yKI0//OBKUwdkxt2OCIyRCnRHoTSUxI5a9JIzpo0Endn9Za6YEDla1v52YK1/HT+WvIyUzgtMqDylAkF5KQnhx22iPQTM0t8tzN+uHurmX2OoG93IjDX3Vea2a1Ahbs/BnwJuMvMbiLoGnJNZH7ulWY2j2DgZCvw2YE848j817fw0/lr+dC0Yi6foUVpRCR2bKisYVBeXu4VFRVhhxG6nQ3NPPtGLfNf38oza2rZ1dhCYoJRPm44Z0b6do8vzNJCDCIDjJktdffyPrrWBuDvwIPA/IG6WE0Y9faG7Y28/47nKB6ewZ8+M1vzZYvIIeltna0W7SFmeGYKFx1fxEXHF9HW7ry0YWfn9IH/+7fX+d+/vc6YEenMGZ/PjNIRzCzNY3Ruethhi0jfmghcAHwWuMfM/kIw/d7CcMMK176WNj59f7AozS+v0qI0IhJ7SrSHsMQEo7xkBOUlI/jKeUezaddeFqzeyoLXa/nLK5v5w+JgJq/i4enMKB3BSaV5zCgdwbi8DLV4iwxi7r4XmAfMM7PhwO3AMwRdQuLWN/7vVVZu2sM9V2tRGhHpHzFNtM3sPIIKPhG4292/2+X4vwHXE/TpqwU+7u5vRo61ASsip25w9wtjGWs8GJ2bzpUzx3HlzHG0tTuvv7WHxVU7WFy1g2dW1/KnZRsBGDkslRmRpHtm6QgmqKuJyKBjZqcBlwLvJRjo+JFwIwrXA4s3MK+ihhvPHM9Zk7QojYj0j5gl2r1czvcloNzdG83s08D3CX4xAOx19+NjFV+8S0wwjh2dw7Gjc7j25FLcnXW19Syq2sGiyh0sqtrOn5cHs3ONyExheslwZpTmMbN0BJNGDdN8syIDmJlVAS8TtGr/u7s3hBxSqFbU7OYbkUVpvqhFaUSkH8WyRbvH5XzdfUHU+f8CrophPHIQZsb4wmzGF2Zz5cxxuDsbdjSyKNLivahqO0+sDKYRzE5LYnrJCGaUBq8pRTkkJyaEfAciEmWqu+8JO4iBYGdDM5/63VLyM1O4XYvSiEg/i2Wi3ZvlfKNdB/wtajvNzCoIupV8190f7VpgoC3lO5SYGePyMhmXl8lHyoPprzbt2suS9Tsird7bmf/6VgDSkxOZNm54Z1eTqWNyNchIJFxHmNkjwEh3n2xmxwEXuvt3wg6sP7W1O1988GVq65qY96lZjNCiNCLSz2KZaPd6SV4zuwooB06L2j3W3TeZWRkw38xWuPu6/S42gJbyjQejc9M7ZzQB2FbfxJKqSOJdtYPb/rEGd0hJTOD4MbnMLAtavE8cO5zMVI27FelHdwH/DvwKwN1fMbPfA3GVaP/0n2/wzJpavnPxZI7XojQiEoJYZj+9WpLXzM4G/gs4zd2bOva7+6bIe6WZPQ2cAKzrWl7Ck5+VynunjOK9U0YBsLuxhSXrd7B4fdDi/f+eXscd89eSlGBMLsphZukIppeM4KiR2RQNT9cjXJHYyXD3xV0GMbeGFUwYFqzeyk/nv8ElJxZx5Uw98RSRcMQy0e5xOV8zO4GgxeU8d98atX840OjuTWaWD5xMMFBSBrCcjGTOPmYkZx8TjOivb2pl2Zs7WVS1ncVVO7j3+fX86tlKIGj1HpuXQWl+JmX5mZTmZ1IS+VyQnapZTkQOzzYzO5LIU0Qz+xCwOdyQ+k/1jka++MDLTByZzX9fPEX1iYiEJmaJdi+X8/0BkAU8FKkIO6bxmwT8yszagQSCPtqruv0iGbCyUpM49agCTj2qAAgWi1ixcTeVtfVUbmugqraBqm0NPLO6lua29s5ymSmJlBZkUpqf1ZmIl0SScS0lL9IrnyXoVne0mW0EqoiTweYdi9K0u/Orj04jPUXjRUQkPDHtOOvujwOPd9n3jajPZx+g3AvAlFjGJv0vLTmR6SVB95Fobe3Opl17qdrW0Pmq3NbAy9U7+csrm4hePDovM4XSSNJdWtDRGp7FuLwMDcAUiYjM9nS2mWUCCe5eF3ZM/eWb/7eSVzfu4e6PlTMuLzPscEQkzmmEmoQuMcEYMyKDMSMyOlu/O+xraaN6R+M7kvCn19Ty0NKazvPMYHRO+ttJeFQiXpSbTpKmH5Q4EFkErLv9ALj7j/s1oH724JINPFhRzWfPOLKzC5uISJiUaMuAlpacyISR2UwYmf2OY3X7Wli/rZGq7R3dUOqp2tbAoy9tpK7p7XFfyYlBIj9uRAaF2WkUDkulIDuVgqzgvTA7jYLsVD1ilqGg4wdlIjAdeCyyfQHwbCgR9ZMVNbv5+v+tZM74fP7tnIlhhyMiAijRlkEsOy2ZKcU5TCnO2W+/u7O9oTloAa9t6EzEN+xo5NVNe9he30R7N5NBZqUmUZidSn7224l44bB3JuQjMlM0Y4oMSO7+LQAzexI4saPLiJndAjzUU3kzOw+4nWBczd3u/t0ux28DzohsZgCF7p4bOfZ94H0E42qeAr7g7v0y7equxmY+fX/HojTH6+dTRAYMJdoy5JgZ+Vmp5GelvqM/OAR9wrc3NFFb9/Zra8fn+iZq9zSxatMeauuaqG9654xoCQZ5WakUdpuQp0WS8uCY5g+XkIwFmqO2m4GSgxUws0Tg58A5BNOzLjGzx6IHorv7TVHn30gw7SpmNptgdqjjIocXEqyL8PRh3keP2tudmx58mS179jHvk7PIy0qN9VeKiPSasgCJO4kJFnQhyU7r8dzG5tb9EvLa+ia27on6XLeP1zbvYVt9M23dNJNnpCRSGGkFz0lPJjcjeA8+7/+ek/72sZQk9SmXw/JbYHFkdUgHPgD8uocyM4C1kYGUmNkDwEXAgWZ8uhz4ZuSzA2lACsFiZcnAlsO5gd66Y/5aFqyu5dsXT+aEscP74ytFRHpNibbIQWSkJDEuL6nH2Qva252djc1vt4xHJeVb6/axs7GZ2vom3thaz+69LdTtO/jaIRkpieSmJ5OTkUJOehK56SmdSfmw6CS9y/7s1CQS9Ng87rn7f5vZ34BTIruudfeXeihWBFRHbdcAM7s70czGAaXA/Mj3vWhmCwjm6jbgZ+7+2mHcQq88vXorP/nnGi45oYirtCiNiAxASrRF+kBCgpGXlUpeViqTRvV8fmtbO3X7Wtm1t4Xde1vY1djM7sjn3Y0tUftb2LO3hcpt9Z3bTa3tB7xugtHZKp4TaT3PTk0iMzWRzNQkslKTyEhJIiuynZmaRGZKcDyrYzs1icyURM3UMsi5+zJg2bso0t1faAfqY30Z8LC7twGY2XiC9Q+KI8efMrNT3f0dAzDN7AbgBoCxYw89Oa7e0cgXH4wsSvMBLUojIgOTEm2RECQlJjA8M4XhmSnvuuy+lrbOpPsdSXr0/sj2xp2NNDS10dDUSkNza7cDQbuTmpTwjuS7I1nPTE2MJOxJkX3Bsbf3BYl7VlpkO0Ut7YNADTAmarsY2HSAcy8jWBSnwweAf7l7PUCkNf0kupnpxN3vJFhMh/Ly8kMaLLmvpY3P3L+Mtnbnl1dpURoRGbiUaIsMMmnJiaQlJzJyWM99zLtyd/a1tFPf1EpDUyv1Ta00Nrd1fg6S8UhS3s2+XY3N1OxspLG5rfNYbxP3zJTEzsQ7Ky2ZrI5kPDWZ7LSopD0tiezUpP0S9ejPqUkJar2MjSXABDMrBTYSJNNXdD3JzCYCw4EXo3ZvAD5hZv9L0DJ+GvCTWAX6rT+vZMXG3dz1sXJK8rUojYgMXEq0ReKImZGekkh6SiIF2Yc/O0N04t7Y3JGY75+41ze1Urfv7e26plbqI9vb6hqpj5xT39Ta7YDSrpITrbNlPSs1ab8kveNzekoSyQlGclICSQlGcmICSYlGckLwnpSYQHJC5D0xcjxqOykh8h7Zn5LU/fGhlPC7e6uZfQ54gmB6v7nuvtLMbgUq3L1jTu7LgQe6TN33MHAmsIKgu8nf3f3PsYhzXkU1f1hczWdOP5JztCiNiAxwSrRF5JBFJ+5weIl7R9Je19RC/b4gYe/4HJ2Md27veztp397QzJvbGzu397a09c0N9iAxwToT+Y7EfFZZHj+9/IR++f6+5u6PA4932feNLtu3dFOuDfhkTIMDXt24m68/+iqzj8zj3845KtZfJyJy2JRoi8iAEJ20F75zIdB3xd1paXNa29uD97Z2WtudlrZ2Wvfb77S0R/a1tdPc9XjUeQe6Vkvb/uceWaCuDLGSn5XKWZMKufWiyRqsKyKDghJtERlyzIyUJCMFJWNDyRE5afy/K6eFHYaISK/pt5CIiIiISAwo0RYRERERiQEl2iIiIiIiMWD7z9A0eJlZLfDmIRTNB7b1cTiDQTzedzzeM8TnfQ/Gex7n7gVhB9GfVG+/K7rn+BGP9z0Y77lXdfaQSbQPlZlVuHt52HH0t3i873i8Z4jP+47He44n8fjvq3uOH/F430P5ntV1REREREQkBpRoi4iIiIjEgBJtuDPsAEISj/cdj/cM8Xnf8XjP8SQe/311z/EjHu97yN5z3PfRFhERERGJBbVoi4iIiIjEQFwn2mZ2npmtNrO1ZnZz2PHEmpmNMbMFZvaama00sy+EHVN/MrNEM3vJzP4Sdiz9wcxyzexhM3s98m8+K+yY+oOZ3RT5//2qmf3BzNLCjkn6RrzV2RDf9Xa81dkQn/X2UK+z4zbRNrNE4OfAe4FjgMvN7Jhwo4q5VuBL7j4JOAn4bBzcc7QvAK+FHUQ/uh34u7sfDUwlDu7dzIqAzwPl7j4ZSAQuCzcq6QtxWmdDfNfb8VZnQ5zV2/FQZ8dtog3MANa6e6W7NwMPABeFHFNMuftmd18W+VxH8ANcFG5U/cPMioH3AXeHHUt/MLNhwKnAPQDu3uzuu8KNqt8kAelmlgRkAJtCjkf6RtzV2RC/9Xa81dkQ1/X2kK6z4znRLgKqo7ZriIPKq4OZlQAnAIvCjaTf/AT4CtAediD9pAyoBe6NPHq928wyww4q1tx9I/BDYAOwGdjt7k+GG5X0kbiusyHu6u14q7MhDuvteKiz4znRtm72xcUULGaWBfwR+KK77wk7nlgzs/cDW919adix9KMk4ETgF+5+AtAADPk+rWY2nKCVsxQYDWSa2VXhRiV9JG7rbIivejtO62yIw3o7HurseE60a4AxUdvFDLHHFd0xs2SCyvp+d/9T2PH0k5OBC81sPcHj5jPN7HfhhhRzNUCNu3e0fD1MUIEPdWcDVe5e6+4twJ+A2SHHJH0jLutsiMt6Ox7rbIjPenvI19nxnGgvASaYWamZpRB0vn8s5JhiysyMoO/Xa+7+47Dj6S/u/lV3L3b3EoJ/5/nuPqT+Yu7K3d8Cqs1sYmTXWcCqEEPqLxuAk8wsI/L//SyG+GCiOBJ3dTbEZ70dj3U2xG29PeTr7KSwAwiLu7ea2eeAJwhGuc5195UhhxVrJwMfBVaY2cuRff/p7o+HGJPEzo3A/ZGkpBK4NuR4Ys7dF5nZw8AygtkaXmIIrzgWT+K0zgbV2/EmrurteKiztTKkiIiIiEgMxHPXERERERGRmFGiLSIiIiISA0q0RURERERiQIm2iIiIiEgMKNEWEREREYkBJdoyZJnZC5H3EjO7oo+v/Z/dfZeIiBwa1dkyFGl6PxnyzOx04Mvu/v53USbR3dsOcrze3bP6Ij4REXmb6mwZStSiLUOWmdVHPn4XOMXMXjazm8ws0cx+YGZLzOwVM/tk5PzTzWyBmf0eWBHZ96iZLTWzlWZ2Q2Tfd4H0yPXuj/4uC/zAzF41sxVmdmnUtZ82s4fN7HUzuz+yCpaIiKA6W4amuF0ZUuLKzUS1jkQq393uPt3MUoHnzezJyLkzgMnuXhXZ/ri77zCzdGCJmf3R3W82s8+5+/HdfNclwPHAVCA/UubZyLETgGOBTcDzBCu+Lez72xURGdRUZ8uQoRZtiUfnAh+LLGe8CMgDJkSOLY6qsAE+b2bLgX8BY6LOO5A5wB/cvc3dtwDPANOjrl3j7u3Ay0BJn9yNiMjQpjpbBi21aEs8MuBGd39iv51Bv8CGLttnA7PcvdHMngbSenHtA2mK+tyGfv5ERHpDdbYMWmrRlnhQB2RHbT8BfNrMkgHM7Cgzy+ymXA6wM1JhHw2cFHWspaN8F88Cl0b6FBYApwKL++QuRETig+psGTL015nEg1eA1sjjxPuA2wkeAS6LDG6pBS7uptzfgU+Z2SvAaoJHkR3uBF4xs2XufmXU/keAWcBywIGvuPtbkUpfRER6pjpbhgxN7yciIiIiEgPqOiIiIiIiEgNKtEVEREREYkCJtoiIiIhIDCjRFhERERGJASXaIiIiIiIxoERbRERERCQGlGiLiIiIiMSAEm0RERERkRj4/3XyMf9J8l4vAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111e31e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "reload(bilstm);\n",
    "bilstm.plot_results(losses, accuracies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(tagger_base);\n",
    "reload(bilstm);\n",
    "confusion = tagger_base.eval_model(model,'bilstm-dev-en.preds', word_to_ix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8585738763674706\n"
     ]
    }
   ],
   "source": [
    "print (scorer.accuracy(confusion))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger_base.apply_model(model,'bilstm-te-en.preds',word_to_ix, all_tags, testfile=TEST_FILE_HIDDEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'data/en-ud-test.conllu'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-257-c40e0e2eafe8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# you don't have en-ud-test.conllu, so you can't run this\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mte_confusion\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscorer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_confusion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mTEST_FILE\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'bilstm-te-en.preds'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mscorer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maccuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mte_confusion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Dev/cs4650/psets/ps2/gtnlplib/scorer.py\u001b[0m in \u001b[0;36mget_confusion\u001b[0;34m(keyfilename, responsefilename)\u001b[0m\n\u001b[1;32m     24\u001b[0m     \"\"\"\n\u001b[1;32m     25\u001b[0m     \u001b[0mcounts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdefaultdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m     \u001b[0;32mwith\u001b[0m \u001b[0mcodecs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeyfilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'utf8'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mkeyfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     27\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponsefilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mresfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mkey_line\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkeyfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Dev/Lib/miniconda/lib/python3.6/codecs.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(filename, mode, encoding, errors, buffering)\u001b[0m\n\u001b[1;32m    893\u001b[0m         \u001b[0;31m# Force opening of the file in binary mode\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m         \u001b[0mmode\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'b'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 895\u001b[0;31m     \u001b[0mfile\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbuiltins\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffering\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    896\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mencoding\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'data/en-ud-test.conllu'"
     ]
    }
   ],
   "source": [
    "# you don't have en-ud-test.conllu, so you can't run this\n",
    "te_confusion = scorer.get_confusion(TEST_FILE,'bilstm-te-en.preds')\n",
    "print (scorer.accuracy(te_confusion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- You may want to try other ideas like: learning rate, hidden layer sizes, optimizers, other pretrained embeddings for english etc. to improve the performance of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part of Speech Tagging for Norwegian\n",
    "Make sure your code runs on the Norwegian dataset as well. You may want to use the pretrained embeddings for norwegian to improve the performance of the model. However, there is no deliverable or test for this part.\n",
    "- Run the `BiLSTM` model on the norwegian dataset using the `BiLSTM` tagger. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {},
   "outputs": [],
   "source": [
    "#recalculating all_tags and tag_to_ix for the norwegian language\n",
    "if START_TAG in all_tags_nr:\n",
    "    all_tags_nr.remove(START_TAG)\n",
    "if END_TAG in all_tags_nr:\n",
    "    all_tags_nr.remove(END_TAG)\n",
    "\n",
    "tag_to_ix_nr={}\n",
    "for tag in all_tags_nr:\n",
    "    if tag not in tag_to_ix_nr:\n",
    "        tag_to_ix_nr[tag] = len(tag_to_ix_nr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "words in the vocabulary:  7600\n",
      "7599\n"
     ]
    }
   ],
   "source": [
    "#recalculating the vocab for the norwegian language: obtains the most common 7600 words from the file\n",
    "vocab_nr, word_to_ix_nr = most_common.get_word_to_ix(NR_TRAIN_FILE, 7600)\n",
    "print ('words in the vocabulary: ',len(word_to_ix_nr))\n",
    "print (word_to_ix_nr[UNK])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Loading training data for norwegian:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(preproc);\n",
    "X_tr_nr, Y_tr_nr = preproc.load_data(NR_TRAIN_FILE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Loading dev data for norwegian:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_dv_nr, Y_dv_nr = preproc.load_data(NR_DEV_FILE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Loading test data for norwegian:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_te_nr, Y_te_nr = preproc.load_data(NR_TEST_FILE_HIDDEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [],
   "source": [
    "#setting up the random seed\n",
    "torch.manual_seed(765);\n",
    "\n",
    "# initializing your model\n",
    "embedding_dim=30\n",
    "hidden_dim=30\n",
    "model = bilstm.BiLSTM(len(word_to_ix_nr),tag_to_ix_nr,embedding_dim, hidden_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1: Dev Accuracy: 0.6328169822145726\n",
      "Epoch 3: Dev Accuracy: 0.7308089500860585\n",
      "Epoch 5: Dev Accuracy: 0.7577739529546759\n",
      "Epoch 7: Dev Accuracy: 0.7927710843373494\n",
      "Epoch 9: Dev Accuracy: 0.8055077452667814\n"
     ]
    }
   ],
   "source": [
    "# training your model for norwegian data\n",
    "loss = torch.nn.CrossEntropyLoss()\n",
    "model, losses, accuracies = bilstm.train_model(loss, model, X_tr_nr,Y_tr_nr, word_to_ix_nr, tag_to_ix_nr, \n",
    "                                        X_dv_nr, Y_dv_nr, num_its=10, status_frequency=2, \n",
    "                                        optim_args = {'lr':0.2,'momentum':0}, param_file = 'best.params')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAACdCAYAAACKLN/OAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xd81fX59/HXlb3JJiEkhL1BNIB7tCpoHbXVumqtrbW/3mqX9Vft9O667a+1rW3tcLeWaq2r4k8BR9W6EATZyCYkIYyEHSAkue4/ziEEDBBMTr5Jzvv5ePDI+Y7POdf3AXy4uM5nmLsjIiIiIiIdKyboAEREREREeiIl2iIiIiIiEaBEW0REREQkApRoi4iIiIhEgBJtEREREZEIUKItIiIiIhIBSrRFRERERCJAibaIiIiISARELNE2swfNbKOZLTzM9TPNbJuZvR/+9YMW1yab2QdmtsLMbotUjCIiIiIikWKR2hnSzE4HdgJ/dfdRrVw/E/iWu19wyPlYYBlwDlABzAKudPfFR/vM3NxcLy0tbX/wIiKd7L333tvs7nlBx9GZ1GeLSHfV1j47LlIBuPvrZlb6EZpOAFa4+yoAM3sMuBg4aqJdWlrK7NmzP8JHiogEy8zWBh1DZ1OfLSLdVVv77KDHaJ9kZvPM7AUzGxk+VwSsa3FPRfhcq8zsBjObbWazN23aFMlYRURERETaLMhEew7Qz93HAr8Dngmft1buPez4Fne/193L3L0sL+/Yv3WN1NAZEREREYluERs6cjTuvr3F6+fN7A9mlkuogl3c4ta+QFUkYpi5qoafvbCUv143gV4p8ZH4CBERERHpAvY2NLKuto6Vm3axatMuVm/eyerNu3jkixNJio+NyGcGlmibWQGwwd3dzCYQqq7XAFuBwWbWH6gErgCuikQMqYlxLKrcxh1TF/Hry4+LxEeIiIiISCdxdzZs38uqTTtZuXkXqzftYlU4oV5XW0dTi4EMuWmJDMhNZfvufd0v0TazR4EzgVwzqwB+CMQDuPufgEuBr5hZA7AbuMJD4zgazOwmYDoQCzzo7osiEeOool7ceNYg7n55OZNHFTBpZEEkPkZEREREOtCOPftYvTlUmV61eRerNoWS6dWbd1FX39h8X3J8LP1zUxlV1IuLx/ahf14qA3LTKM1NpVdy5EczRHLVkSuPcv33wO8Pc+154PlIxHWomz42iBcXb+C7Ty9gfGk22akJnfGxIiIiInIE+xqbWFdbFx7mEapM70+sN+3Y23xfjEHfrBT656YyoX82A3JTGZCXRv/cVAoykoiJaW36X+cIbOhIVxEfG8OvLh/Lhb97g+8/s5B7rj4+6JBEREREerzGJqeuvoGdexsor6k7qDK9atMuymvraGgx1iM7NYH+uamcOSSvOZEemJdKSU4KiXGRGfrRXlGfaAMMK8jg62cP4RfTP2DyvCouHNsn6JBERDqdmU0G7iY0bO9+d7/zkOslwF+AzPA9t4W/gcTMbge+CDQCX3X36Z0Zu4h0PHenvrGJ3fWN1IV/hV43ULevsfn87vqG5ut79rW4d1/Dh9rtrm+kLnxPfUPThz4zMS6G/rmpDC1I57zRBfTPTWNAXioDclPJTOl+ow6UaId9+fQBzFi8ge//ayETB2STn54UdEgiIp0mvCvvPbTYldfMnj1kV97vAY+7+x/NbAShIX6l4ddXACOBPsBLZjbE3RsRkS6pvqGJFRt3smT9dpas387S6h3U7KpvTpr3J8SNTce2DHJiXAwpCbGkJMSRnBBLSkIsyfGx5KYlkJKQcuBcQiwp8XGhexNjKQ4P/SjKTA50qEdHU6IdFhcbw12XjeH8377Bd55ayH2fOwGznvMbLSJyFG3ZldeBjPDrXhxYevVi4DF33wusNrMV4fd7uzMCF5Ej27xzbyiZXr+DJeu3s3j9dlZu2sm+xlASnRgXw5De6RRlJoeT5NjmhDglIY7k+Jbn4g6+Hh9HUkJM832xPShJ7ghKtFsYlJ/OrecO5afPL+GpOZV8+oS+QYckItJZWtuVd+Ih99wBzDCzm4FU4OwWbd85pO1hd/QVkchoaGxi1eZdzcn0knBi3XLiYO+MRIYXZnDWsHyGF2YwojCd0pxU4mKD3iy8Z1KifYgvnNqf6YuquWPqIk4elENhr+SgQxIR6Qxt2ZX3SuBhd7/LzE4CHjGzUW1sG/oQsxuAGwBKSkraEa5IdNtaV39QMr20ejvLNuxsHvccH2sMyk/ntMG5jCjMYHj4l1ZX61xKtA8RG2P88rKxnHf3f/j2kwv4y3XjNYRERLoVM/sl8NAx7kHQll15vwhMBnD3t80sCTimHX3d/V7gXoCysrJjG/wpEoUam5w1Nbuax1LvT6zXb9vTfE9uWgLDCzP4/MmlDC9MZ3hhBgPz0ohXlTpwSrRbUZqbym3nDeOHzy7isVnruHKCqi4i0q0sBe41szjgIeBRd992lDazOPquvOXAx4GHzWw4kARsAp4F/m5mvyI0GXIw8G5HPYxItKirb2Bh5fYWSfV2Ptiwgz37QlXq2BhjYF5orejhzVXqdC3g0IUp0T6Ma07sx7SF1fzkucWcOiiX4uyUoEMSEWkTd78fuN/MhgLXAfPN7E3gPnf/92HatLorr5n9CJjt7s8CtwD3mdk3CA0N+Xx4R99FZvY4oYmTDcCNWnFE5Oj27GtkztotvL2qhrdW1jBv3dbmdaMzU+IZXpDBVRP6NVepB+WnRWyrcIkMC/WRPUNZWZnPnj27w95vXW0dk3/zOmP6ZjLl+ok9arkZEelazOw9dy/rwPeLBS4glGgXA48DpwK73P2Kjvqc9ujoPlukq9vb0Mj75Vt5e1UNb6+sYW75Vuobm4gxGN03k5MG5DC+NIsRfTIoyEjS0NUurK19tiraR1CcncL3LhjB7U8t4JF31nLtyaVBhyQiclThIRwXAS8DP3P3/cM4fm5mHwQXmUh02dfYxPyKbbwTTqxnr61lz74mzGBEYQafO6kfJw/KYXxpNulJ8UGHKxGgRPsorhhfzLSF1dz5wlLOGJJHaW5q0CGJiBzNQuB77l7XyrUJnR2MSLRobHIWVW3j7ZWhoSCz19Syqz40impYQTpXjC/hpIE5TOyf3S13OZRjp0T7KMyMOz89mnN//Trf+uc8/vHlk7QYu4h0dVuA5vKYmWUCZ7r7M22YFCkibdTU5Cyp3s7bK2t4Z1UNM1fXsmNPAwAD81K55PgiThqQy4kDsslJSww4WglCxBJtM3uQ0PjAje4+qpXrVwPfDh/uBL7i7vPC19YAO4BGoKEjxy1+FIW9krnjwpHc8s95PPjGar50+oAgwxEROZofuvvT+w/cfauZ/RB4JsCYRLo9d2f5xp28vTI0FOSd1TVsrdsHQL+cFC4YU8iJA3I4aUAO+RlaCUQiW9F+GPg98NfDXF8NnOHuW8zsPELrqrbchewsd98cwfiOyaeOL+KFhdX8YsYHnDUsj0H56UGHJCJyOK0tnqtvMEWOkbuzevMu3lpZw9urapi5qobNO+sBKMpM5uzhvTlpQA4nDcyhT6Y2uJMPi1jH6+6vm1npEa6/1eLwHUIbHHRZZsbPPjWKc3/9Orc8Po8nv3KytisVka5qdnhC5D2EluG7GXgv2JBEuoeanXt5eclG3lq5mbdX1bBhe2j78t4ZiZw6KJeTB+Zy0sAcLfsrbdJVKhxfBF5ocezADDNz4M/hncRa1Znb+eanJ/Hji0dx86Nz+fPrq7jxrEER/TwRkY/oZuD7wD8IbY8+A7gx0IhEurC9DY28smQjT86p5NUPNtLQ5OSmJYSGgQwMDQXpn5uq5fbkmAWeaJvZWYQS7VNbnD7F3avMLB940cyWuvvrrbXv7O18Lxzbh2kLq/nNS8v42LB8hhdmRPojRUSOibvvAm4LOg6RrszdmVO+hSfnVPLcvCq272kgPz2RL57an4uO68OIwgwl1tJugSbaZjYGuB84z91r9p9396rwz41m9jSh5ahaTbSD8ONPjmLm6hpueXwez9x4CglxGkIiIl2HmeUB/w2MJLRNOgDu/rHAghLpIspr6nh6biVPza1gbU0dyfGxTBrZm08d35dTBuVqZTHpUIEl2mZWAjwFXOPuy1qcTwVi3H1H+PW5wI8CCrNV2akJ/PSS0Xz5kff4/b9X8M1zhgQdkohIS1MIDRu5APgv4FpgU6ARiQRo2+59PL9gPU/NqWDWmi2YwUkDcrj5Y4OZPKqAtMTAv+CXHiqSy/s9CpwJ5JpZBfBDwuu6uvufgB8AOcAfwl/N7F/GrzfwdPhcHPB3d58WqTg/qkkjC7hkXBH3/HsF5wzvzei+vYIOSURkvxx3f8DMvuburwGvmdlrQQcl0pn2NTbxn+WbeHJOJS8u3kB9QxMD81K5ddJQPjmuiCKtEiKdIJKrjlx5lOvXA9e3cn4VMDZScXWkOy4cyVsrN3PLP99n6s2nkhgXG3RIIiIA+8I/15vZJ4AquvjKTiIdwd1ZVLWdJ+dUMHVeFZt31pOVEs9VE0r41PFFjC7qpXHX0qn0XUk79EqJ585Pj+G6h2bx6xeXc9t5w4IOSUQE4Cdm1gu4BfgdkAF8I9iQRCKnetsennm/kqfmVLBsw04SYmP4+PB8PnV8X84Ykqe5VBIYJdrtdNbQfC4vK+be11dy7sjeHF+SFXRIIhLFzCwWGOzuzwHbgLMCDkkkInbtbWD6omqemlPJmys34w4n9Mvip5eM4oLRfeiVEh90iCJKtDvC9y4YzhsrNvOtx+fxv189jeQEDSERkWC4e6OZXQT8OuhYRDpaY5PzzqoanpxTwbSF1dTVN1KcnczNHxvMJeOK6J+bGnSIIgdRot0B0pPi+Z9Lx3D1/TP5xfQP+MGFI4IOSUSi21tm9ntCK4/s2n/S3ecEF5LIR7d8ww6enFPJv96vZP22PaQnxnHxcX341PF9KeuXpXHX0mUp0e4gpwzK5ZoT+/HQW6uZNLI3EwfkBB2SiESvk8M/Wy6N6oDW0ZZuY/POvUydV8VTcypZULmN2BjjjCF5fPcTwzl7eG+S4vXtsXR9SrQ70G3nDeO1ZZv41hPzmPa100nVupwiEgB317hs6fKampwNO/ZQXlPH2to61tXWsbamjvLw65pd9QCMKsrgBxeM4MKxfchLTww4apFjo0ywA6UmxvGLS8dwxX3v8P9eWMJPPjk66JBEJAqZ2Q9aO+/uXWrzL+n5dtc3sm7LwQn02ppdoddbdlPf0NR8b4xBn8xk+uWkcO7I3vTLSeWsofkMLUgP8AlE2keJdgebOCCH607uz4NvrmbyyEJOHZwbdEgiEn12tXidRGiHyCVHa2Rmk4G7gVjgfne/85Drv+bAKiYpQL67Z4avNQILwtfK3f2idj2BdAvuzqadew+qRpfv/1lbx8Ydew+6Py0xjpLsFAbnp3P28N4UZ6dQkp1Cv5wU+mQmEx+rZfikZ1GiHQH/PXkor36wkW8/OZ9pXz+N9CQtMSQincfd72p5bGa/BJ49UpvwsoD3AOcAFcAsM3vW3Re3eN9vtLj/ZmBci7fY7e7HdUD40sXsbWikYsvug5LotTWh6nR5bR279zU232sGhRlJFGencMaQPPrlpFCcnUK/nFRKslPISonXxEWJKkq0IyApPpZffmYsl/7xLX7y3BJ+fumYoEMSkeiWAgw4yj0TgBXh3Xkxs8eAi4HFh7n/SuCHHRahdCkrNu5k6rwqnl+wnhWbduJ+4FpyfCwl2aEE+tTBuZSEq9IlOSkUZSZrkqJIC0q0I+T4kixuOH0gf3ptJZNHFXDWsPygQxKRKGFmCwitMgKhYSB5HLwCSWuKgHUtjiuAiYd5/35Af+CVFqeTzGw20ADc6e7PHKbtDcANACUlJUcJSTrTuto6ps6vYuq89SxZvx0zmNg/m/NHD6ZfTkpzdTovLVFVaZE2UqIdQd84ZzCvLN3AbU/NZ8bXz9AuVSLSWS5o8boB2ODuDUdp01rm5K2cA7gCeMLdG1ucK3H3KjMbALxiZgvcfeWH3tD9XuBegLKyssO9v3SSDdv38L/z1zN1fhVzy7cCMK4kkx9cMIJPjCmkd0ZSwBGKdG8RTbTN7EFCHf5Gdx/VynUjNPHmfKAO+Pz+DRXM7Frge+Fbf+Luf4lkrJGQGBfLXZcdxyf/8CZ3TF3Ery/X8EUR6RSFwCJ33wFgZmlmNtLdZx6hTQVQ3OK4L1B1mHuvAG5secLdq8I/V5nZq4TGb38o0Zbg1e6q54WF65k6r4qZq2txhxGFGXx78jAuGFNIcXZK0CGK9BiRrmg/DPwe+Othrp8HDA7/mgj8EZhoZtmExv6VEaqovBeelLMlwvF2uNF9e3HjWYP47cvLmTyqgEkjC4IOSUR6vj8Cx7c4rmvl3KFmAYPNrD9QSSiZvurQm8xsKJAFvN3iXBZQ5+57zSwXOAX4n/Y+hHSc7Xv28eKiDTw7r4o3VmymsckZkJfKVz82mAvH9mFQflrQIYr0SBFNtN39dTMrPcItFwN/dXcH3jGzTDMrBM4EXnT3WgAzexGYDDwayXgj5aazBvHS4g189+kFjC/NJjs1IeiQRKRns3C/CoC7N5nZEft7d28ws5uA6YTGdT/o7ovM7EfAbHffv2rJlcBjLd8fGA782cyagBhCY7QPN4lSOsnu+kZeXrqBqfOq+PcHm6hvaKIoM5kvnTaAC8cWMqIwQ2OtRSIs6DHarU2+KTrC+W4pIS6GX10+lgt/9wbff2Yh91x9pKKSiEi7rTKzrxKqYgP8H2DV0Rq5+/PA84ec+8Ehx3e00u4tQDt0dQF7Gxp5fdlmps6r4qUlG6irbyQ/PZGrJ5Zw4dg+jCvOVHIt0omCTrQPN/mmzZNyussM9mEFGXz97CH8YvoHTJ5XxYVj+wQdkoj0XP8F/JbQPBcHXibcT0rP09DYxNurapg6r4ppC6vZvqeBrJR4PjmuiAvH9GFC/2xiY5RciwQh6ET7cJNvKggNH2l5/tXW3qA7zWD/8ukDmLF4A9//10ImDsgmP12zuUWk47n7RkJjrKWHampyZq/d0rzWdc2uetIS4zh3ZG8uGtuHUwblapdFkS6gTYm2mX0NeAjYAdxPaDb5be4+o52f/yxwU3hjhInANndfb2bTgZ+FJ9gAnAvc3s7PClxcbAx3XTaG83/7Bt95aiH3fe4EfYUnIh3OzP4CfM3dt4aPs4C73P0LwUYm7eHuzK/YxtR5VTw3fz3V2/eQFB/Dx4f35sIxfThzaJ42ixHpYtpa0f6Cu99tZpMIbXxwHaHE+4iJtpk9SqgynWtmFYRWEokHcPc/ERoLeD6wgtCs+OvC12rN7MeEZsED/Gj/xMjublB+OreeO5SfPr+EJ96r4LKy4qM3EhE5NmP2J9kA7r7FzMYdqYF0XZVbd/PozHKmzq9ibU0d8bHGGUPyuf38YZw9vDepiUF/OS0ih9PWv537y67nAw+5+zxrQynW3a88ynXnkLVYW1x7EHiwjfF1K184tT8zFldz6xPzeWpOJVdNLGHSyAIS4vQ1n4h0iBgzy9q/JGp4yVRlY92Mu/PknErueHYRu/c1cvLAHG48axCTRhRoAzSRbqKtHe97ZjaD0Ja7t5tZOtAUubB6ttgY4/5rx/O3d9by6Lvl3PzoXHLTErisrJgrx5dQkqPNAkSkXe4C3jKzJ8LHlwE/DTAeOUZbdtXznacX8MLCaib0z+auy8ZqIxmRbsgOXgr1MDeZxQDHAavcfWu4OtLX3edHOsBjUVZW5rNnzw46jGPS1OS8vnwTU2aW8/KSDTQ5nD4kj6snlvDxYfnEaTKLSFQws/fcvawD328kcBahbyRf7orrWnfHPrszvLZsE7f+cx5b6uq55dyhfOm0AVo1RKSLaWuf3daK9knA++6+y8w+S2h3sbvbE6CExMQYZw7N58yh+azftpt/zFrHY++u48uPvEfvjEQuH1/CFeOL6ZOZHHSoItKNhDeb2QQkAZhZibuXBxyWHMGefY3c+cJSHn5rDUN6p/HQdeMZ2adX0GGJSDu0taI9HxgLjAEeAR4APuXuZ0Q2vGPTU6ojDY1N/PuDTUyZuZbXlm3CgI8Ny+fqif04fUieKhsiPVBHVrTN7CJCw0f6ABuBfsASdx/ZEe/fUXpKn90RFlZu4+v/eJ8VG3dy3SmlfHvyMK0gItKFdXRFu8Hd3cwuBu529wfM7Nr2hSiHExcbwzkjenPOiN6sq63jsVnl/GNWBS8tmUVRZjJXTijmM2XF5GdoHW4RadWPgROBl9x9nJmdRWjrdOliGpucP722kt+8tIzs1AQe+eIEThucF3RYItJB2ppo7zCz24FrgNPMLJbwMn0SWcXZKdw6aRhf+/gQXlqygSkz1/LLGcv4zUvLOWdEb66e2I+TB+YQoyq3iBywz91rzCzGzGLc/d9m9vOgg5KDraut45uPv8+sNVv4xOhCfnrJKDJTEoIOS0Q6UFsT7cuBqwitp11tZiXALyIXlhwqIS6G80cXcv7oQlZv3sWj75bzz9nreGFhNf1yUrhqQgmXntCXnLTEoEMVkeBtNbM04HVgipltBBoCjknCWi7bZ8CvPjOWS8YVaQMzkR6oTWO0AcysNzA+fPhueIvfLiXaxvvt2dfI9EXVTHmnnHfX1JIQG8PkUQVcPbGECf2z1WmLdCMdPEY7FdgNxABXA72AKe5e0xHv31Girc+G0LJ9331mAc8vCC3b96vPjKVvlpbtE+luOnSMtpl9hlAF+1VCS0X9zsxudfcnjthQIiopPpaLjyvi4uOKWLZhB3+fWc6Tcyp4dl4Vg/LTuGpCCZ8+vq82NhCJMu6+K/yyCfhLkLHIAa8v28S3wsv2fXvyMG44Xcv2ifR0bV11ZB5wzv4qtpnlEZpkMzbC8R2TaKyOHGp3fSPPza9iysxy3l+3lcS4GC4Y04erTyxhXHGmqtwiXVRHr6PdHURLn91y2b7B+Wn8+vLjGFWkZftEurOOXnUk5pChIjWEvpKULiY5IZbLyoq5rKyYRVXb+PvMcp6ZW8mTcyoYXpjBVRNL+ORxfUhPUpVbRCTSFlZu4xv/eJ/lG3fy+ZNLue08LdsnEk3amixPM7PpZvZ5M/s88L/A85ELSzrCyD69+Oklo5n53bP52SWjiTH4/jMLmfizl/nm4+8zY1E1e/Y1Bh2miHQwM/taW85J5DQ2OX98dSWX/OFNtu/ZxyNfnMAdF41Uki0SZY5lMuSngVMIjdF+3d2fjmRgH0W0fA35Ubk78yq28ejMcqYtqmbb7n0kx8dy5tA8Jo8q4Kxh+WSo0i0SiA6eDDnH3Y8/5Nxcdx/XEe/fUXpqn72uto5bHp/Hu2tqOX90AT/95GiyUrVsn0hP0tFDR3D3J4EnjzGIyYS2ao8F7nf3Ow+5/mvgrPBhCpDv7pnha43AgvC1cne/6Fg+Wz7MzDiuOJPjijP5ySWjmLmqlumLqpm+qJoXFlYTH2ucPDCXyaMKOGdEb3K1VKBIt2JmVxJairW/mT3b4lI6oSF/R2vfnj77WuB74Ws/cfeom4Tp7jw9t5If/GsRAHddNpZPHa9l+0Si2REr2ma2A2jtBgPc3TOO0DYWWAacA1QAs4Ar3X3xYe6/GRjn7l8IH+9097S2Pgj03OpIpDU1OXPXbWVGOOEur60jxqCsNJtJIwuYNLK3lp8SibCOqGibWT+gP/D/gNtaXNoBzHf3w66l3Z4+28yygdlAGaF/M94DTnD3LUeKtyf12Vvr6vnu0wv53wXrGV+axa8+cxzF2eo3RXqqDqlou3t6O2KYAKxw91XhgB4DLgZa7bQJbQ/8w3Z8nnxEMTHGCf2yOKFfFredN4yl1TuYtjBU6f7xc4v58XOLGV3Ui0kjezN5VAGD8tvzx0JEIsXd1wJrgZPCSfdgd3/JzJKBZEIJ9+G0p8+eBLzo7rXhti8Ck4FH2/lI3cJ/loeW7avdVc9/Tx7Kl08fqGX7RAQ4hqEjH0ERsK7FcQUwsbUbW1RhXmlxOsnMZhPazexOd38mUoHKAWbG8MIMhhdm8I1zhrBm8y6mL6pm2qJqfjljGb+csYyBealMHlXApJEFjC7qpa9FRboYM/sScAOQDQwE+gJ/Aj5+hGbt6bNba1t0mLY3hGOjpKTkKE/Ste3Z18jPpy3loTfXMCg/jQeuHa9l+0TkIJFMtFvLvg43TuUK4Al3b7kERom7V5nZAOAVM1vg7is/9CE9qNPuikpzU/nyGQP58hkDqd62hxcXh5LuP722inv+vZKizGTOHdmbSSMLGF+arSqOSNdwI6EK9UwAd19uZvlHadOePrvNbd39XuBeCA0dOUpMXdaiqm18/TEt2yciRxbJRLsCKG5x3BeoOsy9VxD6h6GZu1eFf64ys1eBccCHEu2e0ml3BwW9krjmpFKuOamULbvqeXnpRqYtrGbKzHIeenMNOakJnDOiN5NGFXDywBwS4/SPjkhA9rp7/f5vm8wsjsMnzfu1p8+uAM48pO2rbQ+3+2hscu77zyrumvEBWSkJ/OULEzhjSF7QYYlIFxXJRHsWMNjM+gOVhDrmqw69ycyGAlnA2y3OZQF17r7XzHIJLSv4PxGMVY5RVmoCl57Ql0tP6MuuvQ28+sEmpi+q5rn563ls1jrSEuP42LB8Jo8q4IwheaQmRvKPmogc4jUz+w6QbGbnAP8HmHqUNh+5zwamAz8L990A5wK3t+8Rup6KLXV88/F5vLu6lvNGFfCzS7Rsn4gcWcSyH3dvMLObCHXAscCD7r7IzH4EzHb3/UtPXQk85gcvfzIc+LOZNRHaVOfOw818l+ClJsbxiTGFfGJMIXsbGnlrRQ3TF1UzY/EGnp1XRWJcDKcNDq3VffbwfDJT9A+TSITdBnyR0BKpXya0wdj9R2rQnj7b3WvN7MeEknWAH+2fGNlTvLF8M1/523s48MvLxvJpLdsnIm3Q5g1ruoOetFRUT9DQ2MTstVuYtrCaGYuqqdq2h9gYY1SfDMpKsxlfmsUJ/bLJS9d63SIduWHNIe+bDfR19/kd/d7t1V367Jmrarj2oXcpzUnlvs+Vadk+Een4DWtEjlVcbAwnDsjhxAE5/PDCESyo3MaLizfw7upa/vbOWh7bVxYZAAATjElEQVR4YzUA/XNTKeuXxfjSbMpKs+ifm6pKkUg7hOe1XESoj38f2GRmr7n7NwMNrBuaW76FLzw8i6LMZP52/URt5CUix0SJtnQKM2NM30zG9M0EoL6hiYVV25i9ppZZa7bw0pIN/PO9CgByUhMoK82irF8o8R7ZpxcJcTFBhi/S3fRy9+1mdj3wkLv/0My6XEW7q1tYuY1rH3yX3PRE/v6lE5Vki8gxU6ItgUiIi+H4kiyOL8nihtNDWxev3LSrOfGevbaW6Ys2AJAUH8NxxZnhinc2x5dkkp4UH/ATiHRpcWZWCHwG+G7QwXRHyzbs4JoHZpKeFM+U6yfSOyMp6JBEpBtSoi1dgpkxKD+NQflpXDEhtB76xu17mL12C7PW1DJ7zRbu+fcKmhxiDIYVZDC+NCs81jubgl76R1CkhR8RmtT4hrvPCu9HsDzgmLqNVZt2ctV9M4mPjWHK9RPpm6Ux2SLy0SjRli4rPyOJ80cXcv7oQgB27m3g/fKtocR7bS2Pz67gL2+vBaBvVnLzGO/xpdkMyksjRpvnSJRy938C/2xxvAr4dHARdR/rauu4+v6ZuDt/v+FESnNTgw5JRLoxJdrSbaQlxnHq4FxOHZwLwL7GJpas3x4aarKmlv8s38zTcysB6JUcT1m/LE4IJ96ji3pp1zYROaL123Zz1f3vUFffyKNfOpFB+elBhyQi3ZwSbem24mNjmidYfvHU/rg7a2vqmoeazFpby8tLNwKQEBvDqKIMRvbpxfDCDIYVpjOsIJ2UBP0VEBHYuGMPV983k6279jHlSxMZ0Scj6JBEpAdQliE9hplRmptKaW4ql5WFdpKu2bmX2WtDFe+55Vt5em4lj7yzNnw/9MtOYXhhRij5LkhneGEGfbOStbygdGtmFuvujUHH0V3U7qrns/fPpHr7Hh754oTm1ZFERNpLibb0aDlpiUwaWcCkkQUANDU5lVt3s3j9dpas387S9TtYvH47Lyysbm6TnhgXrnhnhJPwdIaq+i3dy2ozmwb8A3jFe9LOZB1s2+59XPPATNbW1PHQdeM5oV920CGJSA+izEGiSkyMUZydQnF2SnPyDbBrbwNLq3ewtPpAAn5o9bs0J7W56r2/Aq7qt3RRQ4ELgRuBB8zsOULbpr8RbFhdy869DXz+oXdZtmEH932ujJMH5gYdkoj0MEq0RYDUxDhO6JfFCf2yms8dS/U7lHir+i1dg7vvBh4HHjezLOBu4DVAM4LDdtc38oWHZzG/Yht/uPp4zhyaH3RIItIDKRsQOYxjqX4/NaeSnXsPrn4PbzH8RNVv6WxmdgZwOXAeMIvQ5jUC7NnXyA2PzGb2mlruvmLcQX+/RUQ6khJtkWPU1ur3oqrtPL9A1W/pfGa2GnifUFX7VnffFXBIXUZ9QxM3TpnDf5Zv5peXjeXCsX2CDklEerCI/gtvZpMJfWUZC9zv7ncecv3zwC+AyvCp37v7/eFr1wLfC5//ibv/JZKxirRHe6rf+1c+2Z98a+UT6QBj3X170EF0NQ2NTXz9H3N5eelGfvLJUVx6Qt+gQxKRHi5iibaZxQL3AOcAFcAsM3vW3Rcfcus/3P2mQ9pmAz8EygAH3gu33RKpeEUiobXqt7tTsWU3S9ZvZ8n6UBK+tHoH0xZVs39tiLTEOIYVpB9UAR9WkE5qoqrf0iYFZvY00NvdR5nZGOAid/9J0IEFpbHJufWJ+Ty/oJrvfWI4nz2xX9AhiUgUiOS/2hOAFeGtfzGzx4CLgUMT7dZMAl5099pw2xeBycCjEYpVpNOYHah+n3tI9XvZhh0Hku/1O/jX3Cr+9k558z39clIYXpDRvPzgiHD1W9vNyyHuA24F/gzg7vPN7O9AVCba7s53n17A03MruXXSUK4/bUDQIYlIlIhkol0ErGtxXAFMbOW+T5vZ6cAy4Bvuvu4wbYsiFahIV5CaGMe4kizGlRxc/a7cujuUfK8PVb6XrN/O9MUHqt+pCbEMC0+4HFaYwYjCdIYWZJCm6nc0S3H3dw8ZftQQVDBBcnf+79TFPDZrHTedNYgbzxoUdEgiEkUi+S9xayW2QzdNmAo86u57zey/gL8AH2tj29CHmN0A3ABQUlLy0aMV6YLMjL5ZKfTNSuGcEb2bz++ub+SDDaHke8n67Syp3sHUeVVMmXmg+l2SncKwgtCEy0H5aQzKT2NgXhpJ8VrhLQpsNrOBhPtNM7sUWB9sSJ3P3blz2lIefmsN15/an1vOHRJ0SCISZSKZaFcAxS2O+wJVLW9w95oWh/cBP2/R9sxD2r7a2oe4+73AvQBlZWXa/UyiQnJCLMcVZ3Jc8YGtot2dqm17Dkq+l6zfzstLN9LYFPqrYQbFWSkMDifeg/LTGNw7nYF5qaQnxQf1ONLxbiTULw4zs0pgNfDZozU62gT28D2fAe4glMTPc/erwucbgQXh28rd/aIOeI52ufvl5fz5tVV89sQSvvuJ4ZpgLCKdLpKJ9ixgsJn1J7SqyBXAVS1vMLNCd99fZbkIWBJ+PR34WXijBYBzgdsjGKtIt2dmFGUmU5SZzMeHH6h+721oZM3mOlZs3MnyjTtYsXEnKzbu5D/LN1Pf2NR8X2GvpObke1B+GoPz0xmcn0ZWakIQjyPtEJ4bc7aZpQIx7r7jaG3aMoHdzAYT6otPcfctZtZyl5fd7n5chz5IO/zptZX85qXlXHpCX3500Sgl2SISiIgl2u7eYGY3EUqaY4EH3X2Rmf0ImO3uzwJfNbOLCI0drAU+H25ba2Y/JpSsA/xo/8RIETk2iXGxDA0PIYHC5vMNjU2s27Kb5Rt2sGLTTlZs2MmKTTv5x6x11NU3Nt+Xk5rQIvlOY1B+OoN7p5GfnqjkpYsxs28e5jwA7v6rIzRvywT2LwH37F8Byt03dkDYHe7hN1dz5wtLuXBsH37+6TGaLCwigYnobCl3fx54/pBzP2jx+nYOU6l29weBByMZn0g0i4uNoX9uKv1zUzm3xfmmJqdq2+7myneoEr6TqfOq2L7nwHy69KS4Fsl3qAI+KD+NokytghKg9PDPocB44Nnw8YXA60dp25YJ7EMAzOxNQgWUO9x9WvhakpnNJlQ4udPdn2ntQyI9r+axd8u5Y+pizh3Rm199Ziyx+rMoIgHSsgQicpCYmAMTMM8cemBkgLuzaefe5sr38g2hJPyVpZt4fHZF831J8TEMzAsl36U5qZRkp1CSk0K/7BTyVAWPKHf/vwBmNgM4fv+QETO7A/jnUZq3ZRJ6HDCY0ByavsB/zGyUu28FSty9yswGAK+Y2QJ3X9lKjBGbV/P03Apuf3oBZwzJ43dXjSM+NqYj315E5Jgp0RaRNjEz8tOTyE9P4uRBuQdd21pX31z53v9z9potPDuvqnkZQggl4cVZKc3Jd0n2gV/F2SlaEaXjlAD1LY7rgdKjtDnqBPbwPe+4+z5gtZl9QCjxnuXuVRAaH25mrwLjgA8l2pHy/IL13PL4PE4akMOfrzmBxDj9WRKR4CnRFpF2y0xJoKw0m7LS7IPO721opHLLbspr61hXW8famjrKa0O/3l5Vc9BYcIDeGYnNSXe/7FRKcpKbj/PSVA0/Bo8A74Z3h3TgEkLLpx7JUSewA88AVwIPm1kuoaEkq8IT1+vCS7XmAqcA/9NhT3MULy/ZwFcfncvxJVnc97ky/YdNRLoMJdoiEjGJcbEMyEtjQF7ah665OzW76puT8PKaOtbuT8JX1vD03MqDquHJ8bHNSXeoCp5Mv5xUirNT6JuVrOSqBXf/qZm9AJwWPnWdu889Spu2TGCfDpxrZouBRuBWd68xs5OBP5tZExBDaIx2W3YBbrf/LN/EV/42hxF9MnjwuvGkaqMmEelCzL3nLD1dVlbms2fPDjoMEekAe/Y1Url1N+UtquDl4YS8vLaO3fsOroYXZCQ1D0cpzkqhKCuZPplJFGUmU9ArqcsPJTCz99y9LOg4OlN7++yZq2q49qF3Kc1J5bEbTiQzRUtRikjnaGufrf/6i0iXlBQfy8C80G6Wh3J3Nu+s/9CQlHW1dbyxfDPV2/d8qE1eeiJ9MpMpykyiT69k+mQmh49DCXl2aoKGpnQjc8q38IWHZ9E3K4W/XT9RSbaIdElKtEWk2zEz8tITyUtP5IR+WR+6vmdfI9Xb9lC1dTeVW3dTtTX0umrbbpZW7+CVpRvZs6/poDaJcTHhpDuUeB+ciCdT2CtJw1O6iIWV27j2wXfJTU9kyvUTyU1LDDokEZFWKdEWkR4nKT6W0txUSnNTW73u7myp29ciEd//aw+VW3fz6geb2Lhj74fa5aYlhBLwXgcS8gPJeTK5aaqKR9oH1Tu45oGZZCTFM+X6ifTOSAo6JBGRw1KiLSJRx8zITk0gOzWBUUW9Wr1nb0MjG7btPTgR37abyq17WLFpJ68t2/ShceIJcTGMKerFE185uTMeI+qU19Rx9f0ziY+N4e9fmkjfrJSgQxIROSIl2iIirUiMiw1NrsxpPZlzd7bW7TskEd9DfKwq2pGSm57AyQNz+OrHB9Evp/VvK0REuhIl2iIiH4GZkZWaQNYRquLSsVIS4vjtleOCDkNEpM20P62IiIiISAQo0RYRERERiQAl2iIiIiIiEdCjdoY0s03A2mNslgtsjkA4XV00Pnc0PjNE53N3x2fu5+55QQfRmT5inw3d8/e3vaLxmSE6nzsanxm633O3qc/uUYn2R2Fms6Nt22OIzueOxmeG6HzuaHzmaBKNv7/R+MwQnc8djc8MPfe5NXRERERERCQClGiLiIiIiESAEm24N+gAAhKNzx2NzwzR+dzR+MzRJBp/f6PxmSE6nzsanxl66HNH/RhtEREREZFIUEVbRERERCQCojrRNrPJZvaBma0ws9uCjifSzKzYzP5tZkvMbJGZfS3omDqTmcWa2Vwzey7oWDqDmWWa2RNmtjT8e35S0DF1BjP7RvjP90Ize9TMkoKOSTpGtPXZEN39drT12RCd/XZP77OjNtE2s1jgHuA8YARwpZmNCDaqiGsAbnH34cCJwI1R8MwtfQ1YEnQQnehuYJq7DwPGEgXPbmZFwFeBMncfBcQCVwQblXSEKO2zIbr77WjrsyHK+u1o6LOjNtEGJgAr3H2Vu9cDjwEXBxxTRLn7enefE369g9Bf4KJgo+ocZtYX+ARwf9CxdAYzywBOBx4AcPd6d98abFSdJg5INrM4IAWoCjge6RhR12dD9Pbb0dZnQ1T32z26z47mRLsIWNfiuIIo6Lz2M7NSYBwwM9hIOs1vgP8GmoIOpJMMADYBD4W/er3fzFKDDirS3L0S+CVQDqwHtrn7jGCjkg4S1X02RF2/HW19NkRhvx0NfXY0J9rWyrmoWILFzNKAJ4Gvu/v2oOOJNDO7ANjo7u8FHUsnigOOB/7o7uOAXUCPH9NqZlmEqpz9gT5Aqpl9NtiopINEbZ8N0dVvR2mfDVHYb0dDnx3NiXYFUNziuC897OuK1phZPKHOeoq7PxV0PJ3kFOAiM1tD6Ovmj5nZ34INKeIqgAp331/5eoJQB97TnQ2sdvdN7r4PeAo4OeCYpGNEZZ8NUdlvR2OfDdHZb/f4PjuaE+1ZwGAz629mCYQG3z8bcEwRZWZGaOzXEnf/VdDxdBZ3v93d+7p7KaHf51fcvUf9j/lQ7l4NrDOzoeFTHwcWBxhSZykHTjSzlPCf94/TwycTRZGo67MhOvvtaOyzIWr77R7fZ8cFHUBQ3L3BzG4CphOa5fqguy8KOKxIOwW4BlhgZu+Hz33H3Z8PMCaJnJuBKeGkZBVwXcDxRJy7zzSzJ4A5hFZrmEsP3W0s2kRpnw3qt6NNVPXb0dBna2dIEREREZEIiOahIyIiIiIiEaNEW0REREQkApRoi4iIiIhEgBJtEREREZEIUKItIiIiIhIBSrSlxzKzt8I/S83sqg5+7++09lkiIvLRqM+WnkjL+0mPZ2ZnAt9y9wuOoU2suzce4fpOd0/riPhEROQA9dnSk6iiLT2Wme0Mv7wTOM3M3jezb5hZrJn9wsxmmdl8M/ty+P4zzezfZvZ3YEH43DNm9p6ZLTKzG8Ln7gSSw+83peVnWcgvzGyhmS0ws8tbvPerZvaEmS01synhXbBERAT12dIzRe3OkBJVbqNFdSTc+W5z9/Fmlgi8aWYzwvdOAEa5++rw8RfcvdbMkoFZZvaku99mZje5+3GtfNangOOAsUBuuM3r4WvjgJFAFfAmoR3f3uj4xxUR6dbUZ0uPoYq2RKNzgc+FtzOeCeQAg8PX3m3RYQN81czmAe8AxS3uO5xTgUfdvdHdNwCvAeNbvHeFuzcB7wOlHfI0IiI9m/ps6bZU0ZZoZMDN7j79oJOhcYG7Djk+GzjJ3evM7FUgqQ3vfTh7W7xuRH//RETaQn22dFuqaEs02AGktzieDnzFzOIBzGyImaW20q4XsCXcYQ8DTmxxbd/+9od4Hbg8PKYwDzgdeLdDnkJEJDqoz5YeQ/87k2gwH2gIf534MHA3oa8A54Qnt2wCPtlKu2nAf5nZfOADQl9F7ncvMN/M5rj71S3OPw2cBMwDHPhvd68Od/oiInJ06rOlx9DyfiIiIiIiEaChIyIiIiIiEaBEW0REREQkApRoi4iIiIhEgBJtEREREZEIUKItIiIiIhIBSrRFRERERCJAibaIiIiISAQo0RYRERERiYD/D10VSmhcPHhvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x111f74400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "reload(bilstm);\n",
    "bilstm.plot_results(losses, accuracies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(tagger_base);\n",
    "reload(bilstm);\n",
    "confusion = tagger_base.eval_model(model,'bilstm-dev-nr.preds', word_to_ix_nr, \n",
    "                                   trainfile=NR_TRAIN_FILE, testfile=NR_DEV_FILE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8139988525530695\n"
     ]
    }
   ],
   "source": [
    "print (scorer.accuracy(confusion))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger_base.apply_model(model,'bilstm-te-nr.preds',word_to_ix_nr, all_tags_nr, \n",
    "                        trainfile=NR_TRAIN_FILE, testfile=NR_TEST_FILE_HIDDEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'data/no_bokmaal-ud-test.conllu'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-269-c69128084b26>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# you don't have no_bokmaal-ud-test.conllu, so you can't run this\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mte_confusion\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscorer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_confusion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mNR_TEST_FILE\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'bilstm-te-nr.preds'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mscorer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maccuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mte_confusion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Dev/cs4650/psets/ps2/gtnlplib/scorer.py\u001b[0m in \u001b[0;36mget_confusion\u001b[0;34m(keyfilename, responsefilename)\u001b[0m\n\u001b[1;32m     24\u001b[0m     \"\"\"\n\u001b[1;32m     25\u001b[0m     \u001b[0mcounts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdefaultdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m     \u001b[0;32mwith\u001b[0m \u001b[0mcodecs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeyfilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'utf8'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mkeyfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     27\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponsefilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mresfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mkey_line\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkeyfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Dev/Lib/miniconda/lib/python3.6/codecs.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(filename, mode, encoding, errors, buffering)\u001b[0m\n\u001b[1;32m    893\u001b[0m         \u001b[0;31m# Force opening of the file in binary mode\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m         \u001b[0mmode\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'b'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 895\u001b[0;31m     \u001b[0mfile\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbuiltins\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffering\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    896\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mencoding\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'data/no_bokmaal-ud-test.conllu'"
     ]
    }
   ],
   "source": [
    "# you don't have no_bokmaal-ud-test.conllu, so you can't run this\n",
    "te_confusion = scorer.get_confusion(NR_TEST_FILE,'bilstm-te-nr.preds')\n",
    "print (scorer.accuracy(te_confusion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Using cuda\n",
    "If you want to run the code on a GPU: \n",
    "- make sure you convert the input and target Tensors to be cuda Variables like below:\n",
    "\n",
    "`inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda())`\n",
    "- and convert your model to cuda:\n",
    "\n",
    " `model.cuda()`\n",
    "\n",
    "The following links would be useful: when you want to run your model on a GPU:  \n",
    " - [pytorch_doc](http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#training-on-gpu)\n",
    " - [mnist pytorch gpu example](https://github.com/pytorch/examples/blob/master/mnist/main.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. Adding a CRF on top of BiLSTM\n",
    "=================\n",
    "\n",
    "We will now be building a `CRF` on top of the `BiLSTM` for Part-of-Speech tagging. The LSTM tagger above is typically sufficient for part-of-speech tagging, but a sequence model like the CRF might be helpful for tasks like Named Entity Recognition where the transitions and the overall structure is very important.\n",
    "\n",
    "Recall that the `CRF` computes a conditional probability. Let Y be a tag sequence and X an input sequence of words. Then we compute  \n",
    "\n",
    "$P (Y \\mid X) = \\frac{exp(Score(X,Y))}{\\sum_{Y_i}exp(Score(X,Y_i))}$    \n",
    "where the the score is determined by defining some log potentials $\\log\\Psi(X,Y)$ such that these potential only look at the local features.  \n",
    "\n",
    "In the Bi-LSTM CRF, we define two kinds of potentials: emission and transition. \n",
    "- The emission potential for the word at index $i$ comes from the hidden state of the Bi-LSTM at timestep $i$. \n",
    "- The transition scores are stored in a `|T|x|T|` matrix `transitions`, where T is the tag set. In my implementation, `transitions[j][k]` is the score of transitioning to tag k from tag j.\n",
    "\n",
    "$ Score(X,Y) = \\sum_{i}\\log\\Psi_{emit}(y_i \\rightarrow x_i) + \\sum_{i}\\log\\Psi_{trans}(y_{i-1} \\rightarrow y_{i}) \\\\\n",
    " Score(X,Y) = \\sum_{i} h_{i}[y_{i}] + transitions_{y_i,y_{i-1}} $\n",
    " \n",
    "See the notes for more details on this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Since, we are now using a `CRF`, we need to add `START_TAG` and `END_TAG` to the tagset: so, we modify `tag_to_ix` and `ix_to_tag` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 428,
   "metadata": {},
   "outputs": [],
   "source": [
    "#adding START_TAG and END_TAG to all_tags, updating tag_to_ix\n",
    "if START_TAG not in all_tags:\n",
    "    all_tags.append(START_TAG)\n",
    "if END_TAG not in all_tags:\n",
    "    all_tags.append(END_TAG)\n",
    "all_tags = sorted(all_tags)\n",
    "if START_TAG not in tag_to_ix:\n",
    "    tag_to_ix[START_TAG] = len(tag_to_ix)\n",
    "if END_TAG not in tag_to_ix:\n",
    "    tag_to_ix[END_TAG] = len(tag_to_ix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We have already provided the transitions parameter to the `bilstm.BiLSTM_CRF` class. Take a look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 429,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting random seed\n",
    "reload(bilstm);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 430,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BiLSTM_CRF(\n",
      "  (word_embeds): Embedding(6900, 30)\n",
      "  (lstm): LSTM(30, 15, bidirectional=True)\n",
      "  (hidden2tag): Linear(in_features=30, out_features=19)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# initializing model\n",
    "torch.manual_seed(765);\n",
    "embedding_dim=30\n",
    "hidden_dim=30\n",
    "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)\n",
    "print (model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Observe that `BiLSTM_CRF` class is derived from the `BiLSTM` class\n",
    "- Note that we would be using the forward function from `bilstm.BiLSTM` class to obtain the lstm hidden states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let us prepare an input sequence\n",
    "sentence = bilstm.prepare_sequence(X_tr[5], word_to_ix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      "-0.2339\n",
      " 0.1689\n",
      " 0.0046\n",
      " 0.0083\n",
      " 0.0977\n",
      "[torch.FloatTensor of size 5]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lstm_feats = model.forward(sentence)\n",
    "print (lstm_feats[0][0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 6.1** (*0.5 points*) \n",
    "Complete the `bilstm.BiLSTM_CRF.forward_alg()` function to do the following:  \n",
    "This function calculates the log likelihood score for a particular sentence. It works very similar to the `viterbi` algorithm: Instead of finding the maximum `prev_tag`, you need to calculate the probability to arrive at the `curr_tag` for the `curr_token`. The forward algorithm is described in chapter 6 of the class notes.\n",
    "\n",
    "- **Inputs**: `feats`, the hidden states for each token in the input_sequence. Consider this to be the emission potential of each token for each tag.\n",
    "    - Make sure to use the `self.transitions` that is defined to capture the tag-transition probabilities\n",
    "- **Outputs**: `alpha`, a pytorch variable containing the scalar score for the entire input, $\\log \\sum_{y_{1:M}} P(w, y)$ \n",
    "- **Tests**: ```test_bilstm_crf.py: test_forward_alg()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 329,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.658s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(765);\n",
    "! nosetests tests/test_bilstm_crf.py:test_forward_alg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 330,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(bilstm);\n",
    "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 331,
   "metadata": {},
   "outputs": [],
   "source": [
    "# passing an input sentence to obtain lstm_feats\n",
    "lstm_feats = model.forward(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 332,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[42.802666]\n"
     ]
    }
   ],
   "source": [
    "alpha = model.forward_alg(lstm_feats)\n",
    "print (alpha.data.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 6.2** (*0.5 points*)\n",
    "Complete the function `bilstm.BiLSTM_CRF.score_sentence()` to obtain the joint log-likelihood score of the particular sequence of tokens and their tags.\n",
    "- **Inputs**:\n",
    "    - `feats`: the hidden state scores for each token in the input sentence. Consider this to be the emission potential of each token for each tag.\n",
    "    - `gold_tags`: a pytorch Variable of the gold sequence of tags: obtain the joint-log-likelihood score of the sequence with the feats and gold_tags.\n",
    "- **Outputs**:\n",
    "    - a pytorch variable of the score. \n",
    "- **Tests**: ```test_bilstm_crf.py: test_score_sentence()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 333,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.639s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(765);\n",
    "! nosetests tests/test_bilstm_crf.py:test_score_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 431,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(bilstm);\n",
    "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 432,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare both the tokens and tags to be passed as input\n",
    "sentence = bilstm.prepare_sequence(X_tr[5], word_to_ix)\n",
    "tags = bilstm.prepare_sequence(Y_tr[5], tag_to_ix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 433,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(765);\n",
    "lstm_feats = model.forward(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 434,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-10.106639]\n"
     ]
    }
   ],
   "source": [
    "score = model.score_sentence(lstm_feats, tags)\n",
    "print (score.data.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Deliverable 6.3 ** (*0.5 points* for 4650, *0.25 points* for 7650) Complete the `bilstm.BiLSTM_CRF.predict()` function to decode the tags using the viterbi algorithm. Make sure to use the viterbi functions defined previously.\n",
    "- **Inputs**:\n",
    "    - `sentence`: a pytorch Variable of sequence of ids of the input_tokens\n",
    "- **Outputs**:\n",
    "    - `best_path`: a list of tags for the sequence of tokens\n",
    "- **Tests**: ```test_bilstm_crf.py: test_predict()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 338,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.611s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "reload(bilstm);\n",
    "! nosetests tests/test_bilstm_crf.py:test_predict\n",
    "torch.manual_seed(765);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['PROPN', 'X', 'PROPN', 'X', 'PROPN', 'X', 'PROPN', 'X', 'PROPN', 'X', 'PROPN', 'X', 'PROPN']\n"
     ]
    }
   ],
   "source": [
    "best_tags = model.predict(sentence)\n",
    "print (best_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deliverable 6.4** (*0.5 points* for 4650, *0.25 points* for 7650)\n",
    "Complete the `bilstm.BiLSTM_CRF.neg_log_likelihood()` function to obtain the negative log likelihood loss. This function calculates the loss function for the `CRF`. Observe that, this can be easily calculated using the previously defined functions.  \n",
    "You should use the `forward(), forward_alg(), score_sentence()` functions defined previously.\n",
    "\n",
    "- **Tests**: ```test_bilstm_crf.py: test_neg_log_likelihood()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.702s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "#setting random seed\n",
    "reload(bilstm);\n",
    "! nosetests tests/test_bilstm_crf.py:test_neg_log_likelihood"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(765);\n",
    "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 443,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm_feats = model.forward(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 344,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 43.0496\n",
      "[torch.FloatTensor of size 1]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "loss = model.neg_log_likelihood(lstm_feats, bilstm.prepare_sequence(Y_tr[5], tag_to_ix))\n",
    "print (loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Putting all the above components together, let us train the `BiLSTM-CRF` model. **Don't worry about matching the scores for the following code blocks from now. Just Make sure that you pass the unit tests.** \n",
    "- You may want to use pretrained embeddings to improve the performance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 439,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.2937723 , -0.32067633, -0.7191576 , ...,  0.30729717,\n",
       "         0.29777217, -0.50366765],\n",
       "       [-0.10237826, -0.24334876, -0.3909557 , ..., -0.15170297,\n",
       "         0.04555601, -0.31999958],\n",
       "       [ 0.05519063, -0.01371501,  0.4883692 , ..., -0.1855296 ,\n",
       "        -0.03656057, -0.3861219 ],\n",
       "       ...,\n",
       "       [ 0.5613533 ,  0.10362698, -0.2936529 , ..., -0.05954225,\n",
       "         0.57234716,  0.2610758 ],\n",
       "       [ 0.024355  ,  0.09903913, -0.11342965, ...,  0.27820015,\n",
       "         0.09090085, -0.10480063],\n",
       "       [-0.2400684 ,  0.0170402 , -0.5328812 , ..., -0.22872141,\n",
       "         0.42429936, -1.0682896 ]], dtype=float32)"
      ]
     },
     "execution_count": 439,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 444,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CONFIG 2\n",
    "reload(bilstm);\n",
    "torch.manual_seed(765);\n",
    "embedding_dim=64\n",
    "hidden_dim=30\n",
    "model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim, embeddings=word_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 445,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1: Dev Accuracy: 0.8239093185712403\n",
      "Epoch 2: Dev Accuracy: 0.8482931329906419\n",
      "Epoch 3: Dev Accuracy: 0.8543561354949255\n",
      "Epoch 4: Dev Accuracy: 0.8564650059311981\n",
      "Epoch 5: Dev Accuracy: 0.8713589033873731\n",
      "Epoch 6: Dev Accuracy: 0.8625280084354817\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-445-8a10c14ed12d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m                                                \u001b[0mX_dv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY_dv\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m                                                \u001b[0mnum_its\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstatus_frequency\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m                                                optim_args = {'lr':0.01,'momentum':0}, param_file = 'best.params')\n\u001b[0m",
      "\u001b[0;32m~/Dev/cs4650/psets/ps2/gtnlplib/bilstm.py\u001b[0m in \u001b[0;36mtrain_model\u001b[0;34m(loss, model, X_tr, Y_tr, word_to_ix, tag_to_ix, X_dv, Y_dv, num_its, status_frequency, optim_args, param_file)\u001b[0m\n\u001b[1;32m    272\u001b[0m             \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlstm_feats\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mY_tr_var\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    273\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 274\u001b[0;31m             \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    275\u001b[0m             \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    276\u001b[0m             \u001b[0mloss_value\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# sending loss function so that we can calculate the loss for the BiLSTM-CRF\n",
    "loss = model.neg_log_likelihood\n",
    "model, losses, accuracies = bilstm.train_model(loss, model, X_tr,Y_tr, word_to_ix, tag_to_ix,\n",
    "                                               X_dv, Y_dv, \n",
    "                                               num_its=10, status_frequency=1,\n",
    "                                               optim_args = {'lr':0.01,'momentum':0}, param_file = 'best.params')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 446,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAACdCAYAAAByzCQ5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8VPW9//HXZ7KSkASSEAKBJGyygyi7WhfU4lJtba0LWG1t7d5e672t7bW9XW732tbfvWqr1hXUqtXWVtyuorYiqyg7yhoChH1JgKzz+f0xQwgYJcEkZzLzfj4e85hzzpwz8zkQDu9853u+X3N3RERERESk5UJBFyAiIiIi0tkoRIuIiIiItJJCtIiIiIhIKylEi4iIiIi0kkK0iIiIiEgrKUSLiIiIiLSSQrSIiIiISCspRIuIiIiItJJCtIiIiIhIKyUHXUBL5Ofne2lpadBliIi02qJFi3a6e4+g6+hIumaLSGfVmmt2pwjRpaWlLFy4MOgyRERazcw2Bl1DR9M1W0Q6q9Zcs9WdQ0RERESklRSiRURERERaKS5D9Oa9h7jmT/OYs3Zn0KWIiIiISAdauGE3l93xOqsrKtv1c+IyROdlprKkfB8z5iZcV0QRERGRhLRp90G++vCbfOoPb7B57yF2VNa06+fFZYhOT0ni8lP78MLybWzfXx10OSIigTOzqWa22szWmNnNzbxebGazzWyxmS0xswuj21PN7D4zW2pmb5vZWR1evIjIB6isruOXz61iym9f5aWV2/jmlEHM/vezOH1Qfrt+blyGaIBpE0uoDzt/XrAp6FJERAJlZknA7cAFwDDgKjMbdsxutwCPufsY4Ergjuj2LwC4+0jgPOBWM4vb/ztEpPNoCDuPzC/j7N+8wp2vrOWikb2Y/e9nceN5J5GR2v4D0HWKIe5ORL/8TE4fmM8j88v4ytkDSQpZ0CWJiARlPLDG3dcBmNmjwKXAiib7OJAdXc4BtkSXhwEvAbj7djPbC4wF5ndA3SIizXp9zU5+8o8VrKqoZGxJd+65dhwn9+3WoTXEdWvCtAnFbNlXzcurtgddiohIkIqApl/LlUe3NfVDYLqZlQOzgK9Ht78NXGpmyWbWDzgV6Nu+5YqING/djio+/8BCpt0zj6qaev736jE8/qVJHR6gIY5bogHOHdaTgqw0ZszdyHnDegZdjohIUJr7Ks6PWb8KuN/dbzWzScBDZjYCuBcYCiwENgJzgPr3fIDZDcANAMXFxW1YuogI7DtYx20vvcuDb2wgPSWJb08dzOdO60d6SlJgNcV1iE5JCnHl+GL+5+V3Kdt1kOK8jKBLEhEJQjlHtx734Uh3jcOuB6YCuPsbZpYO5Lv7duDGwzuZ2Rzg3WM/wN3vAu4CGDt27LEBXUTkhNQ1hJk5dyO/f+ld9h+q44pxffnWeYPpkZUWdGnx3Z0D4KrxfQmZ8fD8sqBLEREJygJgkJn1M7NUIjcOPn3MPmXAFAAzGwqkAzvMLMPMMqPbzwPq3X0FIiLtyN15edU2pv7+NX749xUM753NM984g59fNiomAjTEeUs0QK+cLkwZUsBjCzdx43mDSEsOrtlfRCQI7l5vZl8DngeSgHvdfbmZ/RhY6O5PAzcBd5vZjUS6elzn7m5mBcDzZhYGNgPXBHQaItJK7k75nkM0hJ2SvAzMOscgC6srKvnvZ1bwz3d30j8/k3s+M5YpQwtirv64D9EA0yeW8MKKbTy3rIJLTz72XhoRkc7BzH4D3Ofuy1t7rLvPInLDYNNtP2iyvAI4rZnjNgCDW12siHSocNjZuPsgyzbvY9mWfSzfvJ9lW/ax92AdAIXZ6UwekMekAXlMHphPUbcuAVf8Xjuravjti+/w6PwystJT+MHFw5g+sYTU5NjsOJEQIfr0gfmU5GUwY+5GhWgR6cxWAXeZWTJwH/CIu+8LuCYR6WANYWfdjiqWbt7HsmhYXrFlP1U1kXt+U5NCDC7M4oIRhQzvnYMZvLF2F6++s4MnF28GoCQvIxqq85nUPy/QLhI19Q3c//oG/vflNRysa+Azk0r55pRBdM9MDaymlkiIEB0KGVePL+bnz65iVcV+hhRmH/8gEZEY4+73APeY2WDgs8ASM3sduNvdZwdbnYi0h9r6MO9ur2xsWV62eR8rtu6nui4MQHpKiGG9srnslCJG9M5heFE2gwqy3tN6O21CCe7OO9uqmLN2J3PW7uIfS7byyPzI6Jcn9ezK5AH5TBqQx8R+eeRkpLT7ubk7zy2r4OfPrqJs90HOGVLA9y4cysCCru3+2W0hIUI0wOVj+3Lri+/w8LwyfnzpiKDLERE5IdHZB4dEHzuJjOP8LTP7ortfGWhxIvKhVNc1sLqisjEsL9u8n9UVldQ2RAJz17RkhvXO5urxJYwoymZEUQ798zNJTmpZdwczY3BhFoMLs/jsaf1oCDvLt+xjztpdzFm7iz8v2MT9czZgBiN65zR2/xhXmktmWttGxqXl+/jJMyuYv343g3tm8dD14zljUI82/Yz2ljAhOjczlYtG9uLJNzfznalD2vyHQUSkvZnZb4FLiMwg+DN3Pzxr4C/NbHVwlYlIax2srWfFlv3RPsyR53e3V9EQjowQmdMlhRFF2Xz2tFJGFOUwoiiHktwMQm04A3NSyBjVpxuj+nTjS2cOoLY+zNvle5mzZhdz1u7kvtc38MfX1pEcMkb37dYYqk8p7n7C4zNv21/Nr55bzZOLy8nNSOWnnxjBFWP7tvgXgViSUEly+sRinlq8mb+9tYWrJ2gyABHpdJYBt7j7wWZeG9/RxYhIy4TDzoINu6N9mCOhee2OKjw6onp+11RGFOVw7tCejCjKZnjvHPp079Lho1GkJocYV5rLuNJcvnnuIA7VNrBo457G7h+3z17D/7y8htTkEGNLujf2qR7VJ4eU44TgQ7UN3PXaOv7w6loaws4NH+nPV88eSHZ6+3cbaS8JFaJPKe7OkMIsZszdyFXj+8bcUCkiIsexB2j8H8fMugFnuftfdYOhSGxqCDv//vjbPBW9oa9XTjrDe+dw8ahejOgdaWHumZ0Wk5mkS2oSpw/K5/RB+QDsr65jwfrdvBHt/vGbF94B3iEzNYnx/XIb+1QP65Xd2GIeDjtPv72FXz63iq37qrlwZCE3Tx0aFxPgJVSINjOmTSzh+39dxuJNezmluHvQJYmItMZ/uftTh1fcfa+Z/Rfw1wBrEpH34e7851NLeWrxZr5xzkA+M7mU/K6xMVHIichOT2HK0J5MGdoTgN0Hapm3ble0T/VOZq9eCUC3jBQm9svj1JLu/GPpVt7etJeRRTncduUYxvfLDfIU2lRChWiAT4wp4hezVjJzbplCtIh0Ns19X5pw13GRzsDd+dHfV/Dogk187eyBfOv8+BtuPTczlQtG9uKCkb2ASH/nN6KB+vU1u3hueQU9s9O49fLRfGJMUZv2544FCXfx7ZqWzMfHFPHEonK+f/FQumXE9hiEIiJNLIzeXHg7kVkFvw4sCrYkETmWu/PL51Zz/5wNXH96P246/6SgS+oQPbPT+fiYIj4+JjInx9Z9h+iekXrCNyHGus53K2QbmD6xhJr6ME8sKg+6FBGR1vg6UAv8GXgcqAa+GmhFIvIe//PyGv7w6lqmTSjmlouGxmR/547QK6dL3AZoaMcQbWb3mtl2M1vWZFuumb1oZu9GnwPpTzG0VzanlnRn5rwywtGhZEREYp27H3D3m919rLuf6u7fdfcDQdclIkfc9dpafvviO1x2ShE/uXREwgboRNCeLdH3A1OP2XYz8JK7DyIyzunN7fj5H2j6xGLW7zzAnLW7gipBRKRVzKyHmf3azGaZ2cuHH0HXJSIRD72xgZ/NWsVFo3rxq0+Oirs+wHK0dgvR7v4asPuYzZcCD0SXHwA+3l6ffzwXjOhF94wUZszdGFQJIiKtNRNYBfQDfgRsABYEWZCIRDy+cBPf/9tyzh3ak99fcXKnnDxEWqej/4Z7uvtWgOhzQQd/fqP0lCQuH9uXF1duY9v+6qDKEBFpjTx3/xNQ5+6vuvvngIlBFyWS6J5+ewvf+csSzhiUz/9ePea4E49IfIjZv2Uzu8HMFprZwh07drTLZ1w9vpiGsPPo/E3t8v4iIm2sLvq81cwuMrMxQJ8gCxJJdM8vr+DGP7/F2NJc7rpmbFzfSCdH6+gQvc3MegFEn7e/347uflf05pmxPXr0aJdiSvMzOWNQPo/ML6O+IdwunyEi0ob+28xygJuAfwfuAW4MtiSRxPXK6u18/eHFjCzK4d7rxtElVQE6kXR0iH4auDa6fC3wtw7+/PeYPrGEiv3VvLTqffO8iEjgzCwJGOTu+9x9mbufHR2h4+mgaxNJRG+s3cUXH1rEoJ5deeBz4+malnBTbyS89hzi7hHgDWCwmZWb2fXAL4DzzOxd4LzoeqCmDCmgV066bjAUkZjm7g3AJUHXISKwaOMern9gAcW5GTx0/QRyuqQEXZIEoN1+bXL3q97npSnt9ZknIjkpxJXjivnd/73Dxl0HKMnLDLokEZH3M8fM/pfIZCuN40O7+5vBlSSSWJaW7+O6e+fTMzudmZ+fQG6mZj5OVDF7Y2FHumJcX5JCxsPzyoIuRUTkg0wGhgM/Bm6NPn7TkgPNbKqZrTazNWb2njH6zazYzGab2WIzW2JmF0a3p5jZA2a21MxWmtl32/B8RDqV1RWVXHPvPLK7pDDz8xMoyE4PuiQJkDrwAIU56Zw3tCePLdzEjeedpDtrRSQmufvZJ3JctD/17US60ZUDC8zsaXdf0WS3W4DH3P1OMxsGzAJKgcuBNHcfaWYZwAoze8TdN3yIUxHpdNbuqGLaPfNISw7xyBcm0rtbl6BLkoApREdNn1jCc8sreHbZVj4xRiNGiUjsMbMfNLfd3X98nEPHA2vcfV30fR4lMvlV0xDtQHZ0OQfY0mR7ppklA12AWmD/CZ2ASCe1afdBpt09D3dn5ucnUZyXEXRJEgPUnSNq8oA8+uVnMmOuunSISMw60OTRAFxApLX4eIqApgPil0e3NfVDYLqZlRNphf56dPsT0c/bCpQBv3H3Y2ej7ZCx/UWCsHXfIa66ey6H6hqY8fkJDCzoGnRJEiMUoqNCIWPahGIWbdzDyq1qZBGR2OPutzZ5/BQ4i/eG4eZYc293zPpVwP3u3ge4EHjIzEJEWrEbgN5Ephu/ycz6N1Nbu4/t3x7CYefBNzbw2MJNLCnfy6HahqBLkhiyvbKaaXfPY9/BOh66fjxDe2Uf/yBJGOrO0cSnTu3Dr55fzcx5G/nvj48MuhwRkePJAN4TaJtRDvRtst6HI901DrsemArg7m+YWTqQD1wNPOfudcB2M3sdGAus+5C1x4QXVlTwg78tb1w3g5LcDAYXZjG4MJshhVkMLsyiNC+TpFBzv4tIvNp9oJZr7pnP1n3VPHT9eEb16RZ0SRJjFKKb6JaRysWjevHUm5u5+YKhGjhdRGKKmS3lSAtyEtCDyEgdx7MAGGRm/YDNwJVEwnFTZUSGIL3fzIYC6cCO6PZzzGwGkdA+Efj9hzyVmODu3D57LaV5GfzpunG8u62SVRWVrI4+XlyxjXD0TzstOcSgnl05qWdWNFhHAnZBVhpmCtfxZt+hOj5z7zzW7zrA/deNY2xpbtAlSQxSSjzG9IklPPnmZv66eDPTJ5YEXY6ISFMXN1muB7a5e/3xDnL3ejP7GvA8kfB9r7svN7MfAwujsx7eBNxtZjcSCerXubub2e3AfcAyIt1C7nP3JW17WsH457s7Wbp5H7+4bCQDenRlQI+uTB3Rq/H16roG1myvigbr/ayqqORf7+7kyTc3N+7TLSOFwdFgfVJh9LlnFlnpmnyjs6qqqee6++azuqKSu64Zy+SB+UGXJDFKIfoYY/p2Y1ivbGbM3ci0CcVqYRCRWNILWO7ulQBm1tXMhrv7vOMd6O6ziNww2HTbD5osrwBOa+a4KiLD3MWdO15ZQ2F2Op84pflu5ekpSYwoymFEUc5R2/ccqGX1tkhr9eGA/Zc3N1NVc+T3maJuXRq7ghx+9M/vSmqybkWKZYdqG/j8AwtYUr6P268ew9lDCoIuSWKYQvQxzIzpE0v43lNLebNsD6eW6CscEYkZdwKnNFk/2Mw2aYFFG3czd91uvn/xMNKSWzc3QPfMVCb2z2Ni/7zGbe5O+Z5Dka4g2450CXn1nR3UR/uEpCQZ/fO7NobqIYVZDO+dQ2GOJuyIBTX1DXxxxiLmrd/N7684+ahvJUSaoxDdjEtP7s3PZq1k5twyhWgRiSXm7o2jarh7ODp+s7TSHbPX0j0jhavG9z3+zi1gZvTNzaBvbgbnDuvZuL22Psy6nVWNoXp1RSWLNu7h6beP3Nd5xqB8rp1UytlDCnTzYkDqGsJ87eHFvPbODn71yVFcenJLBr2RRKeLbzMy05K57JQiHl2wie9fPIzumalBlyQiArDOzL5BpPUZ4CvEySgZHWnl1v28tGo7N513Ehmp7fvfYGpyiCGF2QwpPHpotMrqOt7ZVsnra3bx8LwyPv/gQvp078I1E0v49Ni++n+nAzWEnRv//BYvrtjGjy4ZzqfHtc0vVhL/1DnrfUybUEJtfZjHF206/s4iIh3jS8BkIiNslAMTgBsCragTuvOVtWSmJvGZSaWB1ZCVnsKpJbl8Y8og/vmds7lj2ikUdevCz59dxcSfv8R3nljCss37AqsvUYTDznf+soR/LNnKdy8YwrWTS4MuSToRtUS/j8GFWYwr7c7MeWV8/vT+hPQVm4gEzN23ExmeTk7Qhp0H+MeSLXzhI/3JyYiNETRSkkJcOLIXF47sxaqK/Tz4xkaeenMzf164ibEl3fnM5FKmDi/UTYltzN35wdPLeGJROf927iC+eOaAoEuSTqZF/yLN7Jtmlm0RfzKzN83s/PYuLmjTJ5awcddB/rVmZ9CliIhgZg+YWbcm693N7N4ga+ps/vjaWpKTQlx/er+gS2nWkMJsfvaJkcz93hS+f/EwdlbV8I1HFnPaL1/mdy++w7b91UGXGBfcnZ8+s5IZc8v44pn9+eaUQUGXJJ1QS3+t/Zy77wfOJzK4/2eBX7RbVTFi6ohC8jJTmTlvY9CliIgAjHL3vYdX3H0PMCbAejqVin3VPLGonCvG9qUgK7ZHxMjpksL1p/fj5ZvO4r7PjmNE72z+38vvctovXubrjyxm4YbdNLnHVFrpdy++wz3/Ws+1k0q4eeoQDWcrJ6Sl3TkO/3RdSGSg/bctAX7i0pKTuHxsX+7+5zoq9lVrGCIRCVrIzLpHwzNmlou65bXYPf9cR9jhho+0ZKb02BAKGWcPLuDswQVs2HmAGXM38tjCTfz97S0M65XNtZNLuGR0EV1SWzdMXyK7ffYa/t/La7hibF/+62PDFaDlhLW0JXqRmb1AJEQ/b2ZZQLj9yoodV48vJuzOI/PLgi5FRORWYI6Z/cTMfgLMAX4VcE2dwp4DtcycV8alo3vTNzcj6HJOSGl+JrdcPIy535vCzy8bSdid7/xlKRN//hI/m7WSsl0Hgy4xJrg7tfVhKqvr2FlVw+a9h1i/8wCrKvZzxytr+PXzqyND2V42Uvc7yYfS0haM64GTgXXufjDa+vHZ9isrdhTnZfCRQT14dEEZXztnIClJurFDRILh7g+a2SLgbCLfEF4WnWlQjuO+ORs4VNfAl8/q/DePZaQmc9X4Yq4c15f563fz4Bsb+dO/1nP3P9dxzuACPjO5lDMG5sdcQHR3dh2opWz3QfYdqqOmLkxtQ5iaugZq6sPRRwM1dU2W68PUHn6tcb/oc10z+0TXP6iny9Thhdx6+WiNyS0fWktD9CTgLXc/YGbTicyOdVv7lRVbpk8s4QsPLuSllds0g5GIBMrdl5vZDiAdwMyK3V1flX2Aqpp67n99PR8d3pNBPbOCLqfNmBkT+ucxoX8eFfuqeXjeRh6eX8a1986nf34m10wq4ZOn9iE7veNGIampb2DznkOU7T4Yeew6eGR590EO1ja06H1Sk0KkJYdISwmRlpxEWnKI1OQQaSmR5a5pyeRlHnmt6X5pTfaLPJJISwmRmhQiKz2FCf1zSVaDmLSBloboO4HRZjYa+DbwJ+BB4Mz2KiyWnDOkgN456cyYW6YQLSKBMbNLiHTp6A1sB0qAlcDwIOuKdTPnbmR/dT1fOWtg0KW0m8KcdL51/mC+es5AnltWwf1zNvCjv6/g18+v5rJTivjMpFJOaoNfINydPQfrmoTkA0cF5q37q49qBU5LDlGcm0FJXgaTBuRRnJtBcW4G3TNTjwTcY8NyUijmWtFFmtPSEF3v7m5mlwK3ufufzOza9iwsliSFjKvGF3Pri++wfucB+uVnBl2SiCSmnwATgf9z9zFmdjZwVcA1xbTqugbu+dd6Th+Yz+i+3Y5/QCeXlpzEpScXcenJRSwp38uDb2zksYXlzJhbxqT+eVw7uYRzh/b8wJbY2vowW/YeOqoF+XCL8qbdB6msqT9q/x5ZaRTnZjCxfx59oyG5OC+DktwMemSl6cY9iVstDdGVZvZd4BrgDDNLAmJjlPoOcsX4vtz20rs8PG8j/3nRsKDLEZHEVOfuu8wsZGYhd59tZr8MuqhY9sSicnZU1nDblScHXUqHG9WnG7+5vBvfu3Aof16wiRlzN/KlGW/SOyedaRNLmDQg70hYbtLtYsveQ4SbtCanJofo270LxbkZjO+XeyQo52bQN7dLu0+dLhKrWvqTfwVwNZHxoivMrBj4dfuVFXsKstL56PBCHl9Uzk3nDyY9RcMJiUiH22tmXYHXgJlmth2oP84xCau+IcwfXl3LmOJuTOqfF3Q5gcnNTOXLZw3gho/056WV23jwjY38+vnVR+2T3zWVvrkZnFrSncvGFB3VotwzK13dK0Sa0aIQHQ3OM4FxZnYxMN/dH2zf0mLPtInFPLN0K88s2conT+0TdDkiknguBQ4BNwLTgBzgx4FWFMP+vmQL5XsO8UONBQxEuiaeP7yQ84cXsmZ7Fet2VNE3N4O+uRl0TVNrskhrtehfjZl9mkjL8ytEhlX6HzP7D3d/oh1rizmT+ufRv0cmM+ZtVIgWkQ7n7geii2HggSBriXXhsHPnK2sZUpjFOUMKgi4n5gws6MrAgq5BlyHSqbV0jJf/BMa5+7Xu/hlgPPD99isrNpkZ0yaUsLhsL8u37Au6HBEReR//t3Ib72yr4stnDVBXBBFpFy0N0SF3395kfVcrjo0rnzqlD+kpIWbO07CsItJ5mNlUM1ttZmvM7OZmXi82s9lmttjMlpjZhdHt08zsrSaPsJnF9F167s7tr6ylODeDi0ZqWFIRaR8tDcLPmdnzZnadmV0HPAPMar+yYldORgofG9Wbvy7eTGV1XdDliEgCMbNvtmRbM/skAbcDFwDDgKvM7Nhhhm4BHnP3McCVwB0A7j7T3U9295OJjNC0wd3f+nBn0r7mrN3F25v28qUzB2hSDRFpNy26urj7fwB3AaOA0cBd7v6d9iwslk2fWMLB2gb+unhz0KWISGJpbnz+61pw3Hhgjbuvc/da4FEiNyk25UB2dDkH2NLM+1wFPNKyUoNz++w1FGSl8clTi4IuRUTiWItvx3X3vwB/acdaOo3RfbsxsiiHGXPLmD6xRHd9i0i7MrOriAwz2s/Mnm7yUhaR7nXHUwRsarJeDkw4Zp8fAi+Y2deBTODcZt7nCt4bvmPK4rI9zFm7i/+8cChpyRqKVETazweGaDOrJNI68Z6XAHf37GZeOy4z2wBUAg1EZkMceyLvE6RpE4q5+cmlLNy4h3GluUGXIyLxbQ6wFcgnMu33YZXAkhYc39xv+sde268C7nf3W81sEvCQmY1w9zCAmU0ADrr7smY/wOwG4AaA4uLiFpTUPu54ZS05XVK4ekJwNYhIYvjA7hzunuXu2c08sk40QDdxdrSfXacL0ACXnNybrPRkZs7dGHQpIhLn3H2ju7/i7pOADUCKu78KrAS6tOAtyoG+Tdb78N7uGtcDj0U/7w0gnUhoP+xKPqArh7vf5e5j3X1sjx49WlBS21tdUcmLK7bx2dNKydS4xyLSznTHxQnKSE3mk6f0YdbSCnZV1QRdjogkADP7AvAE8Mfopj7AX1tw6AJgkJn1M7NUIoH46WP2KQOmRD9nKJEQvSO6HgIuJ9KXOmbd+coaMlKTuG5yadCliEgCCCpEO5G+d4uiXwG+h5ndYGYLzWzhjh07Ori8lpk2oZjahjCPLyoPuhQRSQxfBU4D9gO4+7vAcWcScfd64GvA80Rarx9z9+Vm9mMzuyS6203AF8zsbSItzte5++EuHx8Byt19XZueTRsq23WQvy/ZyvSJJXTLSA26HBFJAEF933Wau28xswLgRTNb5e6vNd3B3e8iMiIIY8eOba5fduAG9cxiQr9cHp5Xxg1n9NeA/iLS3mrcvfbwzcxmlkzz9628h7vP4pihSd39B02WVxAJ6M0d+wow8cRK7hh/fG0tSWZcf3q/oEsRkQQRSEu0u2+JPm8HniIy/FKnNH1iCWW7D/Lau7HZWi4iceVVM/se0MXMzgMeB/4ecE2B276/mscXlvOpsX3omZ0edDkikiA6PESbWaaZZR1eBs4Hmr3buzP46PBC8rumagZDEekINxPpp7wU+CKRluVbAq0oBtzzr/XUh8N86SMDgi5FRBJIEN05egJPRb+OTAYedvfnAqijTaQmh/j02L784dW1bNl7iN7dWnKjvIhI60WHm7sbuNvMcoE+TfotJ6S9B2uZMXcjl4zuTXFeRtDliEgC6fCW6OiMWaOjj+Hu/tOOrqGtXTW+GAcena/WaBFpP2b2ipllRwP0W8B9ZvbboOsK0gNzNnKwtoEvnzUw6FJEJMFoiLs20Dc3g7MHF/Dogk3UNYSDLkdE4leOu+8HLgPuc/dTaX5mwYRwoKae++as59yhPRlcmBV0OSKSYBSi28j0icVsr6zhxRXbgi5FROJXspn1Aj4N/CPoYoL2yPwy9h6s4ytnqy+0iHQ8heg2cuZJBRR168LMeZrBUETazY+JjPW8xt0XmFl/4N2AawpETX0Dd/9zHZMH5HFKcfegyxGRBKQQ3UaSQsbVE4p5fc0u1u0Fd9pbAAAR1klEQVSoCrocEYlD7v64u49y969E19e5+yeDrisIT765mW37a/jq2eoLLSLBUIhuQ58e25eUJNNwdyIi7ai+IcwfXl3L6D45TB6QF3Q5IpKgFKLbUI+sND46vJDHF25i1tKtHKptCLokEZG488zSrWzcdZCvnD2Qw7M3ioh0tKCm/Y5bXzlrIPPW7+YrM9+kS0oS5wwp4IKRhZwzpICMVP1xi8iJM7Mkd0/o387DYeeO2WsZVNCV84b2DLocEUlgSnVtbFjvbOZ+dwrz1u/i2aUVPLusgmeWbiU9JcRZJxVw4ahenDOkgK5p+qMXkVZbb2bPAX8GXk7EiVZeXrWd1dsq+d0VowmF1AotIsFRkmsHSSFj8oB8Jg/I54eXDGfhht3MWrqVZ5dV8NzyClKTQ5x5Ug8uGtmLKUMLyEpPCbpkEekcBgMfA74K/MnM/gE86u7/CrasjuHu3P7KGvp078LHRvUOuhwRSXAK0e0sKWRM6J/HhP55/NfHhrOobE8kUC+t4MUV20hNCvGRk/K5cGQvpgztSU4XBWoRaZ67HwIeAx4zs+7AbcCrQFKghXWQuet2s7hsLz/5+AiSk3RLj4gESyG6A4VCxrjSXMaV5vL9i4axeNPeaKDeyv+t3E5KknH6wEigPn9YITkZCtQicjQzOxO4ArgAWEBk4pWEcMcra8jvmsblp/YJuhQREYXooIRCxqkl3Tm1pDu3XDSUt6KBetbSCmavXsJ3Q0s5bWA+F44s5PxhhXTPTA26ZBEJmJmtB94i0hr9H+5+IOCSOsyS8r38892dfPeCIaSnJETDu4jEOIXoGGBmjCnuzpji7nzvwqEsKd/HrGVbmbV0K9/5y1K+99QyJg/Ii7ZQ9ySva1rQJYtIMEa7+/6giwjCHbPXkp2ezLSJJUGXIiICKETHHDNjdN9ujO7bjZunDmH5lv08szQSqL/75FJu+esyJvbP5YIRvZg6opB8BWqRRFJoZk8BPd19hJmNAi5x9/8OurD29O62Sp5bXsE3zhmokY1EJGboahTDzIwRRTmMKMrh2x8dzIqt+3l2aQWzlm7llr8u4wd/W8b4frlcNLIXHx1RSEFWetAli0j7uhv4D+CPAO6+xMweBuI6RN/56lq6pCRx3Wn9gi5FRKSRQnQnYWYM753D8N453HT+SazeVsmsJVt5ZulWvv+35fzg6eWMK81lypACRhblMKx3Nt0y1I9aJM5kuPv8Y2bpq2/JgWY2lchoHknAPe7+i2NeLwYeALpF97nZ3WdFXxtFJLhnA2FgnLtXf8hzaZFNuw/yt7e2cN3kUnJ1b4iIxBCF6E7IzBhSmM2Qwmy+df5g3tlWGb0pcSs/f3ZV435F3bowtFc2w3tnM6x3NsN6ZdOnexdNkyvSee00swGAA5jZp4CtxzvIzJKA24HzgHJggZk97e4rmux2C/CYu99pZsOAWUCpmSUDM4Br3P1tM8sD6tr0rD7A3f9cR8jgC2f076iPFBFpEYXoOHBSzyxO6pnFv517Ejurali5dT/Lt+xnxZb9rNi6n5dXbSMcndcsOz2ZYb2zGd47h2G9shlelM2AHl1J0ZirIp3BV4G7gCFmthlYD0xvwXHjgTXuvg7AzB4FLgWahmgn0tIMkANsiS6fDyxx97cB3H3Xhz2JltpeWc2jCzbxyVP6UJij7moiElsUouNMftc0zhjUgzMG9WjcdrC2ntUVlZFgHQ3YM+ZupKY+DEBqcojBPbMY1is7GrCzGdIrWzfwiMSYaAg+18wygZC7V7bw0CJgU5P1cmDCMfv8EHjBzL4OZALnRrefBLiZPQ/0IDJD4q+O/QAzuwG4AaC4uLiFZX2we/+1gfqGMF88c0CbvJ+ISFtSSkoAGanJjUPoHVbfEGbDrgONLdbLt+znhRUV/Hlh5P9ZMyjNy2wM1sN6ZzO8VzYF2WoNEuloZvat99kOgLv/9nhv0cw2P2b9KuB+d7/VzCYBD5nZCCL/T5wOjAMOAi+Z2SJ3f+moN3O/i0grOWPHjj32vVtt38E6ZszdyEWjetMvP/PDvp2ISJtTiE5QyUkhBhZkMbAgi0tPLgLA3anYXx3pBhIN1ks37+OZpUe6XOZ3TWtsrR4W7W9dmpdJKKR+1iLtKCv6PJhImH06uv4x4LUWHF8O9G2y3ocj3TUOux6YCuDub5hZOpAfPfZVd98JYGazgFOAl2hHD76xgaqaer5yllqhRSQ2KURLIzOjV04XeuV0YcrQno3b91fXsTIaqldsjQTse/65jrqGSGNTRmoSQwqzKM3LJK9rKvld08jrmkZe11R6RJ/zMtNITVa/a5ET4e4/AjCzF4BTDnfjMLMfAo+34C0WAIPMrB+wGbgSuPqYfcqAKcD9ZjYUSAd2AM8D3zazDKAWOBP43Yc9pw9ysLae++ZsYMqQAob2yj7+ASIiAVCIluPKTk9hQv88JvTPa9xWU9/Amu1VR93AOH/DbnZW1VBdF36f90kmv2taNGSnHvWc3zWVvCavZaUlaxQRkfcqJhJkD6sFSo93kLvXm9nXiATiJOBed19uZj8GFrr708BNwN1mdiORrh7XubsDe8zst0SCuAOz3P2ZtjypYz06fxO7D9TylbPVCi0isUshWk5IWnJS47jVxzpQU8+uqlp2VNWwq6qGnVW17KqqYdeBI9ve3V7F3HW72HOw+ZGyUpND5GceDtZHAnZ+k/Cdl5lGflYquRmpJGt0EUkMDwHzo7MWOvAJImM7H1d0zOdZx2z7QZPlFcBp73PsDCLD3LW72vowd722jgn9cjm1JLcjPlJE5IQoREuby0xLJjMtmeK8jOPuW9cQZs+BWnZW1bKzqoZdB2rYWVnLzujzrgOREL6qopJdVbXUNjTfyt01LTnySI98dlaT9a5pyWSlv3c9MzWynpWW0rhdXU4klrn7T83sWeCM6KbPuvviIGtqa08tLqdifzW/+tSooEsREflACtESqJSkEAXZ6S0a9cPd2V9df1Tr9s4DkefK6nqqquupqqmnsqaequo6tldWU1UdXa+px1swXkBqcigSwKOhujGQNwnhRwJ6Cl3TkkhLSaJLShLpjc8huqQc2Z6SZOqaIm3G3d8E3gy6jvbQEHb+8Oo6RhblcMag/KDLERH5QArR0mmYGTldUsjpkkL/Hsffvyl352BtAwcaQ3Y0cEefq6rrmgTww9sizxX7q6naUd8YyGvrm28Nfz8ho0nAjoTs5tabC+HHbks/5pi05BBph5+TQ6QlK7RL5/Xssq2s33mAP0w/RT/DIhLzFKIlIZhZYzeTgg/5XjX1DRyoaYiG6jqq68LU1DVwqK6B6row1Y3Lhx/hJuvhxu2H6ho4WFvPrgOR46ubvMehuoYTri9kkT7raSlHgnUkbDdZPrz9ePs0BvSjX09NDpGSZKQkhaIPIzUpRHLSke2pSSENfSgt5u7cPnstA3pkcv6wwqDLERE5LoVokVaKBMokcjNT2+0z3J2a+vBRwftQk6BdEw3atfVhauobqKkPU1PXZLk+Eswbl+sboq9Hlqtq6t93//rwh54no1HIaAzUKdHgnRwKHRXCk5NCpB4TyI8sH7OebKSEQiQ3bou8X0pyiJSQHRXik0MW3f7e/VOTI8/JTcJ/43LISAqpNb+jvbJ6Byu37uc3l4/WL18i0ikoRIvEIDNr7LrR0eobwu8bvg8H+/oGp7YhTF3jwyPP0RBe2xCmrj66Ldxkuem+71kOc7C2oXH56M+I7Fcbff+GNgz67yc1GqyTQ0ZqcojkUIinv3aaZu1sJ7fPXkNRty5cenLvoEsREWkRhWgROUpytGU2My3oSt5fOOzUh/2osF0fbrLcJJwf3q+uwalvEsrrD4f7cNP9m+wT9sZfCg6/V1oAv9QkgvqGMKcNzKdP9y6kaLhKEekkAgnRZjYVuI3IoP/3uPsvgqhDRDqnUMhIjbYQS+eXnBTixvNOCroMEZFW6fD/gcwsCbgduAAYBlxlZsM6ug4RERERkRMVRDPOeGCNu69z91rgUeDSAOoQERERETkhQYToImBTk/Xy6LajmNkNZrbQzBbu2LGjw4oTERERETmeIPpENzd20XtutXf3u4C7AMxsh5ltPIHPygd2nsBxnVkinjMk5nkn4jlD5zvvkqAL6GiLFi3aqWt2iyXiOUNinncinjN0vvNu8TU7iBBdDvRtst4H2PJBB7h7K+enizCzhe4+9kSO7awS8ZwhMc87Ec8ZEve8OxNds1suEc8ZEvO8E/GcIb7PO4juHAuAQWbWz8xSgSuBpwOoQ0RERETkhHR4S7S715vZ14DniQxxd6+7L+/oOkRERERETlQg40S7+yxgVgd81F0d8BmxJhHPGRLzvBPxnCFxzzsRJOLfbSKeMyTmeSfiOUMcn7e5t//0uSIiIiIi8UTTfYmIiIiItFJchmgzm2pmq81sjZndHHQ9HcHM+prZbDNbaWbLzeybQdfUUcwsycwWm9k/gq6lo5hZNzN7wsxWRf/OJwVdU3szsxujP9vLzOwRM0sPuiZpG7pm65od73TNjs9rdtyF6ASeVrweuMndhwITga8myHkDfBNYGXQRHew24Dl3HwKMJs7P38yKgG8AY919BJGbkq8MtippC7pm65qdIHTNjsNrdtyFaBJ0WnF33+rub0aXK4n8A33PTJDxxsz6ABcB9wRdS0cxs2zgI8CfANy91t33BltVh0gGuphZMpDBccaXl05D12xds+Oartnxe82OxxDdomnF45mZlQJjgHnBVtIhfg98GwgHXUgH6g/sAO6LfiV6j5llBl1Ue3L3zcBvgDJgK7DP3V8ItippI7pm65od73TNjtNrdjyG6BZNKx6vzKwr8Bfg39x9f9D1tCczuxjY7u6Lgq6lgyUDpwB3uvsY4AAQ1/1Izaw7kdbJfkBvINPMpgdblbQRXbN1zY53umbH6TU7HkN0q6cVjxdmlkLkYjzT3Z8Mup4OcBpwiZltIPIV8DlmNiPYkjpEOVDu7odbrZ4gcoGOZ+cC6919h7vXAU8CkwOuSdqGrtm6Zsc7XbPj9JodjyE6IacVNzMj0t9qpbv/Nuh6OoK7f9fd+7h7KZG/55fdPe5+0z2Wu1cAm8xscHTTFGBFgCV1hDJgopllRH/WpxDnN+YkEF2zdc2Oa7pmx+81O5AZC9tTAk8rfhpwDbDUzN6KbvtedHZIiT9fB2ZGQ8c64LMB19Ou3H2emT0BvElkVIPFxPEsWIlE12xdsxOErtlxeM3WjIUiIiIiIq0Uj905RERERETalUK0iIiIiEgrKUSLiIiIiLSSQrSIiIiISCspRIuIiIiItJJCtHQ6ZjYn+lxqZle38Xt/r7nPEhGRE6NrtsQrDXEnnZaZnQX8u7tf3Ipjkty94QNer3L3rm1Rn4iIHKFrtsQbtURLp2NmVdHFXwBnmNlbZnajmSWZ2a/NbIGZLTGzL0b3P8vMZpvZw8DS6La/mtkiM1tuZjdEt/0C6BJ9v5lNP8sifm1my8xsqZld0eS9XzGzJ8xslZnNjM7OJCIi6Jot8SvuZiyUhHIzTVo1ohfWfe4+zszSgNfN7IXovuOBEe6+Prr+OXffbWZdgAVm9hd3v9nMvubuJzfzWZcBJwOjgfzoMa9FXxsDDAe2AK8TmYnsX21/uiIinZqu2RJX1BIt8eR84DPRKXTnAXnAoOhr85tcjAG+YWZvA3OBvk32ez+nA4+4e4O7bwNeBcY1ee9ydw8DbwGlbXI2IiLxTdds6dTUEi3xxICvu/vzR22M9MM7cMz6ucAkdz9oZq8A6S147/dT02S5Af27EhFpCV2zpVNTS7R0ZpVAVpP154Evm1kKgJmdZGaZzRyXA+yJXoyHABObvFZ3+PhjvAZcEe3D1wP4CDC/Tc5CRCQx6JotcUW/fUlntgSoj37Fdz9wG5Gv5d6M3iiyA/h4M8c9B3zJzJYAq4l8PXjYXcASM3vT3ac12f4UMAl4G3Dg2+5eEb2gi4jI8emaLXFFQ9yJiIiIiLSSunOIiIiIiLSSQrSIiIiISCspRIuIiIiItJJCtIiIiIhIKylEi4iIiIi0kkK0iIiIiEgrKUSLiIiIiLSSQrSIiIiISCv9f8QaA+58yVaQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x119b11ba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "reload(bilstm);\n",
    "bilstm.plot_results(losses, accuracies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Try and Tune the hyperparameters such that you improve the accuracy on the dev set.\n",
    "You can try changing the number of LSTM layers, the hidden dimension units, the FC layer dimensions, using pretrained embeddings etc.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 399,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reload(bilstm);\n",
    "# torch.manual_seed(765);\n",
    "# embedding_dim=64\n",
    "# hidden_dim=30\n",
    "# model = bilstm.BiLSTM_CRF(len(word_to_ix),tag_to_ix,embedding_dim, hidden_dim, embeddings=word_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 401,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1: Dev Accuracy: 0.8511928298405167\n",
      "Epoch 2: Dev Accuracy: 0.852247265058653\n",
      "Epoch 3: Dev Accuracy: 0.8509292210359826\n",
      "Epoch 4: Dev Accuracy: 0.8506656122314485\n",
      "Epoch 5: Dev Accuracy: 0.852642678265454\n",
      "Epoch 6: Dev Accuracy: 0.8527744826677212\n",
      "Epoch 7: Dev Accuracy: 0.8531698958745222\n",
      "Epoch 8: Dev Accuracy: 0.8551469619085278\n",
      "Epoch 9: Dev Accuracy: 0.8548833531039937\n",
      "Epoch 10: Dev Accuracy: 0.8565968103334651\n"
     ]
    }
   ],
   "source": [
    "# sending loss function so that we can calculate the loss for the BiLSTM-CRF\n",
    "# loss = model.neg_log_likelihood\n",
    "# model, losses, accuracies = bilstm.train_model(loss, model, X_tr,Y_tr, word_to_ix, tag_to_ix,\n",
    "#                                                X_dv, Y_dv, \n",
    "#                                                num_its=10, status_frequency=1,\n",
    "#                                                optim_args = {'lr':0.01,'momentum':0}, param_file = 'best.params')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(bilstm);\n",
    "bilstm.plot_results(losses, accuracies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 447,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(tagger_base);\n",
    "reload(bilstm);\n",
    "confusion = tagger_base.eval_model(model,'bilstm_crf-dev-en.preds', word_to_ix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 448,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8733359694213787\n"
     ]
    }
   ],
   "source": [
    "print (scorer.accuracy(confusion))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 449,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger_base.apply_model(model,'bilstm_crf-te-en.preds',word_to_ix, all_tags, testfile=TEST_FILE_HIDDEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 392,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'data/en-ud-test.conllu'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-392-25fe02b7c77d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# you don't have en-ud-test.conllu, so you can't run this\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mte_confusion\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscorer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_confusion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mTEST_FILE\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'bilstm_crf-te-en.preds'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mprint\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mscorer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maccuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mte_confusion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Dev/cs4650/psets/ps2/gtnlplib/scorer.py\u001b[0m in \u001b[0;36mget_confusion\u001b[0;34m(keyfilename, responsefilename)\u001b[0m\n\u001b[1;32m     24\u001b[0m     \"\"\"\n\u001b[1;32m     25\u001b[0m     \u001b[0mcounts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdefaultdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m     \u001b[0;32mwith\u001b[0m \u001b[0mcodecs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeyfilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'utf8'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mkeyfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     27\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponsefilename\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mresfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mkey_line\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkeyfile\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Dev/Lib/miniconda/lib/python3.6/codecs.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(filename, mode, encoding, errors, buffering)\u001b[0m\n\u001b[1;32m    893\u001b[0m         \u001b[0;31m# Force opening of the file in binary mode\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m         \u001b[0mmode\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'b'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 895\u001b[0;31m     \u001b[0mfile\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbuiltins\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffering\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    896\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mencoding\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfile\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'data/en-ud-test.conllu'"
     ]
    }
   ],
   "source": [
    "# you don't have en-ud-test.conllu, so you can't run this\n",
    "te_confusion = scorer.get_confusion(TEST_FILE,'bilstm_crf-te-en.preds')\n",
    "print (scorer.accuracy(te_confusion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Part of Speech Tagging for Norwegian Language.\n",
    "We will use the same `BiLSTM-CRF` part of speech tagger for Norwegian now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- Running a `BiLSTM-CRF` model on the norwegian language. Make sure your model runs on the norwegian data. \n",
    "- You may want to use the pretrained embeddings for norwegian to improve the performance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 412,
   "metadata": {},
   "outputs": [],
   "source": [
    "if START_TAG not in all_tags_nr:\n",
    "    all_tags_nr.append(START_TAG)\n",
    "if END_TAG not in all_tags_nr:\n",
    "    all_tags_nr.append(END_TAG)\n",
    "all_tags_nr = sorted(all_tags_nr)\n",
    "if START_TAG not in tag_to_ix_nr:\n",
    "    tag_to_ix_nr[START_TAG]=len(tag_to_ix_nr)\n",
    "if END_TAG not in tag_to_ix_nr:\n",
    "    tag_to_ix_nr[END_TAG]=len(tag_to_ix_nr)\n",
    "# we will be using this tag_to_ix and ix_to_tag from now on.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 413,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "words in the vocabulary:  7600\n",
      "7599\n"
     ]
    }
   ],
   "source": [
    "#recalculating the vocab for the norwegian language: obtains the most common 7600 words from the file\n",
    "vocab_nr, word_to_ix_nr = most_common.get_word_to_ix(NR_TRAIN_FILE, 7600)\n",
    "print ('words in the vocabulary: ',len(word_to_ix_nr))\n",
    "print (word_to_ix_nr[UNK])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 416,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7600, 64) 64\n"
     ]
    }
   ],
   "source": [
    "# Get Norwegian Embeddings\n",
    "no_word_embeddings = bilstm.obtain_polyglot_embeddings('data/polyglot-no.pkl', word_to_ix_nr)\n",
    "print(no_word_embeddings.shape, embedding_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 417,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(bilstm);\n",
    "torch.manual_seed(765);\n",
    "embedding_dim=64\n",
    "hidden_dim=30\n",
    "model = bilstm.BiLSTM_CRF(len(word_to_ix_nr),tag_to_ix_nr,embedding_dim, hidden_dim, embeddings=no_word_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 418,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1: Dev Accuracy: 0.8558806655192197\n",
      "Epoch 2: Dev Accuracy: 0.8792885829030407\n",
      "Epoch 3: Dev Accuracy: 0.8773379231210556\n",
      "Epoch 4: Dev Accuracy: 0.8726333907056799\n",
      "Epoch 5: Dev Accuracy: 0.8741250717154332\n",
      "Epoch 6: Dev Accuracy: 0.874469305794607\n",
      "Epoch 7: Dev Accuracy: 0.8860585197934595\n",
      "Epoch 8: Dev Accuracy: 0.8882386689615606\n",
      "Epoch 9: Dev Accuracy: 0.8853700516351118\n",
      "Epoch 10: Dev Accuracy: 0.8916810097532989\n"
     ]
    }
   ],
   "source": [
    "# sending loss function so that we can calculate the loss for the BiLSTM-CRF\n",
    "loss = model.neg_log_likelihood\n",
    "model, losses, accuracies = bilstm.train_model(loss, model, X_tr_nr, Y_tr_nr, word_to_ix_nr, tag_to_ix_nr,\n",
    "                                               X_dv_nr, Y_dv_nr, \n",
    "                                               num_its=10, status_frequency=1,\n",
    "                                               optim_args = {'lr':0.01,'momentum':0}, param_file = 'best.params')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 419,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAACdCAYAAAByzCQ5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8VPW9//HXZ7KSkASSEAKBJGyygyi7WhfU4lJtba0LWG1t7d5e672t7bW9XW732tbfvWqr1hXUqtXWVtyuorYiqyg7yhoChH1JgKzz+f0xQwgYJcEkZzLzfj4e85hzzpwz8zkQDu9853u+X3N3RERERESk5UJBFyAiIiIi0tkoRIuIiIiItJJCtIiIiIhIKylEi4iIiIi0kkK0iIiIiEgrKUSLiIiIiLSSQrSIiIiISCspRIuIiIiItJJCtIiIiIhIKyUHXUBL5Ofne2lpadBliIi02qJFi3a6e4+g6+hIumaLSGfVmmt2pwjRpaWlLFy4MOgyRERazcw2Bl1DR9M1W0Q6q9Zcs9WdQ0RERESklRSiRURERERaKS5D9Oa9h7jmT/OYs3Zn0KWIiIiISAdauGE3l93xOqsrKtv1c+IyROdlprKkfB8z5iZcV0QRERGRhLRp90G++vCbfOoPb7B57yF2VNa06+fFZYhOT0ni8lP78MLybWzfXx10OSIigTOzqWa22szWmNnNzbxebGazzWyxmS0xswuj21PN7D4zW2pmb5vZWR1evIjIB6isruOXz61iym9f5aWV2/jmlEHM/vezOH1Qfrt+blyGaIBpE0uoDzt/XrAp6FJERAJlZknA7cAFwDDgKjMbdsxutwCPufsY4Ergjuj2LwC4+0jgPOBWM4vb/ztEpPNoCDuPzC/j7N+8wp2vrOWikb2Y/e9nceN5J5GR2v4D0HWKIe5ORL/8TE4fmM8j88v4ytkDSQpZ0CWJiARlPLDG3dcBmNmjwKXAiib7OJAdXc4BtkSXhwEvAbj7djPbC4wF5ndA3SIizXp9zU5+8o8VrKqoZGxJd+65dhwn9+3WoTXEdWvCtAnFbNlXzcurtgddiohIkIqApl/LlUe3NfVDYLqZlQOzgK9Ht78NXGpmyWbWDzgV6Nu+5YqING/djio+/8BCpt0zj6qaev736jE8/qVJHR6gIY5bogHOHdaTgqw0ZszdyHnDegZdjohIUJr7Ks6PWb8KuN/dbzWzScBDZjYCuBcYCiwENgJzgPr3fIDZDcANAMXFxW1YuogI7DtYx20vvcuDb2wgPSWJb08dzOdO60d6SlJgNcV1iE5JCnHl+GL+5+V3Kdt1kOK8jKBLEhEJQjlHtx734Uh3jcOuB6YCuPsbZpYO5Lv7duDGwzuZ2Rzg3WM/wN3vAu4CGDt27LEBXUTkhNQ1hJk5dyO/f+ld9h+q44pxffnWeYPpkZUWdGnx3Z0D4KrxfQmZ8fD8sqBLEREJygJgkJn1M7NUIjcOPn3MPmXAFAAzGwqkAzvMLMPMMqPbzwPq3X0FIiLtyN15edU2pv7+NX749xUM753NM984g59fNiomAjTEeUs0QK+cLkwZUsBjCzdx43mDSEsOrtlfRCQI7l5vZl8DngeSgHvdfbmZ/RhY6O5PAzcBd5vZjUS6elzn7m5mBcDzZhYGNgPXBHQaItJK7k75nkM0hJ2SvAzMOscgC6srKvnvZ1bwz3d30j8/k3s+M5YpQwtirv64D9EA0yeW8MKKbTy3rIJLTz72XhoRkc7BzH4D3Ofuy1t7rLvPInLDYNNtP2iyvAI4rZnjNgCDW12siHSocNjZuPsgyzbvY9mWfSzfvJ9lW/ax92AdAIXZ6UwekMekAXlMHphPUbcuAVf8Xjuravjti+/w6PwystJT+MHFw5g+sYTU5NjsOJEQIfr0gfmU5GUwY+5GhWgR6cxWAXeZWTJwH/CIu+8LuCYR6WANYWfdjiqWbt7HsmhYXrFlP1U1kXt+U5NCDC7M4oIRhQzvnYMZvLF2F6++s4MnF28GoCQvIxqq85nUPy/QLhI19Q3c//oG/vflNRysa+Azk0r55pRBdM9MDaymlkiIEB0KGVePL+bnz65iVcV+hhRmH/8gEZEY4+73APeY2WDgs8ASM3sduNvdZwdbnYi0h9r6MO9ur2xsWV62eR8rtu6nui4MQHpKiGG9srnslCJG9M5heFE2gwqy3tN6O21CCe7OO9uqmLN2J3PW7uIfS7byyPzI6Jcn9ezK5AH5TBqQx8R+eeRkpLT7ubk7zy2r4OfPrqJs90HOGVLA9y4cysCCru3+2W0hIUI0wOVj+3Lri+/w8LwyfnzpiKDLERE5IdHZB4dEHzuJjOP8LTP7ortfGWhxIvKhVNc1sLqisjEsL9u8n9UVldQ2RAJz17RkhvXO5urxJYwoymZEUQ798zNJTmpZdwczY3BhFoMLs/jsaf1oCDvLt+xjztpdzFm7iz8v2MT9czZgBiN65zR2/xhXmktmWttGxqXl+/jJMyuYv343g3tm8dD14zljUI82/Yz2ljAhOjczlYtG9uLJNzfznalD2vyHQUSkvZnZb4FLiMwg+DN3Pzxr4C/NbHVwlYlIax2srWfFlv3RPsyR53e3V9EQjowQmdMlhRFF2Xz2tFJGFOUwoiiHktwMQm04A3NSyBjVpxuj+nTjS2cOoLY+zNvle5mzZhdz1u7kvtc38MfX1pEcMkb37dYYqk8p7n7C4zNv21/Nr55bzZOLy8nNSOWnnxjBFWP7tvgXgViSUEly+sRinlq8mb+9tYWrJ2gyABHpdJYBt7j7wWZeG9/RxYhIy4TDzoINu6N9mCOhee2OKjw6onp+11RGFOVw7tCejCjKZnjvHPp079Lho1GkJocYV5rLuNJcvnnuIA7VNrBo457G7h+3z17D/7y8htTkEGNLujf2qR7VJ4eU44TgQ7UN3PXaOv7w6loaws4NH+nPV88eSHZ6+3cbaS8JFaJPKe7OkMIsZszdyFXj+8bcUCkiIsexB2j8H8fMugFnuftfdYOhSGxqCDv//vjbPBW9oa9XTjrDe+dw8ahejOgdaWHumZ0Wk5mkS2oSpw/K5/RB+QDsr65jwfrdvBHt/vGbF94B3iEzNYnx/XIb+1QP65Xd2GIeDjtPv72FXz63iq37qrlwZCE3Tx0aFxPgJVSINjOmTSzh+39dxuJNezmluHvQJYmItMZ/uftTh1fcfa+Z/Rfw1wBrEpH34e7851NLeWrxZr5xzkA+M7mU/K6xMVHIichOT2HK0J5MGdoTgN0Hapm3ble0T/VOZq9eCUC3jBQm9svj1JLu/GPpVt7etJeRRTncduUYxvfLDfIU2lRChWiAT4wp4hezVjJzbplCtIh0Ns19X5pw13GRzsDd+dHfV/Dogk187eyBfOv8+BtuPTczlQtG9uKCkb2ASH/nN6KB+vU1u3hueQU9s9O49fLRfGJMUZv2544FCXfx7ZqWzMfHFPHEonK+f/FQumXE9hiEIiJNLIzeXHg7kVkFvw4sCrYkETmWu/PL51Zz/5wNXH96P246/6SgS+oQPbPT+fiYIj4+JjInx9Z9h+iekXrCNyHGus53K2QbmD6xhJr6ME8sKg+6FBGR1vg6UAv8GXgcqAa+GmhFIvIe//PyGv7w6lqmTSjmlouGxmR/547QK6dL3AZoaMcQbWb3mtl2M1vWZFuumb1oZu9GnwPpTzG0VzanlnRn5rwywtGhZEREYp27H3D3m919rLuf6u7fdfcDQdclIkfc9dpafvviO1x2ShE/uXREwgboRNCeLdH3A1OP2XYz8JK7DyIyzunN7fj5H2j6xGLW7zzAnLW7gipBRKRVzKyHmf3azGaZ2cuHH0HXJSIRD72xgZ/NWsVFo3rxq0+Oirs+wHK0dgvR7v4asPuYzZcCD0SXHwA+3l6ffzwXjOhF94wUZszdGFQJIiKtNRNYBfQDfgRsABYEWZCIRDy+cBPf/9tyzh3ak99fcXKnnDxEWqej/4Z7uvtWgOhzQQd/fqP0lCQuH9uXF1duY9v+6qDKEBFpjTx3/xNQ5+6vuvvngIlBFyWS6J5+ewvf+csSzhiUz/9ePea4E49IfIjZv2Uzu8HMFprZwh07drTLZ1w9vpiGsPPo/E3t8v4iIm2sLvq81cwuMrMxQJ8gCxJJdM8vr+DGP7/F2NJc7rpmbFzfSCdH6+gQvc3MegFEn7e/347uflf05pmxPXr0aJdiSvMzOWNQPo/ML6O+IdwunyEi0ob+28xygJuAfwfuAW4MtiSRxPXK6u18/eHFjCzK4d7rxtElVQE6kXR0iH4auDa6fC3wtw7+/PeYPrGEiv3VvLTqffO8iEjgzCwJGOTu+9x9mbufHR2h4+mgaxNJRG+s3cUXH1rEoJ5deeBz4+malnBTbyS89hzi7hHgDWCwmZWb2fXAL4DzzOxd4LzoeqCmDCmgV066bjAUkZjm7g3AJUHXISKwaOMern9gAcW5GTx0/QRyuqQEXZIEoN1+bXL3q97npSnt9ZknIjkpxJXjivnd/73Dxl0HKMnLDLokEZH3M8fM/pfIZCuN40O7+5vBlSSSWJaW7+O6e+fTMzudmZ+fQG6mZj5OVDF7Y2FHumJcX5JCxsPzyoIuRUTkg0wGhgM/Bm6NPn7TkgPNbKqZrTazNWb2njH6zazYzGab2WIzW2JmF0a3p5jZA2a21MxWmtl32/B8RDqV1RWVXHPvPLK7pDDz8xMoyE4PuiQJkDrwAIU56Zw3tCePLdzEjeedpDtrRSQmufvZJ3JctD/17US60ZUDC8zsaXdf0WS3W4DH3P1OMxsGzAJKgcuBNHcfaWYZwAoze8TdN3yIUxHpdNbuqGLaPfNISw7xyBcm0rtbl6BLkoApREdNn1jCc8sreHbZVj4xRiNGiUjsMbMfNLfd3X98nEPHA2vcfV30fR4lMvlV0xDtQHZ0OQfY0mR7ppklA12AWmD/CZ2ASCe1afdBpt09D3dn5ucnUZyXEXRJEgPUnSNq8oA8+uVnMmOuunSISMw60OTRAFxApLX4eIqApgPil0e3NfVDYLqZlRNphf56dPsT0c/bCpQBv3H3Y2ej7ZCx/UWCsHXfIa66ey6H6hqY8fkJDCzoGnRJEiMUoqNCIWPahGIWbdzDyq1qZBGR2OPutzZ5/BQ4i/eG4eZYc293zPpVwP3u3ge4EHjIzEJEWrEbgN5Ephu/ycz6N1Nbu4/t3x7CYefBNzbw2MJNLCnfy6HahqBLkhiyvbKaaXfPY9/BOh66fjxDe2Uf/yBJGOrO0cSnTu3Dr55fzcx5G/nvj48MuhwRkePJAN4TaJtRDvRtst6HI901DrsemArg7m+YWTqQD1wNPOfudcB2M3sdGAus+5C1x4QXVlTwg78tb1w3g5LcDAYXZjG4MJshhVkMLsyiNC+TpFBzv4tIvNp9oJZr7pnP1n3VPHT9eEb16RZ0SRJjFKKb6JaRysWjevHUm5u5+YKhGjhdRGKKmS3lSAtyEtCDyEgdx7MAGGRm/YDNwJVEwnFTZUSGIL3fzIYC6cCO6PZzzGwGkdA+Efj9hzyVmODu3D57LaV5GfzpunG8u62SVRWVrI4+XlyxjXD0TzstOcSgnl05qWdWNFhHAnZBVhpmCtfxZt+hOj5z7zzW7zrA/deNY2xpbtAlSQxSSjzG9IklPPnmZv66eDPTJ5YEXY6ISFMXN1muB7a5e/3xDnL3ejP7GvA8kfB9r7svN7MfAwujsx7eBNxtZjcSCerXubub2e3AfcAyIt1C7nP3JW17WsH457s7Wbp5H7+4bCQDenRlQI+uTB3Rq/H16roG1myvigbr/ayqqORf7+7kyTc3N+7TLSOFwdFgfVJh9LlnFlnpmnyjs6qqqee6++azuqKSu64Zy+SB+UGXJDFKIfoYY/p2Y1ivbGbM3ci0CcVqYRCRWNILWO7ulQBm1tXMhrv7vOMd6O6ziNww2HTbD5osrwBOa+a4KiLD3MWdO15ZQ2F2Op84pflu5ekpSYwoymFEUc5R2/ccqGX1tkhr9eGA/Zc3N1NVc+T3maJuXRq7ghx+9M/vSmqybkWKZYdqG/j8AwtYUr6P268ew9lDCoIuSWKYQvQxzIzpE0v43lNLebNsD6eW6CscEYkZdwKnNFk/2Mw2aYFFG3czd91uvn/xMNKSWzc3QPfMVCb2z2Ni/7zGbe5O+Z5Dka4g2450CXn1nR3UR/uEpCQZ/fO7NobqIYVZDO+dQ2GOJuyIBTX1DXxxxiLmrd/N7684+ahvJUSaoxDdjEtP7s3PZq1k5twyhWgRiSXm7o2jarh7ODp+s7TSHbPX0j0jhavG9z3+zi1gZvTNzaBvbgbnDuvZuL22Psy6nVWNoXp1RSWLNu7h6beP3Nd5xqB8rp1UytlDCnTzYkDqGsJ87eHFvPbODn71yVFcenJLBr2RRKeLbzMy05K57JQiHl2wie9fPIzumalBlyQiArDOzL5BpPUZ4CvEySgZHWnl1v28tGo7N513Ehmp7fvfYGpyiCGF2QwpPHpotMrqOt7ZVsnra3bx8LwyPv/gQvp078I1E0v49Ni++n+nAzWEnRv//BYvrtjGjy4ZzqfHtc0vVhL/1DnrfUybUEJtfZjHF206/s4iIh3jS8BkIiNslAMTgBsCragTuvOVtWSmJvGZSaWB1ZCVnsKpJbl8Y8og/vmds7lj2ikUdevCz59dxcSfv8R3nljCss37AqsvUYTDznf+soR/LNnKdy8YwrWTS4MuSToRtUS/j8GFWYwr7c7MeWV8/vT+hPQVm4gEzN23ExmeTk7Qhp0H+MeSLXzhI/3JyYiNETRSkkJcOLIXF47sxaqK/Tz4xkaeenMzf164ibEl3fnM5FKmDi/UTYltzN35wdPLeGJROf927iC+eOaAoEuSTqZF/yLN7Jtmlm0RfzKzN83s/PYuLmjTJ5awcddB/rVmZ9CliIhgZg+YWbcm693N7N4ga+ps/vjaWpKTQlx/er+gS2nWkMJsfvaJkcz93hS+f/EwdlbV8I1HFnPaL1/mdy++w7b91UGXGBfcnZ8+s5IZc8v44pn9+eaUQUGXJJ1QS3+t/Zy77wfOJzK4/2eBX7RbVTFi6ohC8jJTmTlvY9CliIgAjHL3vYdX3H0PMCbAejqVin3VPLGonCvG9qUgK7ZHxMjpksL1p/fj5ZvO4r7PjmNE72z+38vvctovXubrjyxm4YbdNLnHVFrpdy++wz3/Ws+1k0q4eeoQDWcrJ6Sl3TkO/3RdSGSg/bctAX7i0pKTuHxsX+7+5zoq9lVrGCIRCVrIzLpHwzNmlou65bXYPf9cR9jhho+0ZKb02BAKGWcPLuDswQVs2HmAGXM38tjCTfz97S0M65XNtZNLuGR0EV1SWzdMXyK7ffYa/t/La7hibF/+62PDFaDlhLW0JXqRmb1AJEQ/b2ZZQLj9yoodV48vJuzOI/PLgi5FRORWYI6Z/cTMfgLMAX4VcE2dwp4DtcycV8alo3vTNzcj6HJOSGl+JrdcPIy535vCzy8bSdid7/xlKRN//hI/m7WSsl0Hgy4xJrg7tfVhKqvr2FlVw+a9h1i/8wCrKvZzxytr+PXzqyND2V42Uvc7yYfS0haM64GTgXXufjDa+vHZ9isrdhTnZfCRQT14dEEZXztnIClJurFDRILh7g+a2SLgbCLfEF4WnWlQjuO+ORs4VNfAl8/q/DePZaQmc9X4Yq4c15f563fz4Bsb+dO/1nP3P9dxzuACPjO5lDMG5sdcQHR3dh2opWz3QfYdqqOmLkxtQ5iaugZq6sPRRwM1dU2W68PUHn6tcb/oc10z+0TXP6iny9Thhdx6+WiNyS0fWktD9CTgLXc/YGbTicyOdVv7lRVbpk8s4QsPLuSllds0g5GIBMrdl5vZDiAdwMyK3V1flX2Aqpp67n99PR8d3pNBPbOCLqfNmBkT+ucxoX8eFfuqeXjeRh6eX8a1986nf34m10wq4ZOn9iE7veNGIampb2DznkOU7T4Yeew6eGR590EO1ja06H1Sk0KkJYdISwmRlpxEWnKI1OQQaSmR5a5pyeRlHnmt6X5pTfaLPJJISwmRmhQiKz2FCf1zSVaDmLSBloboO4HRZjYa+DbwJ+BB4Mz2KiyWnDOkgN456cyYW6YQLSKBMbNLiHTp6A1sB0qAlcDwIOuKdTPnbmR/dT1fOWtg0KW0m8KcdL51/mC+es5AnltWwf1zNvCjv6/g18+v5rJTivjMpFJOaoNfINydPQfrmoTkA0cF5q37q49qBU5LDlGcm0FJXgaTBuRRnJtBcW4G3TNTjwTcY8NyUijmWtFFmtPSEF3v7m5mlwK3ufufzOza9iwsliSFjKvGF3Pri++wfucB+uVnBl2SiCSmnwATgf9z9zFmdjZwVcA1xbTqugbu+dd6Th+Yz+i+3Y5/QCeXlpzEpScXcenJRSwp38uDb2zksYXlzJhbxqT+eVw7uYRzh/b8wJbY2vowW/YeOqoF+XCL8qbdB6msqT9q/x5ZaRTnZjCxfx59oyG5OC+DktwMemSl6cY9iVstDdGVZvZd4BrgDDNLAmJjlPoOcsX4vtz20rs8PG8j/3nRsKDLEZHEVOfuu8wsZGYhd59tZr8MuqhY9sSicnZU1nDblScHXUqHG9WnG7+5vBvfu3Aof16wiRlzN/KlGW/SOyedaRNLmDQg70hYbtLtYsveQ4SbtCanJofo270LxbkZjO+XeyQo52bQN7dLu0+dLhKrWvqTfwVwNZHxoivMrBj4dfuVFXsKstL56PBCHl9Uzk3nDyY9RcMJiUiH22tmXYHXgJlmth2oP84xCau+IcwfXl3LmOJuTOqfF3Q5gcnNTOXLZw3gho/056WV23jwjY38+vnVR+2T3zWVvrkZnFrSncvGFB3VotwzK13dK0Sa0aIQHQ3OM4FxZnYxMN/dH2zf0mLPtInFPLN0K88s2conT+0TdDkiknguBQ4BNwLTgBzgx4FWFMP+vmQL5XsO8UONBQxEuiaeP7yQ84cXsmZ7Fet2VNE3N4O+uRl0TVNrskhrtehfjZl9mkjL8ytEhlX6HzP7D3d/oh1rizmT+ufRv0cmM+ZtVIgWkQ7n7geii2HggSBriXXhsHPnK2sZUpjFOUMKgi4n5gws6MrAgq5BlyHSqbV0jJf/BMa5+7Xu/hlgPPD99isrNpkZ0yaUsLhsL8u37Au6HBEReR//t3Ib72yr4stnDVBXBBFpFy0N0SF3395kfVcrjo0rnzqlD+kpIWbO07CsItJ5mNlUM1ttZmvM7OZmXi82s9lmttjMlpjZhdHt08zsrSaPsJnF9F167s7tr6ylODeDi0ZqWFIRaR8tDcLPmdnzZnadmV0HPAPMar+yYldORgofG9Wbvy7eTGV1XdDliEgCMbNvtmRbM/skAbcDFwDDgKvM7Nhhhm4BHnP3McCVwB0A7j7T3U9295OJjNC0wd3f+nBn0r7mrN3F25v28qUzB2hSDRFpNy26urj7fwB3AaOA0cBd7v6d9iwslk2fWMLB2gb+unhz0KWISGJpbnz+61pw3Hhgjbuvc/da4FEiNyk25UB2dDkH2NLM+1wFPNKyUoNz++w1FGSl8clTi4IuRUTiWItvx3X3vwB/acdaOo3RfbsxsiiHGXPLmD6xRHd9i0i7MrOriAwz2s/Mnm7yUhaR7nXHUwRsarJeDkw4Zp8fAi+Y2deBTODcZt7nCt4bvmPK4rI9zFm7i/+8cChpyRqKVETazweGaDOrJNI68Z6XAHf37GZeOy4z2wBUAg1EZkMceyLvE6RpE4q5+cmlLNy4h3GluUGXIyLxbQ6wFcgnMu33YZXAkhYc39xv+sde268C7nf3W81sEvCQmY1w9zCAmU0ADrr7smY/wOwG4AaA4uLiFpTUPu54ZS05XVK4ekJwNYhIYvjA7hzunuXu2c08sk40QDdxdrSfXacL0ACXnNybrPRkZs7dGHQpIhLn3H2ju7/i7pOADUCKu78KrAS6tOAtyoG+Tdb78N7uGtcDj0U/7w0gnUhoP+xKPqArh7vf5e5j3X1sjx49WlBS21tdUcmLK7bx2dNKydS4xyLSznTHxQnKSE3mk6f0YdbSCnZV1QRdjogkADP7AvAE8Mfopj7AX1tw6AJgkJn1M7NUIoH46WP2KQOmRD9nKJEQvSO6HgIuJ9KXOmbd+coaMlKTuG5yadCliEgCCCpEO5G+d4uiXwG+h5ndYGYLzWzhjh07Ori8lpk2oZjahjCPLyoPuhQRSQxfBU4D9gO4+7vAcWcScfd64GvA80Rarx9z9+Vm9mMzuyS6203AF8zsbSItzte5++EuHx8Byt19XZueTRsq23WQvy/ZyvSJJXTLSA26HBFJAEF933Wau28xswLgRTNb5e6vNd3B3e8iMiIIY8eOba5fduAG9cxiQr9cHp5Xxg1n9NeA/iLS3mrcvfbwzcxmlkzz9628h7vP4pihSd39B02WVxAJ6M0d+wow8cRK7hh/fG0tSWZcf3q/oEsRkQQRSEu0u2+JPm8HniIy/FKnNH1iCWW7D/Lau7HZWi4iceVVM/se0MXMzgMeB/4ecE2B276/mscXlvOpsX3omZ0edDkikiA6PESbWaaZZR1eBs4Hmr3buzP46PBC8rumagZDEekINxPpp7wU+CKRluVbAq0oBtzzr/XUh8N86SMDgi5FRBJIEN05egJPRb+OTAYedvfnAqijTaQmh/j02L784dW1bNl7iN7dWnKjvIhI60WHm7sbuNvMcoE+TfotJ6S9B2uZMXcjl4zuTXFeRtDliEgC6fCW6OiMWaOjj+Hu/tOOrqGtXTW+GAcena/WaBFpP2b2ipllRwP0W8B9ZvbboOsK0gNzNnKwtoEvnzUw6FJEJMFoiLs20Dc3g7MHF/Dogk3UNYSDLkdE4leOu+8HLgPuc/dTaX5mwYRwoKae++as59yhPRlcmBV0OSKSYBSi28j0icVsr6zhxRXbgi5FROJXspn1Aj4N/CPoYoL2yPwy9h6s4ytnqy+0iHQ8heg2cuZJBRR168LMeZrBUETazY+JjPW8xt0XmFl/4N2AawpETX0Dd/9zHZMH5HFKcfegyxGRBKQQ3UaSQsbVE4p5fc0u1u0Fd9pbAAAR1klEQVSoCrocEYlD7v64u49y969E19e5+yeDrisIT765mW37a/jq2eoLLSLBUIhuQ58e25eUJNNwdyIi7ai+IcwfXl3L6D45TB6QF3Q5IpKgFKLbUI+sND46vJDHF25i1tKtHKptCLokEZG488zSrWzcdZCvnD2Qw7M3ioh0tKCm/Y5bXzlrIPPW7+YrM9+kS0oS5wwp4IKRhZwzpICMVP1xi8iJM7Mkd0/o387DYeeO2WsZVNCV84b2DLocEUlgSnVtbFjvbOZ+dwrz1u/i2aUVPLusgmeWbiU9JcRZJxVw4ahenDOkgK5p+qMXkVZbb2bPAX8GXk7EiVZeXrWd1dsq+d0VowmF1AotIsFRkmsHSSFj8oB8Jg/I54eXDGfhht3MWrqVZ5dV8NzyClKTQ5x5Ug8uGtmLKUMLyEpPCbpkEekcBgMfA74K/MnM/gE86u7/CrasjuHu3P7KGvp078LHRvUOuhwRSXAK0e0sKWRM6J/HhP55/NfHhrOobE8kUC+t4MUV20hNCvGRk/K5cGQvpgztSU4XBWoRaZ67HwIeAx4zs+7AbcCrQFKghXWQuet2s7hsLz/5+AiSk3RLj4gESyG6A4VCxrjSXMaV5vL9i4axeNPeaKDeyv+t3E5KknH6wEigPn9YITkZCtQicjQzOxO4ArgAWEBk4pWEcMcra8jvmsblp/YJuhQREYXooIRCxqkl3Tm1pDu3XDSUt6KBetbSCmavXsJ3Q0s5bWA+F44s5PxhhXTPTA26ZBEJmJmtB94i0hr9H+5+IOCSOsyS8r38892dfPeCIaSnJETDu4jEOIXoGGBmjCnuzpji7nzvwqEsKd/HrGVbmbV0K9/5y1K+99QyJg/Ii7ZQ9ySva1rQJYtIMEa7+/6giwjCHbPXkp2ezLSJJUGXIiICKETHHDNjdN9ujO7bjZunDmH5lv08szQSqL/75FJu+esyJvbP5YIRvZg6opB8BWqRRFJoZk8BPd19hJmNAi5x9/8OurD29O62Sp5bXsE3zhmokY1EJGboahTDzIwRRTmMKMrh2x8dzIqt+3l2aQWzlm7llr8u4wd/W8b4frlcNLIXHx1RSEFWetAli0j7uhv4D+CPAO6+xMweBuI6RN/56lq6pCRx3Wn9gi5FRKSRQnQnYWYM753D8N453HT+SazeVsmsJVt5ZulWvv+35fzg6eWMK81lypACRhblMKx3Nt0y1I9aJM5kuPv8Y2bpq2/JgWY2lchoHknAPe7+i2NeLwYeALpF97nZ3WdFXxtFJLhnA2FgnLtXf8hzaZFNuw/yt7e2cN3kUnJ1b4iIxBCF6E7IzBhSmM2Qwmy+df5g3tlWGb0pcSs/f3ZV435F3bowtFc2w3tnM6x3NsN6ZdOnexdNkyvSee00swGAA5jZp4CtxzvIzJKA24HzgHJggZk97e4rmux2C/CYu99pZsOAWUCpmSUDM4Br3P1tM8sD6tr0rD7A3f9cR8jgC2f076iPFBFpEYXoOHBSzyxO6pnFv517Ejurali5dT/Lt+xnxZb9rNi6n5dXbSMcndcsOz2ZYb2zGd47h2G9shlelM2AHl1J0ZirIp3BV4G7gCFmthlYD0xvwXHjgTXuvg7AzB4FLgWahmgn0tIMkANsiS6fDyxx97cB3H3Xhz2JltpeWc2jCzbxyVP6UJij7moiElsUouNMftc0zhjUgzMG9WjcdrC2ntUVlZFgHQ3YM+ZupKY+DEBqcojBPbMY1is7GrCzGdIrWzfwiMSYaAg+18wygZC7V7bw0CJgU5P1cmDCMfv8EHjBzL4OZALnRrefBLiZPQ/0IDJD4q+O/QAzuwG4AaC4uLiFZX2we/+1gfqGMF88c0CbvJ+ISFtSSkoAGanJjUPoHVbfEGbDrgONLdbLt+znhRUV/Hlh5P9ZMyjNy2wM1sN6ZzO8VzYF2WoNEuloZvat99kOgLv/9nhv0cw2P2b9KuB+d7/VzCYBD5nZCCL/T5wOjAMOAi+Z2SJ3f+moN3O/i0grOWPHjj32vVtt38E6ZszdyEWjetMvP/PDvp2ISJtTiE5QyUkhBhZkMbAgi0tPLgLA3anYXx3pBhIN1ks37+OZpUe6XOZ3TWtsrR4W7W9dmpdJKKR+1iLtKCv6PJhImH06uv4x4LUWHF8O9G2y3ocj3TUOux6YCuDub5hZOpAfPfZVd98JYGazgFOAl2hHD76xgaqaer5yllqhRSQ2KURLIzOjV04XeuV0YcrQno3b91fXsTIaqldsjQTse/65jrqGSGNTRmoSQwqzKM3LJK9rKvld08jrmkZe11R6RJ/zMtNITVa/a5ET4e4/AjCzF4BTDnfjMLMfAo+34C0WAIPMrB+wGbgSuPqYfcqAKcD9ZjYUSAd2AM8D3zazDKAWOBP43Yc9pw9ysLae++ZsYMqQAob2yj7+ASIiAVCIluPKTk9hQv88JvTPa9xWU9/Amu1VR93AOH/DbnZW1VBdF36f90kmv2taNGSnHvWc3zWVvCavZaUlaxQRkfcqJhJkD6sFSo93kLvXm9nXiATiJOBed19uZj8GFrr708BNwN1mdiORrh7XubsDe8zst0SCuAOz3P2ZtjypYz06fxO7D9TylbPVCi0isUshWk5IWnJS47jVxzpQU8+uqlp2VNWwq6qGnVW17KqqYdeBI9ve3V7F3HW72HOw+ZGyUpND5GceDtZHAnZ+k/Cdl5lGflYquRmpJGt0EUkMDwHzo7MWOvAJImM7H1d0zOdZx2z7QZPlFcBp73PsDCLD3LW72vowd722jgn9cjm1JLcjPlJE5IQoREuby0xLJjMtmeK8jOPuW9cQZs+BWnZW1bKzqoZdB2rYWVnLzujzrgOREL6qopJdVbXUNjTfyt01LTnySI98dlaT9a5pyWSlv3c9MzWynpWW0rhdXU4klrn7T83sWeCM6KbPuvviIGtqa08tLqdifzW/+tSooEsREflACtESqJSkEAXZ6S0a9cPd2V9df1Tr9s4DkefK6nqqquupqqmnsqaequo6tldWU1UdXa+px1swXkBqcigSwKOhujGQNwnhRwJ6Cl3TkkhLSaJLShLpjc8huqQc2Z6SZOqaIm3G3d8E3gy6jvbQEHb+8Oo6RhblcMag/KDLERH5QArR0mmYGTldUsjpkkL/Hsffvyl352BtAwcaQ3Y0cEefq6rrmgTww9sizxX7q6naUd8YyGvrm28Nfz8ho0nAjoTs5tabC+HHbks/5pi05BBph5+TQ6QlK7RL5/Xssq2s33mAP0w/RT/DIhLzFKIlIZhZYzeTgg/5XjX1DRyoaYiG6jqq68LU1DVwqK6B6row1Y3Lhx/hJuvhxu2H6ho4WFvPrgOR46ubvMehuoYTri9kkT7raSlHgnUkbDdZPrz9ePs0BvSjX09NDpGSZKQkhaIPIzUpRHLSke2pSSENfSgt5u7cPnstA3pkcv6wwqDLERE5LoVokVaKBMokcjNT2+0z3J2a+vBRwftQk6BdEw3atfVhauobqKkPU1PXZLk+Eswbl+sboq9Hlqtq6t93//rwh54no1HIaAzUKdHgnRwKHRXCk5NCpB4TyI8sH7OebKSEQiQ3bou8X0pyiJSQHRXik0MW3f7e/VOTI8/JTcJ/43LISAqpNb+jvbJ6Byu37uc3l4/WL18i0ikoRIvEIDNr7LrR0eobwu8bvg8H+/oGp7YhTF3jwyPP0RBe2xCmrj66Ldxkuem+71kOc7C2oXH56M+I7Fcbff+GNgz67yc1GqyTQ0ZqcojkUIinv3aaZu1sJ7fPXkNRty5cenLvoEsREWkRhWgROUpytGU2My3oSt5fOOzUh/2osF0fbrLcJJwf3q+uwalvEsrrD4f7cNP9m+wT9sZfCg6/V1oAv9QkgvqGMKcNzKdP9y6kaLhKEekkAgnRZjYVuI3IoP/3uPsvgqhDRDqnUMhIjbYQS+eXnBTixvNOCroMEZFW6fD/gcwsCbgduAAYBlxlZsM6ug4RERERkRMVRDPOeGCNu69z91rgUeDSAOoQERERETkhQYToImBTk/Xy6LajmNkNZrbQzBbu2LGjw4oTERERETmeIPpENzd20XtutXf3u4C7AMxsh5ltPIHPygd2nsBxnVkinjMk5nkn4jlD5zvvkqAL6GiLFi3aqWt2iyXiOUNinncinjN0vvNu8TU7iBBdDvRtst4H2PJBB7h7K+enizCzhe4+9kSO7awS8ZwhMc87Ec8ZEve8OxNds1suEc8ZEvO8E/GcIb7PO4juHAuAQWbWz8xSgSuBpwOoQ0RERETkhHR4S7S715vZ14DniQxxd6+7L+/oOkRERERETlQg40S7+yxgVgd81F0d8BmxJhHPGRLzvBPxnCFxzzsRJOLfbSKeMyTmeSfiOUMcn7e5t//0uSIiIiIi8UTTfYmIiIiItFJchmgzm2pmq81sjZndHHQ9HcHM+prZbDNbaWbLzeybQdfUUcwsycwWm9k/gq6lo5hZNzN7wsxWRf/OJwVdU3szsxujP9vLzOwRM0sPuiZpG7pm65od73TNjs9rdtyF6ASeVrweuMndhwITga8myHkDfBNYGXQRHew24Dl3HwKMJs7P38yKgG8AY919BJGbkq8MtippC7pm65qdIHTNjsNrdtyFaBJ0WnF33+rub0aXK4n8A33PTJDxxsz6ABcB9wRdS0cxs2zgI8CfANy91t33BltVh0gGuphZMpDBccaXl05D12xds+Oartnxe82OxxDdomnF45mZlQJjgHnBVtIhfg98GwgHXUgH6g/sAO6LfiV6j5llBl1Ue3L3zcBvgDJgK7DP3V8ItippI7pm65od73TNjtNrdjyG6BZNKx6vzKwr8Bfg39x9f9D1tCczuxjY7u6Lgq6lgyUDpwB3uvsY4AAQ1/1Izaw7kdbJfkBvINPMpgdblbQRXbN1zY53umbH6TU7HkN0q6cVjxdmlkLkYjzT3Z8Mup4OcBpwiZltIPIV8DlmNiPYkjpEOVDu7odbrZ4gcoGOZ+cC6919h7vXAU8CkwOuSdqGrtm6Zsc7XbPj9JodjyE6IacVNzMj0t9qpbv/Nuh6OoK7f9fd+7h7KZG/55fdPe5+0z2Wu1cAm8xscHTTFGBFgCV1hDJgopllRH/WpxDnN+YkEF2zdc2Oa7pmx+81O5AZC9tTAk8rfhpwDbDUzN6KbvtedHZIiT9fB2ZGQ8c64LMB19Ou3H2emT0BvElkVIPFxPEsWIlE12xdsxOErtlxeM3WjIUiIiIiIq0Uj905RERERETalUK0iIiIiEgrKUSLiIiIiLSSQrSIiIiISCspRIuIiIiItJJCtHQ6ZjYn+lxqZle38Xt/r7nPEhGRE6NrtsQrDXEnnZaZnQX8u7tf3Ipjkty94QNer3L3rm1Rn4iIHKFrtsQbtURLp2NmVdHFXwBnmNlbZnajmSWZ2a/NbIGZLTGzL0b3P8vMZpvZw8DS6La/mtkiM1tuZjdEt/0C6BJ9v5lNP8sifm1my8xsqZld0eS9XzGzJ8xslZnNjM7OJCIi6Jot8SvuZiyUhHIzTVo1ohfWfe4+zszSgNfN7IXovuOBEe6+Prr+OXffbWZdgAVm9hd3v9nMvubuJzfzWZcBJwOjgfzoMa9FXxsDDAe2AK8TmYnsX21/uiIinZqu2RJX1BIt8eR84DPRKXTnAXnAoOhr85tcjAG+YWZvA3OBvk32ez+nA4+4e4O7bwNeBcY1ee9ydw8DbwGlbXI2IiLxTdds6dTUEi3xxICvu/vzR22M9MM7cMz6ucAkdz9oZq8A6S147/dT02S5Af27EhFpCV2zpVNTS7R0ZpVAVpP154Evm1kKgJmdZGaZzRyXA+yJXoyHABObvFZ3+PhjvAZcEe3D1wP4CDC/Tc5CRCQx6JotcUW/fUlntgSoj37Fdz9wG5Gv5d6M3iiyA/h4M8c9B3zJzJYAq4l8PXjYXcASM3vT3ac12f4UMAl4G3Dg2+5eEb2gi4jI8emaLXFFQ9yJiIiIiLSSunOIiIiIiLSSQrSIiIiISCspRIuIiIiItJJCtIiIiIhIKylEi4iIiIi0kkK0iIiIiEgrKUSLiIiIiLSSQrSIiIiISCv9f8QaA+58yVaQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1122e76a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "reload(bilstm);\n",
    "bilstm.plot_results(losses, accuracies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 423,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(tagger_base);\n",
    "reload(bilstm);\n",
    "confusion = tagger_base.eval_model(model,'bilstm_crf-dev-nr.preds', word_to_ix_nr, \n",
    "                                   trainfile=NR_TRAIN_FILE, testfile=NR_DEV_FILE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 424,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8937464142283419\n"
     ]
    }
   ],
   "source": [
    "print (scorer.accuracy(confusion))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 425,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger_base.apply_model(model,'bilstm_crf-te-nr.preds',word_to_ix_nr, all_tags_nr, \n",
    "                        trainfile=NR_TRAIN_FILE, testfile=NR_TEST_FILE_HIDDEN)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you don't have no_bokmaal-ud-test.conllu, so you can't run this\n",
    "te_confusion = scorer.get_confusion(NR_TEST_FILE,'bilstm_crf-te-nr.preds')\n",
    "print (scorer.accuracy(te_confusion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7. Improve your Taggers performance\n",
    "==================\n",
    "**Deliverable 7.1** (*2.0 points*)\n",
    "Improve your taggers performance to get atleast `87.00%` on the dev dataset for both English and Norwegian.\n",
    "You can use any model that you have implemented in this assignment: HMM, BiLSTM, or BiLSTM-CRF.\n",
    "- HMM is unlikely to score high enough for full credit\n",
    "- BiLSTM-CRF is more accurate than BiLSTM in theory, but it takes a long time to train. So iterating on BiLSTM might be better.    \n",
    "- Ideas: try varying the input (character-level features?), the optimizer, the model architecture, the pretrained embeddings, dropout, regularization, etc. Here are some more pretrained embeddings to try: [fastText](https://github.com/facebookresearch/fastText), [Polyglot](polyglot.readthedocs.io/en/latest/Embeddings.html), [word2vec](https://code.google.com/archive/p/word2vec/), [Glove](https://nlp.stanford.edu/projects/glove/).\n",
    "- For Norwegian, there are officially [two written forms of language](https://en.wikipedia.org/wiki/Norwegian_language): `Bokmål` and `Nynorsk`. The dataset we are using is for the `Bokmål` language. So, when using pretrained embeddings make sure to use it for that specific form.\n",
    "\n",
    "**Output files:** Make sure to name your files as follows: The unit tests will check for these files.\n",
    "- `model-dev-en.preds` ( Predictions for the dev dataset of english ) \n",
    "- `model-te-en.preds`  ( Predictions for the test dataset of english ) \n",
    "- `model-dev-nr.preds` ( Predictions for the dev dataset of norwegian ) \n",
    "- `model-te-nr.preds`  ( Predictions for the test dataset of norwegian ) \n",
    "\n",
    "**Tests**: `test_performance.py`: \n",
    "- `test_en_dev_accuracy()`\n",
    "- `test_en_test_accuracy()` #you cannot run this unit test.\n",
    "- `test_nr_dev_accuracy()`\n",
    "- `test_nr_test_accuracy()` #you cannot run this unit test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rubric\n",
    "\n",
    "Dev Set (both languages)\n",
    "- $\\geq$ 85% (*0.15 points*)\n",
    "- $\\geq$ 86% (*0.25 points*)\n",
    "- $\\geq$ 87% (*0.5 points*)\n",
    "\n",
    "Test Set (both languages)\n",
    "- $\\geq$ 84% (*0.15 points*)\n",
    "- $\\geq$ 85% (*0.25 points*)\n",
    "- $\\geq$ 86% (*0.5 points*)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "8. Bakeoff (Extra Credit)\n",
    "=================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Try to get the best accuracy possible on both `English` and `Norwegian` tagging.\n",
    "\n",
    "Some ideas:\n",
    "\n",
    "- Better features, such as characters\n",
    "- Better optimization\n",
    "- Number of Layers and Hidden units in the BiLSTM\n",
    "- More hidden layers in the Fully Connected Layer\n",
    "- Better loss function: like structured perceptron\n",
    "- Better preprocessing\n",
    "- Using Pretrained Embeddings like [fastText](https://github.com/facebookresearch/fastText), [Polyglot](polyglot.readthedocs.io/en/latest/Embeddings.html), [word2vec](https://code.google.com/archive/p/word2vec/), [Glove](https://nlp.stanford.edu/projects/glove/)\n",
    "- Dropout or other regularization scheme\n",
    "\n",
    "The current best accuracies from the staff are:\n",
    "- For English data:    \n",
    "    - `88% dev`, \n",
    "    - `87.5% test`\n",
    "- For Norwegian data:  \n",
    "    - `89% dev`, \n",
    "    - `87.8% test`\n",
    "\n",
    "However, these accuracies may improve before the end of the bakeoff!   \n",
    " \n",
    "\n",
    "**Output files:** In addition to participating in the kaggle bakeoff, make sure to name your files as follows:\n",
    "\n",
    "- `bakeoff-dev-en.preds` ( Predictions for the dev dataset of english ) \n",
    "- `bakeoff-te-en.preds`  ( Predictions for the test dataset of english ) \n",
    "- `bakeoff-dev-nr.preds` ( Predictions for the dev dataset of norwegian ) \n",
    "- `bakeoff-te-nr.preds`  ( Predictions for the test dataset of norwegian ) \n",
    "\n",
    "The unit tests will check for these files. To get the extra credit, you must both participate in Kaggle and submit these files.\n",
    "\n",
    "**Rubric**\n",
    "\n",
    "- Top 3 final score on Kaggle in English: +0.5%\n",
    "- Top 3 final score on Kaggle in Norwegian: +0.5%\n",
    "- Beat best staff final score on Kaggle in English: +0.5%\n",
    "- Best best staff final score on Kaggle in Norwegian: +0.5%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 9. 7650 Research Question  \n",
    "\n",
    "(*1 point*)\n",
    "\n",
    "You will select a recent research paper that performs **sequence labeling** either POS tagging or some other task within the last five years (2014-2018). Summarize the paper, answering the following questions:\n",
    "\n",
    "- List the title, author(s) and venue of the paper.\n",
    "- What is the task they are trying to solve?\n",
    "- What tagging methods do they use? HMMs, CRF?, max-margin markov networks, deep learning models?\n",
    "- What features do they use and why?\n",
    "- What methods and features are most effective?\n",
    "- Give a one-line summary of the paper that the authors are trying to leave for the reader.\n",
    "\n",
    "Your selection of papers is determined by the last digit of your GTID.\n",
    "\n",
    "- Digits 0-4: choose from NAACL, EMNLP, or NIPS\n",
    "- Digits 5-9: choose from ACL, EACL, or TACL\n",
    "\n",
    "You must choose a paper in the main conference (not workshops). The paper must be at least four pages long. All papers from these conferences are available for free online."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}