1_retrieve_candidate_paragraphs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Read-in-accession-ID-lists-created-from-initial-data-splitting\" data-toc-modified-id=\"Read-in-accession-ID-lists-created-from-initial-data-splitting-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Read in accession ID lists created from initial data splitting</a></span></li><li><span><a href=\"#Explore-labeled-dataframe\" data-toc-modified-id=\"Explore-labeled-dataframe-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Explore labeled dataframe</a></span><ul class=\"toc-item\"><li><span><a href=\"#funtion:-print_row_detail\" data-toc-modified-id=\"funtion:-print_row_detail-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>funtion: <code>print_row_detail</code></a></span></li></ul></li><li><span><a href=\"#Exploring-tags-that-match-any-of-the-regex-patterns\" data-toc-modified-id=\"Exploring-tags-that-match-any-of-the-regex-patterns-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Exploring tags that match any of the regex patterns</a></span><ul class=\"toc-item\"><li><span><a href=\"#Patterns-to-flag-candidate-paragraphs\" data-toc-modified-id=\"Patterns-to-flag-candidate-paragraphs-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Patterns to flag candidate paragraphs</a></span></li><li><span><a href=\"#Tag-Exploration\" data-toc-modified-id=\"Tag-Exploration-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Tag Exploration</a></span><ul class=\"toc-item\"><li><span><a href=\"#Example-1:-Searching-for-block-elements-adjacent-to-Employee-headings\" data-toc-modified-id=\"Example-1:-Searching-for-block-elements-adjacent-to-Employee-headings-3.2.1\"><span class=\"toc-item-num\">3.2.1&nbsp;&nbsp;</span>Example 1: Searching for block elements adjacent to <code>Employee</code> headings</a></span></li><li><span><a href=\"#Example-2:-An-employee-count-found-before-the-&quot;correct&quot;-one\" data-toc-modified-id=\"Example-2:-An-employee-count-found-before-the-&quot;correct&quot;-one-3.2.2\"><span class=\"toc-item-num\">3.2.2&nbsp;&nbsp;</span>Example 2: An employee count found before the \"correct\" one</a></span></li><li><span><a href=\"#Example-3:-Regex-matches-do-not-include-employee-number-because-of-&lt;span/&gt;-elements\" data-toc-modified-id=\"Example-3:-Regex-matches-do-not-include-employee-number-because-of-<span/>-elements-3.2.3\"><span class=\"toc-item-num\">3.2.3&nbsp;&nbsp;</span>Example 3: Regex matches do not include employee number because of <code>&lt;span/&gt;</code> elements</a></span></li></ul></li></ul></li><li><span><a href=\"#Test-regex\" data-toc-modified-id=\"Test-regex-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Test regex</a></span><ul class=\"toc-item\"><li><span><a href=\"#function:-check_regex_match\" data-toc-modified-id=\"function:-check_regex_match-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>function: check_regex_match</a></span></li></ul></li><li><span><a href=\"#Write-functions-to-flag-paragraphs\" data-toc-modified-id=\"Write-functions-to-flag-paragraphs-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Write functions to flag paragraphs</a></span><ul class=\"toc-item\"><li><span><a href=\"#Find-candidate-paragraphs\" data-toc-modified-id=\"Find-candidate-paragraphs-5.1\"><span class=\"toc-item-num\">5.1&nbsp;&nbsp;</span>Find candidate paragraphs</a></span></li><li><span><a href=\"#Create-dataframes-and-save-as-CSVs\" data-toc-modified-id=\"Create-dataframes-and-save-as-CSVs-5.2\"><span class=\"toc-item-num\">5.2&nbsp;&nbsp;</span>Create dataframes and save as CSVs</a></span></li></ul></li><li><span><a href=\"#Debug-odd-findings-to-refine-regex-or-code\" data-toc-modified-id=\"Debug-odd-findings-to-refine-regex-or-code-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>Debug odd findings to refine regex or code</a></span><ul class=\"toc-item\"><li><span><a href=\"#Explore-dataframe-of-paragraphs-for-training-data\" data-toc-modified-id=\"Explore-dataframe-of-paragraphs-for-training-data-6.1\"><span class=\"toc-item-num\">6.1&nbsp;&nbsp;</span>Explore dataframe of paragraphs for training data</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrieve candidate paragraphs from HTML files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in accession ID lists created from initial data splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:08.941041Z",
     "start_time": "2018-08-09T00:54:08.133816Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "train_accession_ids = pd.read_csv('../data/train_accession_ids.csv', names=['acc_id'])['acc_id'].tolist()\n",
    "val_accession_ids = pd.read_csv('../data/val_accession_ids.csv', names=['acc_id'])['acc_id'].tolist()\n",
    "\n",
    "labeled_df = pd.read_excel('../data/train_val_employee_count_paragraphs.xlsx')\n",
    "labeled_train_df = labeled_df.loc[labeled_df.split == 'train', :].copy()\n",
    "labeled_val_df = labeled_df.loc[labeled_df.split == 'val', :].copy()\n",
    "subset_df = pd.read_excel('../data/subset_employee_count_paragraphs.xlsx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since I already have the files, I can just loop through a directory to aquire the file paths, file names, and filing IDs (which are in the file name). The CIK numbers are useful for looking up additional filings on the [EDGAR website](https://www.sec.gov/cgi-bin/srch-edgar)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:09.355321Z",
     "start_time": "2018-08-09T00:54:08.942046Z"
    }
   },
   "outputs": [],
   "source": [
    "#filings_dir = PurePath('../employee_filings/')\n",
    "#html_path_list = [PurePath(os.getcwd()).joinpath(file).as_posix() for file in glob.iglob(\"/*.\".join([filings_dir.as_posix(), \"html\"]))]\n",
    "\n",
    "#from path import Path, getcwdu\n",
    "import glob\n",
    "import os\n",
    "from pathlib import PurePath\n",
    "filing_suffix, filings_dir = (\".html\",  PurePath('../employee_filings/').as_posix() + \"/\")\n",
    "html_path_list = [PurePath(os.getcwd()).joinpath(file).as_posix() for file in glob.iglob(\"*\".join([filings_dir, filing_suffix]))]\n",
    "html_file_list = [PurePath(path).name for path in html_path_list]\n",
    "\n",
    "full_accession_ids = [PurePath(path).stem for path in html_path_list]\n",
    "full_cik_nbrs = [x.split(sep='-')[0] for x in full_accession_ids]\n",
    "\n",
    "# Create list of paths for subset_df, based on accession_id\n",
    "subset_path_list = [PurePath(os.getcwd()).joinpath(filings_dir).joinpath(file) for file in html_file_list if PurePath(file).stem in subset_df.accession_number.unique().tolist()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 30 files in the train + validation sets for which there are no labeled employee counts. Thus, these are the potential true negatives. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:09.368331Z",
     "start_time": "2018-08-09T00:54:09.357324Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total files in train + validation subsets:  2222\n",
      "Files with employee count information, according to the golden dataset:  2192\n",
      "Total files in training set:  1667\n",
      "Training files with employee count information, according to the golden dataset:  1641\n",
      "Total files in validation set:  555\n",
      "Training files with employee count information, according to the golden dataset:  551\n"
     ]
    }
   ],
   "source": [
    "print(\"Total files in train + validation subsets: \", len(full_accession_ids))\n",
    "print(\"Files with employee count information, according to the golden dataset: \", \n",
    "      len(labeled_df.accession_number.unique().tolist()))\n",
    "print(\"Total files in training set: \", len(train_accession_ids))\n",
    "print(\"Training files with employee count information, according to the golden dataset: \", \n",
    "      len(labeled_train_df.accession_number.unique().tolist()))\n",
    "print(\"Total files in validation set: \", len(val_accession_ids))\n",
    "print(\"Training files with employee count information, according to the golden dataset: \", \n",
    "      len(labeled_val_df.accession_number.unique().tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore labeled dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### funtion: `print_row_detail`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This helper function is useful for inspecting the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:09.420365Z",
     "start_time": "2018-08-09T00:54:09.370331Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------- AAN -----------------------------------\n",
      "----------------------------------- 0000706688-17-000030 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :11500\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employees\n",
      "\n",
      "paragraph_text  :Employees   At December 31, 2016, the Company had approximately 11,500 employees. None of our employees are covered by a   collective bargaining agreement and we believe that our relations with employees are good\n",
      "\n",
      "----------------------------------- AAON -----------------------------------\n",
      "----------------------------------- 0000824142-17-000034 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :1619\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employees\n",
      "\n",
      "paragraph_text  :Employees   As of February 12, 2017, we employed 1,619 permanent employees. Our employees are not represented by  unions\n",
      "\n",
      "----------------------------------- AAXN -----------------------------------\n",
      "----------------------------------- 0001069183-17-000042 -----------------------------------\n",
      "data_key_friendly_name  :Full-Time Employees\n",
      "\n",
      "data_value  :699\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :full-time\n",
      "\n",
      "paragraph_text  :Employees   As of December 31, 2016,  we had 699 full-time  employees and 202 temporary  employees. The breakdown of  our   full-time employees by department is  as follows: 175 direct  manufacturing employees and 524  administrative   and manufacturing support employees. Of the 524 administrative and manufacturing support employees, 213  were   involved in sales,  marketing, communications and  training. Of the  202 temporary employees,  more than  92%   worked in direct manufacturing roles. Our employees  are not covered by any collective bargaining  agreement,   and we have never experienced a work stoppage. We believe that our relations with our employees are good\n",
      "\n",
      "----------------------------------- AAXN -----------------------------------\n",
      "----------------------------------- 0001069183-17-000042 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :202\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :temporary\n",
      "\n",
      "paragraph_text  :Employees   As of December 31, 2016,  we had 699 full-time  employees and 202 temporary  employees. The breakdown of  our   full-time employees by department is  as follows: 175 direct  manufacturing employees and 524  administrative   and manufacturing support employees. Of the 524 administrative and manufacturing support employees, 213  were   involved in sales,  marketing, communications and  training. Of the  202 temporary employees,  more than  92%   worked in direct manufacturing roles. Our employees  are not covered by any collective bargaining  agreement,   and we have never experienced a work stoppage. We believe that our relations with our employees are good\n",
      "\n",
      "----------------------------------- A -----------------------------------\n",
      "----------------------------------- 0001090872-16-000082 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :12500\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employed\n",
      "\n",
      "paragraph_text  :We sell  our  products  primarily  through  direct  sales,  but  we  also  utilize  distributors,  resellers,   manufacturer's representatives and  electronic commerce. Of  our total net  revenue of $4.2  billion for  the   fiscal year ended October 31, 2016, we generated 30 percent in the U.S. and 70 percent outside the U.S. As of   October 31, 2016, we employed approximately 12,500 people worldwide. Our primary research and development and   manufacturing sites  are in  California, Colorado,  Delaware,  Massachusetts and  Texas in  the U.S.  and  in   Australia, China, Denmark, Germany, Italy, Japan, Malaysia, Singapore and the United Kingdom\n",
      "\n",
      "----------------------------------- ABB -----------------------------------\n",
      "----------------------------------- 0001104659-17-015892 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :132300\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :Total\n",
      "\n",
      "paragraph_text  :December 31,                                                                     2016             2015             2014   Europe                                                               61,400           61,600           63,000   The Americas                                                         29,000           30,900           32,200   Asia, Middle East and Africa                                         41,900           43,300           45,200   Total                                                               132,300          135,800          140,400\n",
      "\n",
      "----------------------------------- ABC -----------------------------------\n",
      "----------------------------------- 0001140859-16-000022 -----------------------------------\n",
      "data_key_friendly_name  :Full-Time Employees\n",
      "\n",
      "data_value  :18000\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :full-time\n",
      "\n",
      "paragraph_text  :Employees   As of September 30, 2016, we had approximately 19,000 employees, of which approximately 18,000 were full-time   employees. Approximately 2% of our employees are covered by collective bargaining agreements. We believe that   our relationship with our employees is good. If any  of our employees in locations that are unionized  should   engage in strikes  or other  such bargaining tactics  in connection  with the negotiation  of new  collective   bargaining agreements upon  the expiration  of any existing  collective bargaining  agreements, such  tactics   could be disruptive to our operations and adversely affect our results of operations, but we believe we  have   adequate contingency plans in place to  assure delivery of pharmaceuticals to  our customers in the event  of   any such disruptions\n",
      "\n",
      "----------------------------------- ABC -----------------------------------\n",
      "----------------------------------- 0001140859-16-000022 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :19000\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employees\n",
      "\n",
      "paragraph_text  :Employees   As of September 30, 2016, we had approximately 19,000 employees, of which approximately 18,000 were full-time   employees. Approximately 2% of our employees are covered by collective bargaining agreements. We believe that   our relationship with our employees is good. If any  of our employees in locations that are unionized  should   engage in strikes  or other  such bargaining tactics  in connection  with the negotiation  of new  collective   bargaining agreements upon  the expiration  of any existing  collective bargaining  agreements, such  tactics   could be disruptive to our operations and adversely affect our results of operations, but we believe we  have   adequate contingency plans in place to  assure delivery of pharmaceuticals to  our customers in the event  of   any such disruptions\n",
      "\n",
      "----------------------------------- AAP -----------------------------------\n",
      "----------------------------------- 0001158449-17-000034 -----------------------------------\n",
      "data_key_friendly_name  :Full-Time Employees\n",
      "\n",
      "data_value  :41000\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :full-time Team\n",
      "\n",
      "paragraph_text  :Team Members   As of February 23, 2017, we employed approximately 41,000 full-time Team Members and approximately 33,000   part-time Team Members. Our workforce consisted of 87% of our Team Members employed in store-level   operations, 9% employed in distribution and 4% employed in our corporate offices. As of February 23, 2017,   less than 1% of our Team Members were represented by labor unions. We have never experienced any labor   disruption. We believe that our Team Member relations are solid\n",
      "\n",
      "----------------------------------- AAP -----------------------------------\n",
      "----------------------------------- 0001158449-17-000034 -----------------------------------\n",
      "data_key_friendly_name  :Part-Time Employees\n",
      "\n",
      "data_value  :33000\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :part-time Team Members\n",
      "\n",
      "paragraph_text  :Team Members   As of February 23, 2017, we employed approximately 41,000 full-time Team Members and approximately 33,000   part-time Team Members. Our workforce consisted of 87% of our Team Members employed in store-level   operations, 9% employed in distribution and 4% employed in our corporate offices. As of February 23, 2017,   less than 1% of our Team Members were represented by labor unions. We have never experienced any labor   disruption. We believe that our Team Member relations are solid\n",
      "\n",
      "----------------------------------- AAWW -----------------------------------\n",
      "----------------------------------- 0001193125-17-053796 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :2646\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employees\n",
      "\n",
      "paragraph_text  :Our business depends on highly qualified management, operations and flight personnel. As a percentage of our   consolidated operating expenses, salaries, wages and benefits accounted for approximately 25.4% in 2016,   20.7% in 2015 and 19.2% in 2014. As of December 31, 2016, we had 2,646 employees, 1,581 of whom were pilots\n",
      "\n",
      "----------------------------------- AABA -----------------------------------\n",
      "----------------------------------- 0001193125-17-065791 -----------------------------------\n",
      "data_key_friendly_name  :Full-Time Employees\n",
      "\n",
      "data_value  :8500\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :full-time\n",
      "\n",
      "paragraph_text  :As of December  31, 2016,  we had approximately  8,500 full-time  employees and 600  contractors. Our  future   success is substantially dependent on the performance  of our senior management and key technical  personnel,   as well as our continuing ability to attract, maintain the caliber of, and retain highly qualified technical,   executive, and managerial personnel.  We remain committed  to our talented employees  and providing the  best   possible workplace culture for them. Additional information regarding certain risks related to our  employees   is included in Part I, Item 1A \"Risk Factors\" of  this Annual Report on Form 10-K and is incorporated  herein   by reference\n",
      "\n",
      "----------------------------------- AA -----------------------------------\n",
      "----------------------------------- 0001193125-17-083862 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :14000\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employees\n",
      "\n",
      "paragraph_text  :Alcoa's total worldwide employment at the end of 2016 was approximately 14,000 employees in 15 countries\n",
      "\n",
      "----------------------------------- ABCD -----------------------------------\n",
      "----------------------------------- 0001466815-17-000007 -----------------------------------\n",
      "data_key_friendly_name  :Full-Time Employees\n",
      "\n",
      "data_value  :565\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :full-time\n",
      "\n",
      "paragraph_text  :Employees   As of December 31, 2016, we had a total of 619 employees, including 565 full-time, 20 regularly scheduled   part-time employees, and 34 need-based part-time employees. We consider our current relationship with our   employees to be good. Our employees are not represented by labor unions and are not subject to collective   bargaining agreements\n",
      "\n",
      "----------------------------------- ABCD -----------------------------------\n",
      "----------------------------------- 0001466815-17-000007 -----------------------------------\n",
      "data_key_friendly_name  :Part-Time Employees\n",
      "\n",
      "data_value  :34\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :part-time\n",
      "\n",
      "paragraph_text  :Employees   As of December 31, 2016, we had a total of 619 employees, including 565 full-time, 20 regularly scheduled   part-time employees, and 34 need-based part-time employees. We consider our current relationship with our   employees to be good. Our employees are not represented by labor unions and are not subject to collective   bargaining agreements\n",
      "\n",
      "----------------------------------- ABCD -----------------------------------\n",
      "----------------------------------- 0001466815-17-000007 -----------------------------------\n",
      "data_key_friendly_name  :Part-Time Employees\n",
      "\n",
      "data_value  :20\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :part-time\n",
      "\n",
      "paragraph_text  :Employees   As of December 31, 2016, we had a total of 619 employees, including 565 full-time, 20 regularly scheduled   part-time employees, and 34 need-based part-time employees. We consider our current relationship with our   employees to be good. Our employees are not represented by labor unions and are not subject to collective   bargaining agreements\n",
      "\n",
      "----------------------------------- ABBV -----------------------------------\n",
      "----------------------------------- 0001551152-17-000004 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :30000\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employed\n",
      "\n",
      "paragraph_text  :Employees   AbbVie employed approximately 30,000 persons as of January 31, 2017. Outside the United States, some of   AbbVie's employees are represented by unions or works councils. AbbVie believes that it has good relations   with its employees\n",
      "\n",
      "----------------------------------- AAOI -----------------------------------\n",
      "----------------------------------- 0001558370-17-001556 -----------------------------------\n",
      "data_key_friendly_name  :Full-Time Employees\n",
      "\n",
      "data_value  :2776\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :full-time\n",
      "\n",
      "paragraph_text  :As of December 31, 2016, we employed 2,776 full-time employees, of which 31 held Ph.D. degrees in a science   or engineering field. Of our employees, 287 are located in the U.S., 1,218 are located in Taiwan and 1,271   are located in China. None of our employees are represented by any collective bargaining agreement, but   certain employees of our China subsidiary are members of a trade union. We have never suffered any work   stoppage as a result of an employment related strike or any employee related dispute and believe that we have   satisfactory relations with our employees\n",
      "\n",
      "----------------------------------- AAC -----------------------------------\n",
      "----------------------------------- 0001564590-17-003590 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :2100\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employed\n",
      "\n",
      "paragraph_text  :As of December 31, 2016, we employed approximately 2,100 people. During 2016, employees at Sunrise House   voted to join the Health Professionals and Allied Employees (\"HPAE\") labor union. None of our other employees   are represented by a labor union or covered by a collective bargaining agreement. We believe that our   employee relations are good\n",
      "\n",
      "----------------------------------- ABM -----------------------------------\n",
      "----------------------------------- 0001628280-16-022122 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :110000\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employed\n",
      "\n",
      "paragraph_text  :Employees   As of October 31, 2016, we employed approximately 110,000 employees\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def print_row_detail(df=subset_df, nrow=10, header_list = ['ticker', 'accession_number' ],\n",
    "                    detail_list = ['data_key_friendly_name', 'data_value', 'reported_units', 'text', 'paragraph_text'],\n",
    "                    sortby=['accession_number', 'data_key_friendly_name'], \n",
    "                     ascending=True):\n",
    "    \"\"\"Print nrows of df values for columns in detail_list, sequentially.\"\"\"\n",
    "    df_sorted = df.sort_values(sortby, ascending=ascending).reset_index()\n",
    "    nrow = min(len(df_sorted), nrow)\n",
    "    for i in range(0, nrow):\n",
    "        for h in header_list:\n",
    "            print('-'*35  + ' ' +  str(df_sorted[h][i]) + ' ' + '-'*35)\n",
    "        for d in detail_list:\n",
    "            print(d + '  :' + str(df_sorted[d][i]))\n",
    "            print('')\n",
    "\n",
    "print_row_detail(df=subset_df, nrow=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only 15 files are included in the subset_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:09.434374Z",
     "start_time": "2018-08-09T00:54:09.422366Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(subset_df.accession_number.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:09.449382Z",
     "start_time": "2018-08-09T00:54:09.436375Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "data_key_friendly_name  text                  \n",
       "Full-Time Employees     full-time                 5\n",
       "                        full-time Team            1\n",
       "Other Employees         employees                 5\n",
       "                        employed                  4\n",
       "                        Total                     1\n",
       "                        temporary                 1\n",
       "Part-Time Employees     part-time                 2\n",
       "                        part-time Team Members    1\n",
       "Name: text, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset_df.groupby('data_key_friendly_name').text.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring tags that match any of the regex patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Patterns to flag candidate paragraphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modular components somestimes make it easier to organize (and update) regular expressions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:09.509424Z",
     "start_time": "2018-08-09T00:54:09.451386Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------- TREC -----------------------------------\n",
      "----------------------------------- 0000007039-17-000009 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :310.0\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employees\n",
      "\n",
      "paragraph_text  :Personnel   The number of  our regular, U.S.  based employees was  approximately 310, 296,  and 271 for  the years  ended   December 31,  2016,  2015, and  2014,  respectively.  Of these  employees,  none are  covered  by  collective   bargaining agreements. Regular employees are defined as active executive, management, professional, technical   and wage employees who work full time or part time  for the Company and are covered by our benefit plans  and   programs. Our workforce has increased primarily due to expansions at both facilities\n",
      "\n",
      "----------------------------------- BA -----------------------------------\n",
      "----------------------------------- 0000012927-17-000006 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :150500.0\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :Total workforce\n",
      "\n",
      "paragraph_text  :Employees   Total workforce level at December 31, 2016 was approximately 150,500. As of December 31, 2016, our  principal   collective bargaining agreements were with the following unions:\n",
      "\n",
      "----------------------------------- LUB -----------------------------------\n",
      "----------------------------------- 0000016099-16-000059 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :7988.0\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employees\n",
      "\n",
      "paragraph_text  :Employees   As of November 9, 2016, we had an active workforce of 7,988 employees consisting of restaurant management   employees, non-management restaurants employees, CCS management employees, CCS non-management employees, and   office and facility service employees. Employee relations are considered to be good. We have never had a   strike or work stoppage, and we are not subject to collective bargaining agreements\n",
      "\n",
      "----------------------------------- OMC -----------------------------------\n",
      "----------------------------------- 0000029989-17-000004 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :78500.0\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :people\n",
      "\n",
      "paragraph_text  :Our Employees   At December 31, 2016, we employed approximately 78,500 people. The skill sets of our workforce across our   agencies and within each discipline are similar. Common to all is the ability to understand a client's brand   or product and their selling proposition and to develop a unique message to communicate the value of the   brand or product to the client's target audience. Recognizing the importance of this core competency, we have   established tailored training and education programs for our client service professionals around this   competency. See the MD&A for a discussion of the effect of salary and related costs on our results of   operations\n",
      "\n",
      "----------------------------------- MLHR -----------------------------------\n",
      "----------------------------------- 0000066382-16-000082 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :7607.0\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employees\n",
      "\n",
      "paragraph_text  :As of May 28, 2016, the company had 7,607 employees, representing a 1.3 percent increase as compared with May   30, 2015. The  increase in  employees was  driven principally by  an increase  in workforce  within our  West   Michigan manufacturing facilities. In addition to its employee workforce, the company uses temporary labor to   meet uneven demand in its manufacturing operations\n",
      "\n",
      "----------------------------------- NWN -----------------------------------\n",
      "----------------------------------- 0000073020-17-000007 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :611.0\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :members\n",
      "\n",
      "paragraph_text  :At December 31, 2016, the utility workforce consisted of 611 members of the Office and Professional Employees   International Union (OPEIU) Local No. 11, AFL-CIO, and 497 non-union employees. Our labor agreement with   members of OPEIU covers wages, benefits and working conditions. On May 22, 2014, our union employees ratified   a new labor agreement (Joint Accord) that extends to November 30, 2019, and thereafter from year to year   unless either party serves notice of its intent to negotiate modifications to the collective bargaining   agreement\n",
      "\n",
      "----------------------------------- NWN -----------------------------------\n",
      "----------------------------------- 0000073020-17-000007 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :497.0\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :non-union\n",
      "\n",
      "paragraph_text  :At December 31, 2016, the utility workforce consisted of 611 members of the Office and Professional Employees   International Union (OPEIU) Local No. 11, AFL-CIO, and 497 non-union employees. Our labor agreement with   members of OPEIU covers wages, benefits and working conditions. On May 22, 2014, our union employees ratified   a new labor agreement (Joint Accord) that extends to November 30, 2019, and thereafter from year to year   unless either party serves notice of its intent to negotiate modifications to the collective bargaining   agreement\n",
      "\n",
      "----------------------------------- NWN -----------------------------------\n",
      "----------------------------------- 0000073020-17-000007 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :15.0\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :non-union\n",
      "\n",
      "paragraph_text  :At December 31, 2016, our non-utility subsidiaries had a combined workforce of 15 non-union employees, of   which the majority of our employees at the Gill Ranch facility voted to unionize as part of IBEW Local Union   No. 1245. We are currently in the process of bargaining the first contract for 8 of these employees and the   ultimate outcome of such negotiations is unknown at this time. Our subsidiaries receive certain services from   centralized operations at the utility, and the utility is reimbursed for those services pursuant to a Shared   Services Agreement\n",
      "\n",
      "----------------------------------- OII -----------------------------------\n",
      "----------------------------------- 0000073756-17-000013 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :9300.0\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employees\n",
      "\n",
      "paragraph_text  :EMPLOYEES   As of December 31, 2016, we had approximately 9,300 employees. Taking into account reductions in workforce we   announced in 2016 to take effect in 2017, we would have approximately 8,800 employees. Our workforce varies   seasonally and peaks during the second and third quarters. We consider our relations with our employees to be   satisfactory\n",
      "\n",
      "----------------------------------- EGN -----------------------------------\n",
      "----------------------------------- 0000277595-17-000011 -----------------------------------\n",
      "data_key_friendly_name  :Other Employees\n",
      "\n",
      "data_value  :390.0\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :employees\n",
      "\n",
      "paragraph_text  :The Company has approximately 390 employees. On January 22, 2016 and March 18, 2016, we reduced our workforce   as part of an overall  plan to reduce costs and  better align our workforce with  the needs of our  business\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "# Initial components\n",
    "employee_terms = \"(associates|employees|full[ -]time[ -]equivalent(s)?|staff|team members|workers)\"\n",
    "person_terms = \"(individuals|people|persons)\" # These need additional cues\n",
    "workforce_terms = \"(((employee|employment|head|personnel|staff|worker|workforce) (count(s)|level(s)|total(s))+)|(head-count|headcount|workforce))\"\n",
    "employee_type_terms = \"(full time|full-time|permanent|part time|part-time|regular|seasonal|temporary|total)\"\n",
    "\n",
    "numeral_pat = \"(([0-9]{1,3},)*[0-9]{1,3}([.][0-9])?)\" # Include numerals, requiring comma separation when appropriate, and allowing for decimals. \n",
    "rel_qualifiers = \"(a total of|approximately|in aggregate|in total|(an|the) equivalent of|total)\"\n",
    "\n",
    "space_pat = \"( |\\n)\" # In html, a space is often missing if the paragraph continues on the next line\n",
    "\n",
    "magnitude_words = \"(hundred|thousand|million|billion)\" \n",
    "num_words = \"(one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve|thirteen|fourteen|fifteen|sixteen|seventeen|eighteen|nineteen)\"\n",
    "\n",
    "# Composites to allow numerals and/or number words with relative modifiers\n",
    "num_pat = \"\".join([\"((\",numeral_pat, \"|\", num_words, \")(\", space_pat, magnitude_words, \")*\", \")\"]) \n",
    "num_emps_pat = \"\".join([num_pat, space_pat, \"(\", rel_qualifiers, space_pat, \")*\",  \"(\", employee_type_terms, space_pat, \")*\"])\n",
    "\n",
    "# Actual patterns to be used, named by the format they're meant to capture\n",
    "number_employees_pat = \"\".join([num_emps_pat, employee_terms]) # A number followed by an employee term, allowing for qualifiers\n",
    "employed_num_pat = \"\".join([\"employ((ed|s)?)?\", space_pat, \"(\", rel_qualifiers, space_pat, \")*\", \n",
    "                        num_emps_pat]) # The verb employed, followed by a number\n",
    "emp_type_emp_term_pat = \"\".join([employee_type_terms, space_pat, employee_terms]) # Part or full-time term, followed by an employee term\n",
    "employed_end_span_pat = \"\".join([\"employed(\", space_pat, rel_qualifiers, \")*$\"]) # The actual number is often cut off by a <span/> element in html\n",
    "span_start_employees_pat = \"\".join([\"^\", \"(\", rel_qualifiers, space_pat, \")*\",  \n",
    "                                   \"(\", employee_type_terms, space_pat, \")*\", \n",
    "                                   employee_terms]) # Employee terms at the beginning of a span\n",
    "\n",
    "emp_pat_list = [number_employees_pat, employed_num_pat, emp_type_emp_term_pat, \n",
    "                employed_end_span_pat, span_start_employees_pat, workforce_terms] # Workforce terms just need a broad net\n",
    "emp_pats = [re.compile(x, re.I) for x in emp_pat_list]\n",
    "\n",
    "print_row_detail(labeled_train_df[labeled_train_df.paragraph_text.str.contains(re.compile(\"workforce\"))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regex for identifying \"block\" html elements - useful for finding actual paragraph boundaries, or the first set of *n* paragraphs or tables immediately after a secion heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:09.514431Z",
     "start_time": "2018-08-09T00:54:09.510424Z"
    }
   },
   "outputs": [],
   "source": [
    "block_re = re.compile(r\"^(p|div|table)$\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tag Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:09.597483Z",
     "start_time": "2018-08-09T00:54:09.516431Z"
    }
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup as bs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 1: Searching for block elements adjacent to `Employee` headings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:14.439302Z",
     "start_time": "2018-08-09T00:54:09.598484Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accession number: 0001069183-17-000042\n",
      "Matches: 9\n",
      "First match is:  Employees\n",
      "Type of first match:  <class 'bs4.element.NavigableString'>\n",
      "Tag parent is: <div class=\"c92\">Employees</div>\n"
     ]
    }
   ],
   "source": [
    "with open(subset_path_list[2], encoding=\"utf8\") as file: \n",
    "            file1_html = file.read()\n",
    "            soup1 = bs(file1_html, 'lxml')\n",
    "soup1_emp_count = soup1.find_all(string=[emp_pats])\n",
    "\n",
    "print(\"Accession number:\", subset_path_list[2].stem)\n",
    "print(\"Matches:\", len(soup1_emp_count))\n",
    "\n",
    "emp1_tag = soup1_emp_count[0]\n",
    "print(\"First match is: \", emp1_tag)\n",
    "print(\"Type of first match: \", type(emp1_tag))\n",
    "print(\"Tag parent is:\", emp1_tag.parent) # The match is a string within a tag; the parent is the actual tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first tag is a section header, so I'm now interested in its siblings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:14.450308Z",
     "start_time": "2018-08-09T00:54:14.441303Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sibling number 1 :\n",
      "============================================================\n",
      "<div class=\"c88\"><span class=\"c16\">As of December 31,</span> <span class=\"c16\">2016</span><span class=\"c16\">,\n",
      "we had</span> <span class=\"c16\">699</span> <span class=\"c16\">full-time employees and</span> <span class=\"c16\">202</span> <span class=\"c16\">temporary employees. The breakdown of our full-time employees by department\n",
      "is as follows:</span> <span class=\"c16\">175</span> <span class=\"c16\">direct manufacturing employees and</span>\n",
      "<span class=\"c16\">524</span> <span class=\"c16\">administrative and manufacturing support employees. Of\n",
      "the</span> <span class=\"c16\">524</span> <span class=\"c16\">administrative and manufacturing support\n",
      "employees,</span> <span class=\"c16\">213</span> <span class=\"c16\">were involved in sales, marketing,\n",
      "communications and training. Of the</span> <span class=\"c16\">202</span> <span class=\"c16\">temporary employees,\n",
      "more than</span> <span class=\"c16\">92%</span> <span class=\"c16\">worked in direct manufacturing roles. Our\n",
      "employees are not covered by any collective bargaining agreement, and we have never experienced a work\n",
      "stoppage. We believe that our relations with our employees are good.</span></div>\n",
      "Sibling number 2 :\n",
      "============================================================\n",
      "<div class=\"c66\">11</div>\n",
      "Sibling number 3 :\n",
      "============================================================\n",
      "<div class=\"c2\"><a class=\"c1\" href=\"#s6AA0E9B102C8EE19B8CCB5E21E0F79E3\">Table of Contents</a></div>\n",
      "Sibling number 4 :\n",
      "============================================================\n",
      "<div class=\"c92\">Available Information</div>\n",
      "Sibling number 5 :\n",
      "============================================================\n",
      "<div class=\"c86\">We were incorporated in Arizona in September 1993 as ICER Corporation. We changed our name to\n",
      "AIR TASER, Inc. in December 1993 and to TASER International, Incorporated in April 1998. In January 2001, we\n",
      "reincorporated in Delaware as TASER International, Inc.</div>\n",
      "Sibling number 6 :\n",
      "============================================================\n",
      "<div class=\"c88\"><span class=\"c16\">Our Annual Report on Form 10-K, quarterly reports on Form 10-Q, current\n",
      "reports on Form 8-K and amendments to those reports filed or furnished pursuant to Section 13(a) or 15(d) of\n",
      "the Securities Exchange Act of 1934 are available free of charge on our website at</span> <span class=\"c91\">http://www.TASER.com</span> <span class=\"c16\">as soon as reasonably practicable after we electronically\n",
      "file such material with, or furnish such material to, the SEC. The SEC maintains a website that contains\n",
      "reports, proxy and information statements and other information regarding issuers that file electronically with\n",
      "the SEC at</span> <span class=\"c91\">http://www.sec.gov</span><span class=\"c16\">.</span></div>\n"
     ]
    }
   ],
   "source": [
    "for i, v in enumerate(emp1_tag.parent.find_next_siblings(block_re, limit=6)):\n",
    "    print(\"Sibling number\", i+1, \":\")\n",
    "    print(\"==\" * 30)\n",
    "    print(v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 2: An employee count found before the \"correct\" one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:19.839904Z",
     "start_time": "2018-08-09T00:54:14.451308Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accession number: 0001558370-17-001556\n",
      "Matches: 7\n"
     ]
    }
   ],
   "source": [
    "with open(subset_path_list[12], encoding=\"utf8\") as file: \n",
    "            file2_html = file.read()\n",
    "            soup2 = bs(file2_html, 'lxml')\n",
    "soup2_emp_count = soup2.find_all(string=[emp_pats])\n",
    "\n",
    "print(\"Accession number:\", subset_path_list[12].stem)\n",
    "print(\"Matches:\", len(soup2_emp_count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:19.849910Z",
     "start_time": "2018-08-09T00:54:19.840907Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------- AAOI -----------------------------------\n",
      "----------------------------------- 0001558370-17-001556 -----------------------------------\n",
      "data_key_friendly_name  :Full-Time Employees\n",
      "\n",
      "data_value  :2776\n",
      "\n",
      "reported_units  :ones\n",
      "\n",
      "text  :full-time\n",
      "\n",
      "paragraph_text  :As of December 31, 2016, we employed 2,776 full-time employees, of which 31 held Ph.D. degrees in a science   or engineering field. Of our employees, 287 are located in the U.S., 1,218 are located in Taiwan and 1,271   are located in China. None of our employees are represented by any collective bargaining agreement, but   certain employees of our China subsidiary are members of a trade union. We have never suffered any work   stoppage as a result of an employment related strike or any employee related dispute and believe that we have   satisfactory relations with our employees\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print_row_detail(subset_df[subset_df.accession_number == '0001558370-17-001556'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, the correct count is below an `Employees` section name. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:19.862920Z",
     "start_time": "2018-08-09T00:54:19.850911Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Match number 1 :\n",
      "============================================================\n",
      "As of December 31, 2016, we had a total of 276 employees working in the R&D department, including 16 with Ph.D. degrees. We continue to recruit talented engineers to further enhance our research and development capabilities. We have research and development departments in our facilities in Texas, Georgia, China and Taiwan. Our research and development teams collaborate on joint projects, and by co-locating with our manufacturing operations enable us to achieve an efficient cost structure and improve our time to market.\n",
      "Match number 2 :\n",
      "============================================================\n",
      "Employees\n",
      "Match number 3 :\n",
      "============================================================\n",
      "As of December 31, 2016, we employed 2,776 full-time employees, of which 31 held Ph.D. degrees in a science or engineering field. Of our employees, 287 are located in the U.S., 1,218 are located in Taiwan and 1,271 are located in China. None of our employees are represented by any collective bargaining agreement, but certain employees of our China subsidiary are members of a trade union. We have never suffered any work stoppage as a result of an employment related strike or any employee related dispute and believe that we have satisfactory relations with our employees.\n"
     ]
    }
   ],
   "source": [
    "newline_pat = re.compile(r\"([0-9a-zA-Z.(),])\\n([0-9a-zA-Z.(),])\") # To replace newline characters that have nothing to do with document structure\n",
    "for i, v in enumerate(soup2_emp_count[:3]):\n",
    "    print(\"Match number\", i+1, \":\")\n",
    "    print(\"==\" * 30)\n",
    "    print(re.sub(newline_pat, r\"\\1 \\2\", v))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A \"heading\" should be in a header tag, but it's not. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:19.874927Z",
     "start_time": "2018-08-09T00:54:19.864920Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<p class=\"c75\"><span class=\"c58\">Employees</span></p>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup2_emp_count[1].parent.parent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 3: Regex matches do not include employee number because of `<span/>` elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[SEC filing link](https://www.sec.gov/Archives/edgar/data/706688/000070668817000030/a10k4q2016.htm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:25.058386Z",
     "start_time": "2018-08-09T00:54:19.876929Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accession number: 0000706688-17-000030\n",
      "Matches: 13\n"
     ]
    }
   ],
   "source": [
    "with open(subset_path_list[0], encoding=\"utf8\") as file: \n",
    "            file3_html = file.read()\n",
    "            soup3 = bs(file3_html, 'lxml')\n",
    "soup3_emp_count = soup3.find_all(string=[emp_pats])\n",
    "\n",
    "print(\"Accession number:\", subset_path_list[0].stem)\n",
    "print(\"Matches:\", len(soup3_emp_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a clear example in which the regex failed to actually match the string containing the number of employees; the employee count should be right in front of `'employees.'` in Match 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:25.068393Z",
     "start_time": "2018-08-09T00:54:25.060387Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Match number 1 :\n",
      "Employees\n",
      "============================================================\n",
      "Match number 2 :\n",
      "employees. None of our employees are covered by a collective bargaining agreement and we believe that our relations with employees are good.\n",
      "============================================================\n",
      "Match number 3 :\n",
      "Our workforce is comprised primarily of employees who work on an hourly basis. To grow our operations and meet the needs and expectations of our customers, we must attract, train, and retain a large number of hourly associates, while at the same time controlling labor costs. These positions have historically had high turnover rates, which can lead to increased training, retention and other costs. In certain areas where we operate, there is significant competition for employees, including from retailers and the restaurant industries. The lack of availability of an adequate number of hourly employees, or our inability to attract and retain them, or an increase in wages and benefits to current employees could adversely affect our business, results of operations, cash flows and financial condition. We are subject to applicable rules and regulations relating to our relationship with our employees, including wage and hour regulations, health benefits, unemployment and payroll taxes, overtime and working conditions and immigration status. Accordingly, federal, state or local legislated increases in the minimum wage, as well as increases in additional labor cost components such as employee benefit costs, workers' compensation insurance rates, compliance costs and fines, would increase our labor costs, which could have a material adverse effect on our business, prospects, results of operations and financial condition.\n",
      "============================================================\n"
     ]
    }
   ],
   "source": [
    "for i, v in  enumerate([re.sub(newline_pat, r\"\\1 \\2\", x) for x in  soup3_emp_count[:3]]):\n",
    "    print(\"Match number\", i+1, \":\")\n",
    "    print(re.sub(newline_pat, r\"\\1 \\2\", v))\n",
    "    print(\"==\" * 30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-07-26T21:27:30.456065Z",
     "start_time": "2018-07-26T21:27:30.448060Z"
    }
   },
   "source": [
    "To verify, I'll use a helper function to show characters before and after a substring from the results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:25.081401Z",
     "start_time": "2018-08-09T00:54:25.070394Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<span class=\"c21\">At</span> <span class=\"c21\">December 31, 2016</span><span class=\"c21\">, the Company had approximately</span> <span class=\"c21\">11,500</span> <span class=\"c21\">employees. None of our employees are covered by a collective bargaining agreement and we believe that our relations with employees are good.</span></div>\n",
      "<div><a name=\"s0F8F0C9941555F6EB10D590A6566D4FF\" id=\"s0F8F0C9941555F6EB10D590A6566D4FF\"></a></div>\n",
      "<div class=\"c59\">Available Information</div>\n",
      "<div class=\"c61\"><span class=\"c21\">W\n"
     ]
    }
   ],
   "source": [
    "def show_str_context(search_str: str, in_string: str, char_before: int=500, char_after: int=2000, match_num: int=0):\n",
    "    \"\"\"Return characters before and after a given string.\"\"\"\n",
    "    sub_idx = in_string.find(search_str) \n",
    "    len_offset = len(search_str)\n",
    "    if match_num:\n",
    "        idx = sub_idx\n",
    "        for i in range(match_num):\n",
    "            idx = in_string.find(search_str, idx + len_offset)\n",
    "        sub_idx = idx\n",
    "    context_beg = max(sub_idx - char_before, 0)\n",
    "    context_end = sub_idx + len_offset + char_after\n",
    "    return in_string[context_beg:context_end]   \n",
    "\n",
    "search_str = \"employees are covered by a collective bargaining agreement and we believe that our relations with employees are\"\n",
    "print(re.sub(newline_pat, r\"\\1 \\2\",\n",
    "       show_str_context(search_str, file3_html, char_before=200, char_after=200)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test regex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  function: check_regex_match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:25.092407Z",
     "start_time": "2018-08-09T00:54:25.083403Z"
    }
   },
   "outputs": [],
   "source": [
    "def check_regex_match(pattern, text_list):\n",
    "    for idx, s in enumerate(text_list):\n",
    "        mo = re.search(pattern, s)\n",
    "        if mo:\n",
    "            ms = mo.span()[1]\n",
    "            print(\"------    \" + str(idx) + \"   Matched!    -----\")\n",
    "            print('str length  :' + str(len(s)) + '    match span  :' + str(ms))\n",
    "            print(s[:ms])\n",
    "            print(s[ms:])\n",
    "            print(re.search(pattern, s))\n",
    "        else:\n",
    "            print(\"------    \" + str(idx) + \"  NO MATCH    -----\")\n",
    "            print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write functions to flag paragraphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "notes:\n",
    "- if r\"^Employees$\" matches, find the next block element that matches another pattern \n",
    "- Include tables as \"matches\" when siblings of Employees block elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Need regex that will capture heading tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:25.107416Z",
     "start_time": "2018-08-09T00:54:25.094408Z"
    }
   },
   "outputs": [],
   "source": [
    "head_block_re = re.compile(r\"^(p|div|h[1-6])$\") # Identify block-level elements, aside from tables\n",
    "emp_head_pre = \"((([0-9a-z](([.][0-9a-z])|([0-9a-z][.]))*[0-9a-z.]?)|Full-Time|Our|Number of|Total) ?)?\"\n",
    "emp_head_terms = \"(Associates|Employees|Headcount|Personnel|Team Members|Staff|Workforce)\"\n",
    "#emp_head_raw = re.compile(r\"[>]((Our|Number of|Total) )?(Associates|Employees|Headcount|Personnel|Team Members|Workforce)[<]\", re.I)\n",
    "#emp_head = re.compile(r\"^((Our|Number of) )?(Associates|Employees|Headcount|Personnel|Team Members|Workforce)$\", re.I)\n",
    "emp_head_raw = re.compile(\"r\" + \"\".join([\"[>]\", emp_head_pre, emp_head_terms, \"([.:])?[<]\"]), re.I)\n",
    "emp_head = re.compile(\"\".join([\"^\",   emp_head_pre, emp_head_terms, \"([.:])?$\"]), re.I)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing the regex for different employee heading formats:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:25.119427Z",
     "start_time": "2018-08-09T00:54:25.109418Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------    0   Matched!    -----\n",
      "str length  :12    match span  :12\n",
      "10 Employees\n",
      "\n",
      "<_sre.SRE_Match object; span=(0, 12), match='10 Employees'>\n",
      "------    1   Matched!    -----\n",
      "str length  :11    match span  :11\n",
      "D.EMPLOYEES\n",
      "\n",
      "<_sre.SRE_Match object; span=(0, 11), match='D.EMPLOYEES'>\n",
      "------    2   Matched!    -----\n",
      "str length  :16    match span  :16\n",
      "8.3.18 Employees\n",
      "\n",
      "<_sre.SRE_Match object; span=(0, 16), match='8.3.18 Employees'>\n",
      "------    3   Matched!    -----\n",
      "str length  :13    match span  :13\n",
      "IX. Employees\n",
      "\n",
      "<_sre.SRE_Match object; span=(0, 13), match='IX. Employees'>\n"
     ]
    }
   ],
   "source": [
    "emp_head_test_list = [\"10 Employees\", \"D.EMPLOYEES\", \"8.3.18 Employees\", \"IX. Employees\"]\n",
    "check_regex_match(emp_head, emp_head_test_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lists to accumulate flagged items; later turned into dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find candidate paragraphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although the filings have to be downloaded from Edgar initially, it's useful to be able to save the html files and later run the extraction from a file list. The below helper functions were created for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:25.127431Z",
     "start_time": "2018-08-09T00:54:25.121426Z"
    }
   },
   "outputs": [],
   "source": [
    "def path_handler(path, force_html=False):\n",
    "    \"\"\"Return request text if path is url; otherwise returns file context.\n",
    "    Automatically recognizes Edgar URLs. \n",
    "    \"\"\"\n",
    "    \n",
    "    edgar_link_pat = re.compile(r\"(?:https://www.sec.gov/)(?:[a-z0-9/-]+)([0-9]{10}-?[0-9]{2}[-]?[0-9]{6})(?:[a-z0-9/-]*[.]+[a-z]+)\", re.I)\n",
    "    if re.match(edgar_link_pat, path) or force_html:\n",
    "        r = requests.get(path)\n",
    "        r_html = r.text \n",
    "        return r_html \n",
    "    \n",
    "    with open(path, encoding=\"utf8\") as file:\n",
    "        file_html = file.read()\n",
    "        return file_html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:25.316562Z",
     "start_time": "2018-08-09T00:54:25.129431Z"
    }
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:25.336571Z",
     "start_time": "2018-08-09T00:54:25.319560Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_filing_info(accession_id):\n",
    "    \"\"\"Return Edgar (url, filing_name, filing_date, company) for Accession Id\"\"\"\n",
    "    \n",
    "    search_url = \"\".join([\"https://searchwww.sec.gov/EDGARFSClient/jsp/EDGAR_MainAccess.jsp?search_text=\", \n",
    "                           accession_id, \n",
    "                           \"&isAdv=false\"])\n",
    "    search_html = requests.get(search_url).text\n",
    "    \n",
    "    edgar_trs = bs(search_html, 'lxml').find(\n",
    "        \"table\", {'xmlns:autn': 'http://schemas.autonomy.com/aci/'}).find_all(\n",
    "        \"tr\", {'class': None})\n",
    "    try: \n",
    "        js_info_pat = re.compile(r\"javascript:open[a-z]*[(](.*)[)][;]\", re.I)\n",
    "        filing_result = [tr.find(\"a\", id=\"viewFiling\") for tr in edgar_trs if tr.find_next_sibling(\"tr\", {\"class\":\"infoBorder\"}).findChild(\"a\", {\"title\" :\"Parent Filing\"}) is None]\n",
    "        if filing_result:\n",
    "            edgar_link_tag = filing_result[0]\n",
    "        else:\n",
    "            edgar_link_tag = edgar_trs[0].find_next(\"a\", {\"title\" :\"Parent Filing\"})\n",
    "        href_list = re.search(js_info_pat, edgar_link_tag['href']).group(1).split(',')[:2]\n",
    "        filing_name = edgar_link_tag.get_text(strip=True)\n",
    "        filing_date = edgar_link_tag.parent.previous_sibling.i.get_text()\n",
    "        url, company = href_list[0], href_list[1]\n",
    "        return url, filing_name, filing_date, company\n",
    "    except:\n",
    "        print(\"Error retrieving url for input\", accession_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:28.051594Z",
     "start_time": "2018-08-09T00:54:25.338571Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(\"'http://www.sec.gov/Archives/edgar/data/14846/000001484616000062/brtrealty10-k2016.htm'\", '10-K for BRT REALTY TRUST', '12/13/2016', \"'BRT REALTY TRUST'\")\n",
      "(\"'http://www.sec.gov/Archives/edgar/data/34088/000003408817000017/filename136.htm'\", '10-K for EXXON MOBIL CORP', '02/22/2017', \"'EXXON MOBIL CORP\")\n",
      "(\"'http://www.sec.gov/Archives/edgar/data/12927/000001292717000006/a201612dec3110k.htm'\", '10-K for BOEING CO', '02/08/2017', \"'BOEING CO'\")\n"
     ]
    }
   ],
   "source": [
    "test_acc_ids = [\"000001484616000062\", \"000003408817000017\", \"000001292717000006\"]\n",
    "for acc_id in test_acc_ids:\n",
    "    print(get_filing_info(acc_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When retaining tags from a document parsed by BeautifulSoup, it's important to simply copy or extract the tag you need. This gives you all descendants of the tag without keeping the entire parse tree in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:28.055598Z",
     "start_time": "2018-08-09T00:54:28.052595Z"
    }
   },
   "outputs": [],
   "source": [
    "import copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:28.167862Z",
     "start_time": "2018-08-09T00:54:28.057599Z"
    },
    "code_folding": [
     28
    ]
   },
   "outputs": [],
   "source": [
    "def extract_paragraph_df(path_list, regex_list, header_regex, header_raw_regex,\n",
    "                       head_block_regex=head_block_re, table_df=True):\n",
    "    \"\"\"Return dict with df of paragraphs that match the given regex.\n",
    "    Optionally return df from matching tables as well.\"\"\"\n",
    "\n",
    "    acc_id_pat = re.compile(r\"[0-9]{10}-?([0-9]{2})[-]?([0-9]{6})\")\n",
    "    block_re = re.compile(r\"^(p|div|table)$\")\n",
    "    acc_id_list = []  # Replace with generic ID list\n",
    "    para_list_orig = []\n",
    "    tag_list = []\n",
    "    emp_head_list = []\n",
    "    emp_head_first_list = []\n",
    "    tbl_acc_id_list = []\n",
    "    tbl_tag_list = []\n",
    "    for i, fl in enumerate(path_list):\n",
    "        acc_id_check = re.search(acc_id_pat, fl)\n",
    "        if acc_id_check:\n",
    "            acc_id = acc_id_check.group(0)\n",
    "            filing_year = \"20\" + acc_id_check.group(1)\n",
    "        else:\n",
    "            acc_id = str(i)\n",
    "        tag_set = set()\n",
    "        file_html = path_handler(fl)\n",
    "        soup = bs(file_html, 'lxml')\n",
    "        #        emp_head_flag = False\n",
    "        #        emp_head_first_match = False\n",
    "        if re.search(header_raw_regex, file_html):\n",
    "            for ihead, hblock in enumerate(soup.body.find_all(string=header_regex, limit=4)):\n",
    "                try:\n",
    "                    emp_head_tag = hblock.find_parent(name=head_block_regex)\n",
    "                    if emp_head_tag.name != 'table' and emp_head_tag.find_parent('table') is None:\n",
    "                        emp_head_matched = False\n",
    "                        #            print(emp_head_tag.name) ;print(emp_head_tag)\n",
    "                        for i2, block in enumerate(emp_head_tag.find_next_siblings(block_re, limit=6)):\n",
    "                            if block.find(string=[regex_list]) is not None and block.name != 'table':\n",
    "                                block_tag = copy.copy(block)\n",
    "                                if block_tag not in tag_set:\n",
    "                                    acc_id_list.append(acc_id)\n",
    "                                    tag_list.append(block_tag)\n",
    "                                    para_list_orig.append(block_tag.get_text())\n",
    "                                    tag_set.add(block_tag)\n",
    "                                    emp_head_list.append(True)\n",
    "                                    if not emp_head_matched:\n",
    "                                        emp_head_flag = True\n",
    "                                        emp_head_matched = True\n",
    "                                        emp_head_first_list.append(True)\n",
    "                                    else:\n",
    "                                        emp_head_first_list.append(False)\n",
    "                            if block.find('table') is not None:\n",
    "                                #                    print('Found table match!')\n",
    "                                tbl_block_tag = copy.copy(block)\n",
    "                                if tbl_block_tag not in tag_set:\n",
    "                                    tbl_acc_id_list.append(acc_id)\n",
    "                                    tbl_tag_list.append(tbl_block_tag)\n",
    "                                    tag_set.add(tbl_block_tag)\n",
    "                except:\n",
    "                    continue\n",
    "        #        else:\n",
    "        #            print('No Employees header')\n",
    "        soup_emp_count = soup.body.find_all(string=[regex_list])\n",
    "        soup_emp_paras = [x.find_parent(name=block_re) for x in soup_emp_count]\n",
    "        soup_emp_paras = [x for x in soup_emp_paras if x is not None]\n",
    "        for i2, block in enumerate(soup_emp_paras):\n",
    "            #                print('Para number: ' + str(i2)); print(block)\n",
    "            block_tag = copy.copy(block)\n",
    "            if block_tag not in tag_set:\n",
    "                if block.find('table') is not None:\n",
    "                    tbl_acc_id_list.append(acc_id)\n",
    "                    tbl_tag_list.append(block_tag)\n",
    "                    tag_set.add(block_tag)\n",
    "                else:\n",
    "                    acc_id_list.append(acc_id)\n",
    "                    tag_list.append(block_tag)\n",
    "                    tag_set.add(block_tag)\n",
    "                    para_list_orig.append(block_tag.get_text())\n",
    "                    emp_head_list.append(False)\n",
    "                    emp_head_first_list.append(False)\n",
    "\n",
    "    paragraph_dict = {'acc_id': acc_id_list,\n",
    "                        'para_text': [p.replace('\\n', ' ').strip().replace(' ,', ',') for p in para_list_orig],\n",
    "                        'len': [len(p) for p in para_list_orig],\n",
    "                        'emp_header': emp_head_list,\n",
    "                        'first_emp_head_block': emp_head_first_list,\n",
    "                        'para_text_orig': para_list_orig,\n",
    "                        'para_tag': tag_list,\n",
    "                        'split': 'train',\n",
    "                        'label': 0}\n",
    "    # paragraph_input_df['para_text'] = paragraph_input_df.para_text_orig.replace('\\n', ' ')\n",
    "    p_columns = ['acc_id', 'para_text', 'len', 'emp_header', 'first_emp_head_block', 'para_text_orig',\n",
    "                 'para_tag', 'split', 'label']\n",
    "\n",
    "    paragraph_df = pd.DataFrame(paragraph_dict, columns=p_columns)\n",
    "\n",
    "    if table_df:\n",
    "        # Make DataFrame from <table> elements\n",
    "        tbl_df = pd.DataFrame(data={'acc_id': tbl_acc_id_list,\n",
    "                                    'tbl_html': tbl_tag_list,\n",
    "                                    'split': 'train'})\n",
    "\n",
    "        return {'paragraphs': paragraph_df,\n",
    "                'tables': tbl_df}\n",
    "\n",
    "    return {'paragraphs': paragraph_df}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:54:57.828142Z",
     "start_time": "2018-08-09T00:54:28.169870Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>acc_id</th>\n",
       "      <th>para_text</th>\n",
       "      <th>len</th>\n",
       "      <th>emp_header</th>\n",
       "      <th>first_emp_head_block</th>\n",
       "      <th>para_text_orig</th>\n",
       "      <th>para_tag</th>\n",
       "      <th>split</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Specifically, we offer group, individual, and ...</td>\n",
       "      <td>822</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Specifically, we offer group, individual, and ...</td>\n",
       "      <td>&lt;div class=\"c66\"&gt;Specifically, we offer group,...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Our Unum US segment includes group long-term a...</td>\n",
       "      <td>1027</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Our Unum US segment includes group long-term a...</td>\n",
       "      <td>&lt;div class=\"c33\"&gt;Our Unum US segment includes ...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Our Unum UK segment includes insurance for gro...</td>\n",
       "      <td>638</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Our Unum UK segment includes insurance for gro...</td>\n",
       "      <td>&lt;div class=\"c33\"&gt;Our Unum UK segment includes ...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Our Colonial Life segment includes insurance f...</td>\n",
       "      <td>674</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Our Colonial Life segment includes insurance f...</td>\n",
       "      <td>&lt;div class=\"c33\"&gt;Our Colonial Life segment inc...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Employees</td>\n",
       "      <td>9</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Employees</td>\n",
       "      <td>&lt;div class=\"c67\"&gt;Employees&lt;/div&gt;</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>At December 31, 2016, we had approximately 9,4...</td>\n",
       "      <td>69</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>At December 31, 2016, we\\nhad approximately 9,...</td>\n",
       "      <td>&lt;div class=\"c10\"&gt;&lt;span class=\"c34\"&gt;At&lt;/span&gt; &lt;...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>In particular, factors such as unemployment le...</td>\n",
       "      <td>1003</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>In particular, factors such as unemployment le...</td>\n",
       "      <td>&lt;div class=\"c33\"&gt;In particular, factors such a...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Generally, we do not expect our mortality and ...</td>\n",
       "      <td>556</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Generally, we do not expect our mortality and ...</td>\n",
       "      <td>&lt;div class=\"c33\"&gt;Generally, we do not expect o...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Core Market (&lt; 2,000 employees)</td>\n",
       "      <td>31</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Core Market (&lt; 2,000 employees)</td>\n",
       "      <td>&lt;div class=\"c71\"&gt;Core Market (&amp;lt; 2,000 emplo...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Group disability sales decreased compared to 2...</td>\n",
       "      <td>560</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Group disability sales decreased compared to 2...</td>\n",
       "      <td>&lt;div class=\"c68\"&gt;&lt;span class=\"c34\"&gt;Group disab...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Core Market (&lt; 500 employees)</td>\n",
       "      <td>29</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Core Market (&lt; 500 employees)</td>\n",
       "      <td>&lt;div class=\"c71\"&gt;Core Market (&amp;lt; 500 employe...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Group long-term disability sales increased com...</td>\n",
       "      <td>231</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Group long-term disability sales increased com...</td>\n",
       "      <td>&lt;div class=\"c68\"&gt;&lt;span class=\"c34\"&gt;Group long-...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>Core Market (&lt; 1,000 employees)</td>\n",
       "      <td>31</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Core Market (&lt; 1,000 employees)</td>\n",
       "      <td>&lt;div class=\"c71\"&gt;Core Market (&amp;lt; 1,000 emplo...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>We reported year-over-year sales growth in bot...</td>\n",
       "      <td>556</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>We reported year-over-year sales growth in bot...</td>\n",
       "      <td>&lt;div class=\"c68\"&gt;&lt;span class=\"c34\"&gt;We reported...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0000005513-17-000018</td>\n",
       "      <td>We remain committed to providing employees and...</td>\n",
       "      <td>879</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>We remain committed to providing employees and...</td>\n",
       "      <td>&lt;div class=\"c33\"&gt;We remain committed to provid...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0000006281-16-000097</td>\n",
       "      <td>An element of our business strategy involves e...</td>\n",
       "      <td>387</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>An element of our business strategy involves e...</td>\n",
       "      <td>&lt;div class=\"c88\"&gt;An element of our business st...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0000006281-16-000097</td>\n",
       "      <td>Employees</td>\n",
       "      <td>9</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Employees</td>\n",
       "      <td>&lt;div class=\"c87\"&gt;Employees&lt;/div&gt;</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0000006281-16-000097</td>\n",
       "      <td>As of October 29, 2016, we employed approximat...</td>\n",
       "      <td>690</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>As of October 29, 2016, we\\nemployed approxima...</td>\n",
       "      <td>&lt;div class=\"c71\"&gt;&lt;span class=\"c85\"&gt;As of&lt;/span...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0000006281-16-000097</td>\n",
       "      <td>The cyclical nature of the semiconductor indus...</td>\n",
       "      <td>1475</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>The cyclical nature of the semiconductor indus...</td>\n",
       "      <td>&lt;div class=\"c128\"&gt;The cyclical nature of the s...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0000006281-16-000097</td>\n",
       "      <td>An element of our business strategy involves e...</td>\n",
       "      <td>997</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>An element of our business strategy involves e...</td>\n",
       "      <td>&lt;div class=\"c128\"&gt;An element of our business s...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0000006281-16-000097</td>\n",
       "      <td>During fiscal 2016, we recorded a special char...</td>\n",
       "      <td>598</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>During fiscal 2016, we recorded a special char...</td>\n",
       "      <td>&lt;div class=\"c129\"&gt;&lt;span class=\"c85\"&gt;During fis...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0000006281-16-000097</td>\n",
       "      <td>Workforce reductions</td>\n",
       "      <td>20</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Workforce reductions</td>\n",
       "      <td>&lt;div class=\"c34\"&gt;Workforce reductions&lt;/div&gt;</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0000006281-16-000097</td>\n",
       "      <td>During fiscal 2016, the Company recorded speci...</td>\n",
       "      <td>492</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>During fiscal 2016, the Company recorded speci...</td>\n",
       "      <td>&lt;div class=\"c129\"&gt;&lt;span class=\"c85\"&gt;During fis...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0000006281-16-000097</td>\n",
       "      <td>The goodwill recognized is attributable to syn...</td>\n",
       "      <td>411</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>The goodwill recognized is attributable to syn...</td>\n",
       "      <td>&lt;div class=\"c84\"&gt;The goodwill recognized is at...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0000006885-16-000411</td>\n",
       "      <td>At January 30, 2016, we employed approximately...</td>\n",
       "      <td>431</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>At January 30, 2016, we\\nemployed approximatel...</td>\n",
       "      <td>&lt;div class=\"c46\"&gt;&lt;span class=\"c26\"&gt;At&lt;/span&gt; &lt;...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0000006885-16-000411</td>\n",
       "      <td>Forward-looking statements are based upon a nu...</td>\n",
       "      <td>1952</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Forward-looking statements are based upon a nu...</td>\n",
       "      <td>&lt;div class=\"c47\"&gt;Forward-looking statements ar...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0000006885-16-000411</td>\n",
       "      <td>An economic downturn or decline in consumer co...</td>\n",
       "      <td>1815</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>An economic downturn or decline in consumer co...</td>\n",
       "      <td>&lt;div class=\"c46\"&gt;&lt;span class=\"c9\"&gt;An economic ...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0000006885-16-000411</td>\n",
       "      <td>The following supplemental information present...</td>\n",
       "      <td>1608</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>The following supplemental information present...</td>\n",
       "      <td>&lt;div class=\"c46\"&gt;&lt;span class=\"c26\"&gt;The followi...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0000006885-16-000411</td>\n",
       "      <td>Severance charges associated with workforce re...</td>\n",
       "      <td>107</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Severance charges associated with workforce re...</td>\n",
       "      <td>&lt;div class=\"c176\"&gt;Severance charges associated...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0000006885-16-000411</td>\n",
       "      <td>Severance charges associated with workforce re...</td>\n",
       "      <td>87</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Severance charges associated with workforce re...</td>\n",
       "      <td>&lt;div class=\"c176\"&gt;Severance charges associated...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>0000006885-16-000411</td>\n",
       "      <td>Selling, general and administrative (\"SG&amp;A\") e...</td>\n",
       "      <td>576</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Selling, general and administrative (\"SG&amp;A\") e...</td>\n",
       "      <td>&lt;div class=\"c46\"&gt;&lt;span class=\"c26\"&gt;Selling, ge...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>0000006885-16-000411</td>\n",
       "      <td>401(k) Plan. We have a contributory 401(k) sav...</td>\n",
       "      <td>739</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>401(k) Plan. We have a\\ncontributory 401(k) sa...</td>\n",
       "      <td>&lt;div class=\"c46\"&gt;&lt;span class=\"c42\"&gt;401(k) Plan...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>0000006951-16-000068</td>\n",
       "      <td>Employees</td>\n",
       "      <td>9</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Employees</td>\n",
       "      <td>&lt;div class=\"c109\"&gt;Employees&lt;/div&gt;</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>0000006951-16-000068</td>\n",
       "      <td>At October 30, 2016, Applied employed approxim...</td>\n",
       "      <td>334</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>At October 30, 2016,\\nApplied employed approxi...</td>\n",
       "      <td>&lt;div class=\"c69\"&gt;&lt;span class=\"c31\"&gt;At&lt;/span&gt; &lt;...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>0000006951-16-000068</td>\n",
       "      <td>a diverse workforce with different experience ...</td>\n",
       "      <td>180</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>a diverse workforce with different experience ...</td>\n",
       "      <td>&lt;div class=\"c135\"&gt;a diverse workforce with dif...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>0000006951-16-000068</td>\n",
       "      <td>the increasing need for a mobile workforce to ...</td>\n",
       "      <td>89</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>the increasing need for a mobile workforce to ...</td>\n",
       "      <td>&lt;div class=\"c135\"&gt;the increasing need for a mo...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>0000006951-16-000068</td>\n",
       "      <td>Applied's success, competitiveness and ability...</td>\n",
       "      <td>1176</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Applied's success, competitiveness and ability...</td>\n",
       "      <td>&lt;div class=\"c140\"&gt;Applied's success, competiti...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>0000006951-16-000068</td>\n",
       "      <td>New orders and net sales for fiscal 2016 sligh...</td>\n",
       "      <td>744</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>New orders and net sales for fiscal 2016 sligh...</td>\n",
       "      <td>&lt;div class=\"c248\"&gt;&lt;span class=\"c31\"&gt;New orders...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>0000007039-17-000009</td>\n",
       "      <td>The number of our regular, U.S. based employee...</td>\n",
       "      <td>510</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>The number of our regular, U.S. based employee...</td>\n",
       "      <td>&lt;div class=\"c53\"&gt;The number of our regular, U....</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>0000007039-17-000009</td>\n",
       "      <td>The AMAK facility includes an underground mine...</td>\n",
       "      <td>1704</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>The AMAK facility includes an underground mine...</td>\n",
       "      <td>&lt;div class=\"c53\"&gt;The AMAK facility includes an...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>0000007039-17-000009</td>\n",
       "      <td>•     Geographically dispersed workforce;</td>\n",
       "      <td>41</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>•     Geographically dispersed workforce;</td>\n",
       "      <td>&lt;div class=\"c53\"&gt;•     Geographically disperse...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>0000007039-17-000009</td>\n",
       "      <td>The facility includes an underground mine, ore...</td>\n",
       "      <td>819</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>The facility includes an underground mine, ore...</td>\n",
       "      <td>&lt;div class=\"c53\"&gt;The facility includes an unde...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>0000007039-17-000009</td>\n",
       "      <td>Labor costs declined 0.2% from 2015 to 2016.  ...</td>\n",
       "      <td>174</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>Labor costs declined 0.2% from 2015 to 2016.  ...</td>\n",
       "      <td>&lt;div class=\"c53\"&gt;Labor costs declined 0.2% fro...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>0000007039-17-000009</td>\n",
       "      <td>In July 2015 we entered into a retirement agre...</td>\n",
       "      <td>433</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>In July 2015 we entered into a retirement agre...</td>\n",
       "      <td>&lt;div class=\"c371\"&gt;In July 2015 we entered into...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  acc_id                                          para_text  \\\n",
       "0   0000005513-17-000018  Specifically, we offer group, individual, and ...   \n",
       "1   0000005513-17-000018  Our Unum US segment includes group long-term a...   \n",
       "2   0000005513-17-000018  Our Unum UK segment includes insurance for gro...   \n",
       "3   0000005513-17-000018  Our Colonial Life segment includes insurance f...   \n",
       "4   0000005513-17-000018                                          Employees   \n",
       "5   0000005513-17-000018  At December 31, 2016, we had approximately 9,4...   \n",
       "6   0000005513-17-000018  In particular, factors such as unemployment le...   \n",
       "7   0000005513-17-000018  Generally, we do not expect our mortality and ...   \n",
       "8   0000005513-17-000018                    Core Market (< 2,000 employees)   \n",
       "9   0000005513-17-000018  Group disability sales decreased compared to 2...   \n",
       "10  0000005513-17-000018                      Core Market (< 500 employees)   \n",
       "11  0000005513-17-000018  Group long-term disability sales increased com...   \n",
       "12  0000005513-17-000018                    Core Market (< 1,000 employees)   \n",
       "13  0000005513-17-000018  We reported year-over-year sales growth in bot...   \n",
       "14  0000005513-17-000018  We remain committed to providing employees and...   \n",
       "15  0000006281-16-000097  An element of our business strategy involves e...   \n",
       "16  0000006281-16-000097                                          Employees   \n",
       "17  0000006281-16-000097  As of October 29, 2016, we employed approximat...   \n",
       "18  0000006281-16-000097  The cyclical nature of the semiconductor indus...   \n",
       "19  0000006281-16-000097  An element of our business strategy involves e...   \n",
       "20  0000006281-16-000097  During fiscal 2016, we recorded a special char...   \n",
       "21  0000006281-16-000097                               Workforce reductions   \n",
       "22  0000006281-16-000097  During fiscal 2016, the Company recorded speci...   \n",
       "23  0000006281-16-000097  The goodwill recognized is attributable to syn...   \n",
       "24  0000006885-16-000411  At January 30, 2016, we employed approximately...   \n",
       "25  0000006885-16-000411  Forward-looking statements are based upon a nu...   \n",
       "26  0000006885-16-000411  An economic downturn or decline in consumer co...   \n",
       "27  0000006885-16-000411  The following supplemental information present...   \n",
       "28  0000006885-16-000411  Severance charges associated with workforce re...   \n",
       "29  0000006885-16-000411  Severance charges associated with workforce re...   \n",
       "30  0000006885-16-000411  Selling, general and administrative (\"SG&A\") e...   \n",
       "31  0000006885-16-000411  401(k) Plan. We have a contributory 401(k) sav...   \n",
       "32  0000006951-16-000068                                          Employees   \n",
       "33  0000006951-16-000068  At October 30, 2016, Applied employed approxim...   \n",
       "34  0000006951-16-000068  a diverse workforce with different experience ...   \n",
       "35  0000006951-16-000068  the increasing need for a mobile workforce to ...   \n",
       "36  0000006951-16-000068  Applied's success, competitiveness and ability...   \n",
       "37  0000006951-16-000068  New orders and net sales for fiscal 2016 sligh...   \n",
       "38  0000007039-17-000009  The number of our regular, U.S. based employee...   \n",
       "39  0000007039-17-000009  The AMAK facility includes an underground mine...   \n",
       "40  0000007039-17-000009          •     Geographically dispersed workforce;   \n",
       "41  0000007039-17-000009  The facility includes an underground mine, ore...   \n",
       "42  0000007039-17-000009  Labor costs declined 0.2% from 2015 to 2016.  ...   \n",
       "43  0000007039-17-000009  In July 2015 we entered into a retirement agre...   \n",
       "\n",
       "     len  emp_header  first_emp_head_block  \\\n",
       "0    822       False                 False   \n",
       "1   1027       False                 False   \n",
       "2    638       False                 False   \n",
       "3    674       False                 False   \n",
       "4      9       False                 False   \n",
       "5     69       False                 False   \n",
       "6   1003       False                 False   \n",
       "7    556       False                 False   \n",
       "8     31       False                 False   \n",
       "9    560       False                 False   \n",
       "10    29       False                 False   \n",
       "11   231       False                 False   \n",
       "12    31       False                 False   \n",
       "13   556       False                 False   \n",
       "14   879       False                 False   \n",
       "15   387       False                 False   \n",
       "16     9       False                 False   \n",
       "17   690       False                 False   \n",
       "18  1475       False                 False   \n",
       "19   997       False                 False   \n",
       "20   598       False                 False   \n",
       "21    20       False                 False   \n",
       "22   492       False                 False   \n",
       "23   411       False                 False   \n",
       "24   431       False                 False   \n",
       "25  1952       False                 False   \n",
       "26  1815       False                 False   \n",
       "27  1608       False                 False   \n",
       "28   107       False                 False   \n",
       "29    87       False                 False   \n",
       "30   576       False                 False   \n",
       "31   739       False                 False   \n",
       "32     9       False                 False   \n",
       "33   334       False                 False   \n",
       "34   180       False                 False   \n",
       "35    89       False                 False   \n",
       "36  1176       False                 False   \n",
       "37   744       False                 False   \n",
       "38   510       False                 False   \n",
       "39  1704       False                 False   \n",
       "40    41       False                 False   \n",
       "41   819       False                 False   \n",
       "42   174       False                 False   \n",
       "43   433       False                 False   \n",
       "\n",
       "                                       para_text_orig  \\\n",
       "0   Specifically, we offer group, individual, and ...   \n",
       "1   Our Unum US segment includes group long-term a...   \n",
       "2   Our Unum UK segment includes insurance for gro...   \n",
       "3   Our Colonial Life segment includes insurance f...   \n",
       "4                                           Employees   \n",
       "5   At December 31, 2016, we\\nhad approximately 9,...   \n",
       "6   In particular, factors such as unemployment le...   \n",
       "7   Generally, we do not expect our mortality and ...   \n",
       "8                     Core Market (< 2,000 employees)   \n",
       "9   Group disability sales decreased compared to 2...   \n",
       "10                      Core Market (< 500 employees)   \n",
       "11  Group long-term disability sales increased com...   \n",
       "12                    Core Market (< 1,000 employees)   \n",
       "13  We reported year-over-year sales growth in bot...   \n",
       "14  We remain committed to providing employees and...   \n",
       "15  An element of our business strategy involves e...   \n",
       "16                                          Employees   \n",
       "17  As of October 29, 2016, we\\nemployed approxima...   \n",
       "18  The cyclical nature of the semiconductor indus...   \n",
       "19  An element of our business strategy involves e...   \n",
       "20  During fiscal 2016, we recorded a special char...   \n",
       "21                               Workforce reductions   \n",
       "22  During fiscal 2016, the Company recorded speci...   \n",
       "23  The goodwill recognized is attributable to syn...   \n",
       "24  At January 30, 2016, we\\nemployed approximatel...   \n",
       "25  Forward-looking statements are based upon a nu...   \n",
       "26  An economic downturn or decline in consumer co...   \n",
       "27  The following supplemental information present...   \n",
       "28  Severance charges associated with workforce re...   \n",
       "29  Severance charges associated with workforce re...   \n",
       "30  Selling, general and administrative (\"SG&A\") e...   \n",
       "31  401(k) Plan. We have a\\ncontributory 401(k) sa...   \n",
       "32                                          Employees   \n",
       "33  At October 30, 2016,\\nApplied employed approxi...   \n",
       "34  a diverse workforce with different experience ...   \n",
       "35  the increasing need for a mobile workforce to ...   \n",
       "36  Applied's success, competitiveness and ability...   \n",
       "37  New orders and net sales for fiscal 2016 sligh...   \n",
       "38  The number of our regular, U.S. based employee...   \n",
       "39  The AMAK facility includes an underground mine...   \n",
       "40          •     Geographically dispersed workforce;   \n",
       "41  The facility includes an underground mine, ore...   \n",
       "42  Labor costs declined 0.2% from 2015 to 2016.  ...   \n",
       "43  In July 2015 we entered into a retirement agre...   \n",
       "\n",
       "                                             para_tag  split  label  \n",
       "0   <div class=\"c66\">Specifically, we offer group,...  train      0  \n",
       "1   <div class=\"c33\">Our Unum US segment includes ...  train      0  \n",
       "2   <div class=\"c33\">Our Unum UK segment includes ...  train      0  \n",
       "3   <div class=\"c33\">Our Colonial Life segment inc...  train      0  \n",
       "4                    <div class=\"c67\">Employees</div>  train      0  \n",
       "5   <div class=\"c10\"><span class=\"c34\">At</span> <...  train      0  \n",
       "6   <div class=\"c33\">In particular, factors such a...  train      0  \n",
       "7   <div class=\"c33\">Generally, we do not expect o...  train      0  \n",
       "8   <div class=\"c71\">Core Market (&lt; 2,000 emplo...  train      0  \n",
       "9   <div class=\"c68\"><span class=\"c34\">Group disab...  train      0  \n",
       "10  <div class=\"c71\">Core Market (&lt; 500 employe...  train      0  \n",
       "11  <div class=\"c68\"><span class=\"c34\">Group long-...  train      0  \n",
       "12  <div class=\"c71\">Core Market (&lt; 1,000 emplo...  train      0  \n",
       "13  <div class=\"c68\"><span class=\"c34\">We reported...  train      0  \n",
       "14  <div class=\"c33\">We remain committed to provid...  train      0  \n",
       "15  <div class=\"c88\">An element of our business st...  train      0  \n",
       "16                   <div class=\"c87\">Employees</div>  train      0  \n",
       "17  <div class=\"c71\"><span class=\"c85\">As of</span...  train      0  \n",
       "18  <div class=\"c128\">The cyclical nature of the s...  train      0  \n",
       "19  <div class=\"c128\">An element of our business s...  train      0  \n",
       "20  <div class=\"c129\"><span class=\"c85\">During fis...  train      0  \n",
       "21        <div class=\"c34\">Workforce reductions</div>  train      0  \n",
       "22  <div class=\"c129\"><span class=\"c85\">During fis...  train      0  \n",
       "23  <div class=\"c84\">The goodwill recognized is at...  train      0  \n",
       "24  <div class=\"c46\"><span class=\"c26\">At</span> <...  train      0  \n",
       "25  <div class=\"c47\">Forward-looking statements ar...  train      0  \n",
       "26  <div class=\"c46\"><span class=\"c9\">An economic ...  train      0  \n",
       "27  <div class=\"c46\"><span class=\"c26\">The followi...  train      0  \n",
       "28  <div class=\"c176\">Severance charges associated...  train      0  \n",
       "29  <div class=\"c176\">Severance charges associated...  train      0  \n",
       "30  <div class=\"c46\"><span class=\"c26\">Selling, ge...  train      0  \n",
       "31  <div class=\"c46\"><span class=\"c42\">401(k) Plan...  train      0  \n",
       "32                  <div class=\"c109\">Employees</div>  train      0  \n",
       "33  <div class=\"c69\"><span class=\"c31\">At</span> <...  train      0  \n",
       "34  <div class=\"c135\">a diverse workforce with dif...  train      0  \n",
       "35  <div class=\"c135\">the increasing need for a mo...  train      0  \n",
       "36  <div class=\"c140\">Applied's success, competiti...  train      0  \n",
       "37  <div class=\"c248\"><span class=\"c31\">New orders...  train      0  \n",
       "38  <div class=\"c53\">The number of our regular, U....  train      0  \n",
       "39  <div class=\"c53\">The AMAK facility includes an...  train      0  \n",
       "40  <div class=\"c53\">•     Geographically disperse...  train      0  \n",
       "41  <div class=\"c53\">The facility includes an unde...  train      0  \n",
       "42  <div class=\"c53\">Labor costs declined 0.2% fro...  train      0  \n",
       "43  <div class=\"c371\">In July 2015 we entered into...  train      0  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = extract_paragraph_df(html_path_list[5:10], emp_pats, emp_head, emp_head_raw, head_block_re)['paragraphs']\n",
    "test_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This part takes some time, so it's ideal to split the work and run in parallel (even four notebooks running locally cuts the time from over 4 hours to a little over 1 hour - the task is mainly CPU bound). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create dataframes and save as CSVs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:55:27.620018Z",
     "start_time": "2018-08-09T00:54:57.829143Z"
    }
   },
   "outputs": [],
   "source": [
    "results_dict = extract_paragraph_df(html_path_list[5:10], emp_pats, emp_head_raw, head_block_re)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:55:27.635026Z",
     "start_time": "2018-08-09T00:55:27.621016Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 44 entries, 0 to 43\n",
      "Data columns (total 10 columns):\n",
      "acc_id                  44 non-null object\n",
      "para_text               44 non-null object\n",
      "len                     44 non-null int64\n",
      "emp_header              44 non-null bool\n",
      "first_emp_head_block    44 non-null bool\n",
      "para_text_orig          44 non-null object\n",
      "para_tag                44 non-null object\n",
      "split                   44 non-null object\n",
      "label                   44 non-null int64\n",
      "idx                     44 non-null int64\n",
      "dtypes: bool(2), int64(3), object(5)\n",
      "memory usage: 2.9+ KB\n"
     ]
    }
   ],
   "source": [
    "paragraph_input_df = results_dict['paragraphs']\n",
    "validation_id_bs = paragraph_input_df.acc_id.isin(val_accession_ids)\n",
    "paragraph_input_df.loc[validation_id_bs, 'split'] = 'val'\n",
    "paragraph_input_df.loc[:,'idx'] = paragraph_input_df.index.values\n",
    "\n",
    "paragraph_input_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:52:50.240346Z",
     "start_time": "2018-08-09T00:52:50.230339Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 0 entries\n",
      "Data columns (total 4 columns):\n",
      "acc_id      0 non-null float64\n",
      "split       0 non-null object\n",
      "tbl_html    0 non-null float64\n",
      "idx         0 non-null int64\n",
      "dtypes: float64(2), int64(1), object(1)\n",
      "memory usage: 76.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "tbl_html_df = results_dict['tables']\n",
    "tbl_html_df.loc[tbl_html_df.acc_id.isin(val_accession_ids),'split'] = 'val'\n",
    "tbl_html_df.loc[:,'idx'] = tbl_html_df.index.values\n",
    "\n",
    "tbl_html_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-07-23T18:33:49.467984Z",
     "start_time": "2018-07-23T18:33:49.325164Z"
    }
   },
   "outputs": [],
   "source": [
    "#### Write to csv for later use\n",
    "paragraph_input_df.to_csv('data/paragraph_input_df_1.csv')\n",
    "tbl_html_df.to_csv('data/tbl_html_df_1.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-07-23T18:33:49.498426Z",
     "start_time": "2018-07-23T18:33:49.467984Z"
    }
   },
   "outputs": [],
   "source": [
    "train_df = paragraph_input_df[paragraph_input_df.split == 'train']\n",
    "\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Debug odd findings to refine regex or code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Code for printing results from paragraph search for a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-08-09T00:53:09.363683Z",
     "start_time": "2018-08-09T00:53:09.294639Z"
    },
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "def print_parse_objects(file: str, use_acc_id: bool=True, acc_id_list = full_accession_ids, \n",
    "                       file_path_list = html_path_list):\n",
    "    fl = file\n",
    "    if use_acc_id:\n",
    "        fl = html_path_list[full_accession_ids.index(fl)]        \n",
    "        print('Index number: ')\n",
    "        print(full_accession_ids.index(file))\n",
    "    print('File path: ')\n",
    "    print(fl)\n",
    "    #for i, fl in enumerate(html_path_list[303:304]):\n",
    "    with open(fl, encoding=\"utf8\") as file: \n",
    "        file_html = file.read()\n",
    "        soup = bs(file_html, 'lxml')\n",
    "        emp_head_flag = False\n",
    "        emp_head_first_match = False\n",
    "        if re.search(emp_head_raw, file_html):\n",
    "            for ihead, hblock in enumerate(soup.body.find_all(string=emp_head)):\n",
    "                emp_head_tag = hblock.find_parent(name=head_block_re)\n",
    "\n",
    "                if emp_head_tag != None and emp_head_tag.name != 'table' and emp_head_tag.find_parent('table') == None:\n",
    "                    emp_head_matched = False\n",
    "                    print(ihead); \n",
    "                    try:\n",
    "                        print(emp_head_tag.name) \n",
    "                    except:\n",
    "                        print('emp_head_tag has no name')\n",
    "                    print(emp_head_tag)\n",
    "                    for i2, block in enumerate(emp_head_tag.find_next_siblings(block_re, limit=6)):\n",
    "                        print('Block sibling number: ' + str(i2))\n",
    "                        if block.find(string=emp_pats) != None:\n",
    "                            block_tag = copy.copy(block)\n",
    "                            print('Found match!'); print(block_tag) ; print(block_tag.get_text())\n",
    "                        if block.find('table') != None:\n",
    "                            print('Found table match!')\n",
    "                            tbl_block_tag = copy.copy(block)\n",
    "                            tbl_df = pd.read_html(block.find('table').prettify())[0].dropna(axis=1,how='all')\n",
    "                            print(tbl_df); print(tbl_df.info()); print(block)\n",
    "        #else:\n",
    "        print('No Employees header')\n",
    "        soup_emp_count = soup.body.find_all(string=[emp_pats])\n",
    "        soup_emp_paras = [x.find_parent(name=head_block_re) for x in soup_emp_count]\n",
    "        soup_emp_paras = [x for x in soup_emp_paras if x != None]\n",
    "        soup_emp_blocks = [x  for x in soup_emp_count if x.name in ['p', 'div']]\n",
    "        for i2, block in enumerate(soup_emp_paras):\n",
    "            block_tag = copy.copy(block)\n",
    "            print('Para number: ' + str(i2)); \n",
    "            try:\n",
    "                print('Original match name:  ' +  soup_emp_count[i2].name)\n",
    "            except:\n",
    "                print('Original match has no name')\n",
    "            print('Original match:  ' +  soup_emp_count[i2])\n",
    "            try:\n",
    "                print('Block match name:  ' +  block_tag.name)\n",
    "            except:\n",
    "                print('Block match has no name')\n",
    "            print('Block:  ')\n",
    "            print(block)\n",
    "            print(type(block))\n",
    "            print('Block Text:  ')\n",
    "            print(block.get_text().replace('\\n', ' '))\n",
    "        for i2, block in enumerate(soup_emp_blocks):\n",
    "            print('Block number: ' + str(i2)); print(block)\n",
    "#emp2_tag.find_next_siblings(block_re, limit = 1, string=False)[0].find(string=[emp_pats]).parent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Code for investigating files for which no candidate paragraphs were identified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-29T15:04:35.677889Z",
     "start_time": "2018-06-29T15:04:35.650871Z"
    }
   },
   "outputs": [],
   "source": [
    "missed_ids = [x for x in full_accession_ids[:100] if x not in set(acc_id_list)]\n",
    "# For investigating files with no candidate paragraphs if they are part of the training set\n",
    "#[x for x in missed_ids if x in train_accession_ids]  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 350,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-27T20:51:38.295108Z",
     "start_time": "2018-06-27T20:51:38.290106Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[89]"
      ]
     },
     "execution_count": 350,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#find index of file for specific accession id\n",
    "#[i for i, x in enumerate(html_path_list[:300]) if re.search(r\"0000042316-17-000014\", x)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore dataframe of paragraphs for training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 460,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-28T18:53:02.487241Z",
     "start_time": "2018-06-28T18:53:02.472232Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>acc_id</th>\n",
       "      <th>para_text</th>\n",
       "      <th>len</th>\n",
       "      <th>emp_header</th>\n",
       "      <th>first_emp_head_block</th>\n",
       "      <th>para_text_orig</th>\n",
       "      <th>para_tag</th>\n",
       "      <th>split</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0000004127-16-000068</td>\n",
       "      <td>As of September 30, 2016, we employed approxim...</td>\n",
       "      <td>245</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>As of September 30, 2016,\\nwe employed approxi...</td>\n",
       "      <td>&lt;div class=\"c80\"&gt;&lt;span class=\"c32\"&gt;As of&lt;/span...</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0000004127-16-000068</td>\n",
       "      <td>EMPLOYEES</td>\n",
       "      <td>9</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>EMPLOYEES</td>\n",
       "      <td>&lt;div class=\"c90\"&gt;EMPLOYEES&lt;/div&gt;</td>\n",
       "      <td>train</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 acc_id                                          para_text  \\\n",
       "2  0000004127-16-000068  As of September 30, 2016, we employed approxim...   \n",
       "3  0000004127-16-000068                                          EMPLOYEES   \n",
       "\n",
       "   len  emp_header  first_emp_head_block  \\\n",
       "2  245        True                  True   \n",
       "3    9       False                 False   \n",
       "\n",
       "                                      para_text_orig  \\\n",
       "2  As of September 30, 2016,\\nwe employed approxi...   \n",
       "3                                          EMPLOYEES   \n",
       "\n",
       "                                            para_tag  split  label  \n",
       "2  <div class=\"c80\"><span class=\"c32\">As of</span...  train      0  \n",
       "3                   <div class=\"c90\">EMPLOYEES</div>  train      0  "
      ]
     },
     "execution_count": 460,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-29T15:04:47.957461Z",
     "start_time": "2018-06-29T15:04:47.931445Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>len</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>6170.000000</td>\n",
       "      <td>6170.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>415.585575</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>475.612180</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>8.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>26.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>300.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>618.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>5054.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               len   label\n",
       "count  6170.000000  6170.0\n",
       "mean    415.585575     0.0\n",
       "std     475.612180     0.0\n",
       "min       8.000000     0.0\n",
       "25%      26.000000     0.0\n",
       "50%     300.000000     0.0\n",
       "75%     618.000000     0.0\n",
       "max    5054.000000     0.0"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 490,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-28T20:21:36.047911Z",
     "start_time": "2018-06-28T20:21:36.030900Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>len_counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5054</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4944</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4868</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4213</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3972</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3904</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3862</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3795</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3791</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3605</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3559</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3413</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3383</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3343</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3331</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3306</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3302</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3191</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3171</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2857</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2830</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2802</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2510</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2490</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2477</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2397</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2369</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2344</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2340</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1119 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      len_counts\n",
       "5054           1\n",
       "4944           1\n",
       "4868           1\n",
       "4213           1\n",
       "3972           1\n",
       "3904           1\n",
       "3878           1\n",
       "3862           1\n",
       "3795           1\n",
       "3791           1\n",
       "3605           1\n",
       "3559           1\n",
       "3413           1\n",
       "3383           1\n",
       "3343           1\n",
       "3331           1\n",
       "3306           1\n",
       "3302           1\n",
       "3191           1\n",
       "3171           1\n",
       "2857           1\n",
       "2830           1\n",
       "2802           1\n",
       "2510           1\n",
       "2490           1\n",
       "2477           1\n",
       "2397           1\n",
       "2369           1\n",
       "2344           1\n",
       "2340           1\n",
       "...          ...\n",
       "37             5\n",
       "36             4\n",
       "35             4\n",
       "34             4\n",
       "33             3\n",
       "32             8\n",
       "31            15\n",
       "30            21\n",
       "29            24\n",
       "28             2\n",
       "27             2\n",
       "26             6\n",
       "25             6\n",
       "24            10\n",
       "23             6\n",
       "22             4\n",
       "21            11\n",
       "20             8\n",
       "19             8\n",
       "18             5\n",
       "17            12\n",
       "16            14\n",
       "15            13\n",
       "14             6\n",
       "13            10\n",
       "12            13\n",
       "11             6\n",
       "10            44\n",
       "9            697\n",
       "8              3\n",
       "\n",
       "[1119 rows x 1 columns]"
      ]
     },
     "execution_count": 490,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.len.value_counts().to_frame('len_counts').sort_index(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-29T15:05:16.998791Z",
     "start_time": "2018-06-29T15:05:16.971774Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------- 0001193125-17-106055 -----------------------------------\n",
      "len  :5054\n",
      "\n",
      "emp_header  :False\n",
      "\n",
      "first_emp_head_block  :False\n",
      "\n",
      "para_text  :Current format     Previous format                             Millions of Euros       Millions of Euros    ASSETS   December  2015     December  2014      ASSETS   December  2015     December  2014              CASH, CASH BALANCES AT CENTRAL BANKS AND OTHER DEMAND DEPOSITS (1)   29,282    27,719     CASH AND BALANCES WITH CENTRAL BANKS   43,467    31,430    FINANCIAL ASSETS HELD FOR TRADING   78,326    83,258     FINANCIAL ASSETS HELD FOR TRADING   78,326    83,258    Derivatives   40,902    44,229     Loans and advances to credit institutions   -    -    Equity instruments   4,534    5,017     Loans and advances to customers   65    128    Debt securities   32,825    33,883     Debt securities   32,825    33,883    Loans and advances to central banks   -    -     Equity instruments   4,534    5,017    Loans and advances to credit institutions   -    -     Trading derivatives   40,902    44,229    Loans and advances to customers   65    128                         OTHER FINANCIAL ASSETS DESIGNATED AT FAIR VALUE THROUGH PROFIT OR LOSS   2,311    2,761     OTHER FINANCIAL ASSETS DESIGNATED AT FAIR VALUE THROUGH PROFIT OR LOSS   2,311    2,761    Equity instruments   2,075    2,024     Loans and advances to credit institutions   62    -    Debt securities   173    737     Loans and advances to customers   -    -    Loans and advances to central banks   -    -     Debt securities   173    737    Loans and advances to credit institutions   62    -     Equity instruments   2,075    2,024    Loans and advances to customers   -    -                AVAILABLE-FOR-SALE FINANCIAL ASSETS   113,426    94,875     AVAILABLE-FOR-SALE FINANCIAL ASSETS   113,426    94,875    Equity instruments   5,116    7,267     Debt securities   108,310    87,608    Debt securities   108,310    87,608     Equity instruments   5,116    7,267    LOANS AND RECEIVABLES (2)   471,828    376,086     LOANS AND RECEIVABLES   457,644    372,375    Debt securities   10,516    6,659     Loans and advances to credit institutions   32,962    27,059    Loans and advances to central banks   17,830    5,429     Loans and advances to customers   414,165    338,657    Loans and advances to credit institutions   29,317    25,342     Debt securities   10,516    6,659    Loans and advances to customers   414,165    338,657                HELD-TO-MATURITY INVESTMENTS   -    -     HELD-TO-MATURITY INVESTMENTS   -    -    HEDGING DERIVATIVES   3,538    2,551     FAIR VALUE CHANGES OF THE HEDGED ITEMS IN PORTFOLIO HEDGES OF INTEREST RATE RISK   45    121    FAIR VALUE CHANGES OF THE HEDGED ITEMS IN PORTFOLIO HEDGES OF INTEREST RATE RISK   45    121     HEDGING DERIVATIVES   3,538    2,551    INVESTMENTS IN SUBSIDARIES, JOINT VENTURES AND ASSOCIATES   879    4,509     NON-CURRENT ASSETS HELD FOR SALE   3,369    3,793    Joint ventures   243    4,092     INVESTMENTS IN ENTITIES ACCOUNTED FOR USING THE EQUITY METHOD   879    4,509    Associates   636    417     Associates   636    417    REINSURANCE ASSETS   511    559     Joint ventures   243    4,092    TANGIBLE ASSETS   9,944    7,820     INSURANCE CONTRACTS LINKED TO PENSIONS   -    -    Property, plants and equipment   8,477    6,428     REINSURANCE ASSETS   511    559    For own use   8,021    5,985     TANGIBLE ASSETS   9,944    7,820    Other assets leased out under an operating lease   456    443     Property, plants and equipment   8,477    6,428    Investment properties   1,467    1,392     For own use   8,021    5,985    INTANGIBLE ASSETS   10,052    7,371     Other assets leased out under an operating lease   456    443    Goodwill   6,915    5,697     Investment properties   1,467    1,392    Other intangible assets   3,137    1,673     INTANGIBLE ASSETS   10,052    7,371    TAX ASSETS   17,779    12,426     Goodwill   6,915    5,697    Current   1,901    2,035     Other intangible assets   3,137    1,673    Deferred   15,878    10,391     TAX ASSETS   17,779    12,426    OTHER ASSETS   8,566    8,094     Current   1,901    2,035    Insurance contracts linked to pensions   -    -     Deferred   15,878    10,391    Inventories   4,303    4,443     OTHER ASSETS   8,566    8,094    Rest   4,263    3,651     Inventories   4,303    4,443    NON-CURRENT ASSETS AND DISPOSAL GROUPS HELD FOR SALE (3)   3,369    3,793     Rest   4,263    3,651    TOTAL ASSETS   749,855    631,942     TOTAL ASSETS   749,855    631,942      (1) The main differences with regard to the heading \" Cash and deposits with central banks\" are the inclusion of the balances deposited in Credit Institutions and the reclassification of Loans at central banks to the heading Loans and Receivables (2) The main differences with regard to the heading \"Loans and receivables\" are the inclusion of loans at central banks and the reclassification of balances deposited in Credit institutions to the heading \"Cash, cash balances at central banks and other demand deposits\" (3) Corresponding to the heading \"Non-current assest held-for-sale\" of the previous format\n",
      "\n",
      "----------------------------------- 0001047469-17-006466 -----------------------------------\n",
      "len  :4944\n",
      "\n",
      "emp_header  :False\n",
      "\n",
      "first_emp_head_block  :False\n",
      "\n",
      "para_text  :(a) Represents:  (i) costs and expenses in connection with various restructuring initiatives since the AEA Acquisition, including severance costs, relocation costs, recruiting expenses, write-offs of inventory and fixed assets and third-party consultant costs to assist with these initiatives (includes (A) $16.9 million in fiscal 2016 (all of which is reflected as a component of Restructuring charges in \"Note 12 Restructuring and Related Charges\" to our audited consolidated financial statements) and $19.2 million for the nine months ended June 30, 2017 (all of which is reflected as a component of Restructuring charges in \"Note 11 Restructuring and Related Charges\" to our unaudited condensed consolidated financial statements) related to our voluntary separation plan pursuant to which approximately 220 employees accepted separation packages, and (B) $12.4 million (of which $4.4 million is reflected as a component of Restructuring charges in \"Note 12 Restructuring and Related Charges\" to our audited consolidated financial statements) in the Successor Period, $9.2 million (of which $6.5 million is reflected as a component of Restructuring charges in \"Note 12 Restructuring and Related Charges\" to our audited consolidated financial statements) in fiscal 2015, $11.1 million (all of which is reflected as a component of Restructuring charges in \"Note 12 Restructuring and Related Charges\" to our audited consolidated financial statements) in fiscal 2016, $9.3 million (of which $8.1 million is reflected as a component of Restructuring charges in \"Note 11 Restructuring and Related Charges\" to our unaudited condensed consolidated financial statements) in the nine months ended June 30, 2016 and $6.5 million (of which $2.9 million is reflected as a component of Restructuring charges in \"Note 11 Restructuring and Related Charges\" to our unaudited condensed consolidated financial statements) in the nine months ended June 30, 2017 related to various other initiatives implemented to restructure and reorganize our business as a standalone business with an appropriate management team and cost structure). Differences between amounts reflected as Restructuring charges in \"Note 12 Restructuring and Related Charges\" to our audited financial statements and amounts reflected in this adjustment relate primarily to retention payments made to employees in the Successor Period of $5.6 million that have been reflected as components of Cost of product sales and services ($1.2 million), Research and development expense ($0.2 million), Sales and marketing expense ($1.9 million) and General and administrative expense ($2.3 million) as well as other administrative costs associated with separation from the Predecessor that have been reflected in the Successor Period as a component of General and administrative expense (2.4 million). Differences between amounts reflected as Restructuring charges in \"Note 12 Restructuring and Related Charges\" to our audited financial statements and amounts reflected in this adjustment also relate to retention payments made to employees in fiscal 2015 of $2.5 million that have been reflected as components of Cost of product sales and services ($1.1 million), Research and development expense ($0.1 million), Sales and marketing expense ($1.0 million) and General and administrative expense ($0.3 million). Differences between amounts reflected as Restructuring charges in \"Note 11 Restructuring and Related Charges\" to our unaudited condensed consolidated financial statements in the nine months ended June 30, 2016 and June 30, 2017 and amounts reflected in this adjustment relate primarily to consulting costs and other charges related to implementing such initiatives that have primarily been recorded as a component of General and administrative expense ($1.2 million) in the nine months ended June 30, 2016 and Cost of product sales and services ($1.4 million), Sales and marketing expenses ($1.0 million) and General and administrative expense ($1.0 million) in the nine months ended June 30, 2017; (ii) legal settlement costs and intellectual property related fees associated with legacy matters prior to the AEA Acquisition, including fees and settlement costs related to product warranty litigation on MEMCOR products and certain discontinued products ($11.7 million in the Successor Period, reflected as a component of Cost of product sales and services, $11.7 million in fiscal 2015, reflected as components of Cost of product sales and services ($10.7 million) and General and administrative expenses ($1.0 million), $5.1 million, primarily reflected as a component of Cost of product sales and services in fiscal 2016, $2.5 million, primarily reflected as a component of Cost of product sales and services in the nine months ended June 30, 2016 and $1.8 million, primarily reflected as a component of General and administrative expense in the nine months ended June 30, 2017);\n",
      "\n",
      "----------------------------------- 0001193125-17-059212 -----------------------------------\n",
      "len  :4868\n",
      "\n",
      "emp_header  :False\n",
      "\n",
      "first_emp_head_block  :False\n",
      "\n",
      "para_text  :Exhibit No.     Description         4.27   Instruments defining the Rights of Security Holders, Including Indentures-Except as set forth in Exhibits 4.1 through 4.18 above, the instruments defining the rights of holders of long-term debt securities of Morgan Stanley and its subsidiaries are omitted pursuant to Section (b)(4)(iii) of Item 601 of Regulation S-K. Morgan Stanley hereby agrees to furnish copies of these instruments to the U.S. Securities and Exchange Commission upon request.        10.1   Amended and Restated Trust Agreement dated as of October 18, 2011 by and between Morgan Stanley and State Street Bank and Trust Company (Exhibit 10.1 to Morgan Stanley's Annual Report on Form 10-K for the year ended December 31, 2011).        10.2   Amended and Restated Investor Agreement dated as of June 30, 2011 by and between Morgan Stanley and Mitsubishi UFJ Financial Group, Inc. (Exhibit 10.1 to Morgan Stanley's Current Report on Form 8-K dated June 30, 2011), as amended by Third Amendment, dated October 3, 2013 (Exhibit 10.1 to Morgan Stanley's Quarterly Report on Form 10-Q for the quarter ended September 30, 2013) and Fourth Amendment, dated April 6, 2016 (Exhibit 10.1 to Morgan Stanley's Quarterly Report on Form 10-Q for the quarter ended March 31, 2016).        10.3†   Morgan Stanley 401(k) Plan, amended and restated as of January 1, 2013 (Exhibit 10.6 to Morgan Stanley Annual Report on Form 10-K for the year ended December 31, 2012), as amended by Amendment (Exhibit 10.5 to Morgan Stanley's Annual Report on Form 10-K for the year ended December 31, 2013), Amendment (Exhibit 10.6 to Morgan Stanley's Annual Report on Form 10-K for the year ended December 31, 2013). Amendment (Exhibit 10.5 to Morgan Stanley's Annual Report on Form 10-K for the year ended December 31, 2014) and Amendment (Exhibit 10.5 to Morgan Stanley's Annual Report on Form 10-K for the year ended December 31, 2015).        10.4†*   Amendment to Morgan Stanley 401(k) Plan, dated as of December 13, 2016.        10.5†   Tax Deferred Equity Participation Plan as amended and restated as of November 26, 2007 (Exhibit 10.9 to Morgan Stanley's Annual Report on Form 10-K for the fiscal year ended November 30, 2007).        10.6†   Directors' Equity Capital Accumulation Plan as amended and restated as of August 1, 2016 (Exhibit 10.1 to Morgan Stanley's Quarterly Report on Form 10-Q for the quarter ended June 30, 2016).        10.7†   Employees' Equity Accumulation Plan as amended and restated as of November 26, 2007 (Exhibit 10.12 to Morgan Stanley's Annual Report on Form 10-K for the fiscal year ended November 30, 2007).        10.8†   Employee Stock Purchase Plan as amended and restated as of February 1, 2009 (Exhibit 10.20 to Morgan Stanley's Annual Report on Form 10-K for the fiscal year ended November 30, 2008).        10.9†   Morgan Stanley Supplemental Executive Retirement and Excess Plan, amended and restated effective December 31, 2008 (Exhibit 10.2 to Morgan Stanley's Quarterly Report on Form 10-Q for the quarter ended March 31, 2009) as amended by Amendment (Exhibit 10.5 to Morgan Stanley's Quarterly Report on Form 10-Q for the quarter ended June 30, 2009), Amendment (Exhibit 10.19 to Morgan Stanley's Annual Report on Form 10-K for the year ended December 31, 2010), Amendment (Exhibit 10.3 to Morgan Stanley's Quarterly Report on Form 10-Q for the quarter ended June 30, 2011) and Amendment (Exhibit 10.1 to Morgan Stanley's Quarterly Report on Form 10-Q for the quarter ended September 30, 2014).        10.10†   1995 Equity Incentive Compensation Plan (Annex A to MSG's Proxy Statement for its 1996 Annual Meeting of Stockholders) as amended by Amendment (Exhibit 10.39 to Morgan Stanley's Annual Report on Form 10-K for the fiscal year ended November 30, 2000), Amendment (Exhibit 10.5 to Morgan Stanley's Quarterly Report on Form 10-Q for the quarter ended August 31, 2005), Amendment (Exhibit 10.3 to Morgan Stanley's Quarterly Report on Form 10-Q for the quarter ended February 28, 2006), Amendment (Exhibit 10.24 to Morgan Stanley's Annual Report on Form 10-K for the fiscal year ended November 30, 2006) and Amendment (Exhibit 10.22 to Morgan Stanley's Annual Report on Form 10-K for the fiscal year ended November 30, 2007).        10.11†   Form of Management Committee Equity Award Certificate for Discretionary Retention Award of Stock Units and Stock Options (Exhibit 10.30 to Morgan Stanley's Annual Report on Form 10-K for the fiscal year ended November 30, 2006).        10.12†   Form of Deferred Compensation Agreement under the Pre-Tax Incentive Program 2 (Exhibit 10.12 to MSG's Annual Report for the fiscal year ended November 30, 1996).        10.13†   Key Employee Private Equity Recognition Plan (Exhibit 10.43 to Morgan Stanley's Annual Report on Form 10-K for the fiscal year ended November 30, 2000).\n",
      "\n",
      "----------------------------------- 0001413329-17-000014 -----------------------------------\n",
      "len  :4541\n",
      "\n",
      "emp_header  :False\n",
      "\n",
      "first_emp_head_block  :False\n",
      "\n",
      "para_text  :In the first class action pending in Canada, Cecilia Létourneau v. Imperial Tobacco Ltd., Rothmans, Benson & Hedges Inc. and JTI Macdonald Corp., Quebec Superior Court, Canada, filed in September 1998, our subsidiary and other Canadian manufacturers (Imperial Tobacco Canada Ltd. and JTI-MacDonald Corp.) are defendants.  The plaintiff, an individual smoker, sought compensatory and punitive damages for each member of the class who is deemed addicted to smoking. The class was certified in 2005.  Trial began in March 2012 and concluded in December 2014. The trial court issued its judgment on May 27, 2015. The trial court found our subsidiary and two other Canadian manufacturers liable and awarded a total of CAD 131 million (approximately $100 million) in punitive damages, allocating CAD 46 million (approximately $35 million) to our subsidiary. The trial court found that defendants violated the Civil Code of Quebec, the Quebec Charter of Human Rights and Freedoms, and the Quebec Consumer Protection Act by failing to warn adequately of the dangers of smoking. The trial court also found that defendants conspired to prevent consumers from learning the dangers of smoking. The trial court further held that these civil faults were a cause of the class members' addiction. The trial court rejected other grounds of fault advanced by the class, holding that: (i) the evidence was insufficient to show that defendants marketed to youth, (ii) defendants' advertising did not convey false information about the characteristics of cigarettes, and (iii) defendants did not commit a fault by using the descriptors light or mild for cigarettes with a lower tar delivery. The trial court estimated the size of the addiction class at 918,000 members but declined to award compensatory damages to the addiction class because the evidence did not establish the claims with sufficient accuracy. The trial court ordered defendants to pay the full punitive damage award into a trust within 60 days and found that a claims process to allocate the awarded damages to individual class members would be too expensive and difficult to administer. The trial court ordered a briefing on the proposed process for the distribution of sums remaining from the punitive damage award after payment of attorneys' fees and legal costs. In June 2015, our subsidiary commenced the appellate process by filing its inscription of appeal of the trial court's judgment with the Court of Appeal of Quebec. Our subsidiary also filed a motion to cancel the trial court's order for payment into a trust within 60 days notwithstanding appeal. In July 2015, the Court of Appeal granted the motion to cancel and overturned the trial court's ruling that our subsidiary make the payment into a trust within 60 days. In August 2015, plaintiffs filed a motion with the Court of Appeal seeking security in both the Létourneau case and the Blais case described below. In October 2015, the Court of Appeal granted the motion and ordered our subsidiary to furnish security totaling CAD 226 million (approximately $173 million), in the form of cash into a court trust or letters of credit, in six equal consecutive quarterly installments of approximately CAD 37.6 million (approximately $28.7 million) beginning in December 2015 through March 2017. See the Blais description for further detail concerning the security order. The Court of Appeal heard oral arguments on the merits appeal in November 2016. Our subsidiary and PMI believe that the findings of liability and damages were incorrect and should ultimately be set aside on any one of many grounds, including the following: (i) holding that defendants violated Quebec law by failing to warn class members of the risks of smoking even after the court found that class members knew, or should have known, of the risks, (ii) finding that plaintiffs were not required to prove that defendants' alleged misconduct caused injury to each class member in direct contravention of binding precedent, (iii) creating a factual presumption, without any evidence from class members or otherwise, that defendants' alleged misconduct caused all smoking by all class members, (iv) holding that the addiction class members' claims for punitive damages were not time-barred even though the case was filed more than three years after a prominent addiction warning appeared on all packages, and (v) awarding punitive damages to punish defendants without proper consideration as to whether punitive damages were necessary to deter future misconduct.\n",
      "\n",
      "----------------------------------- 0001144204-17-012180 -----------------------------------\n",
      "len  :4213\n",
      "\n",
      "emp_header  :False\n",
      "\n",
      "first_emp_head_block  :False\n",
      "\n",
      "para_text  :TABLE OF CONTENTS The federal False Claims Act imposes civil penalties, including through civil whistleblower or qui tam actions, against individuals or entities for knowingly presenting, or causing to be presented, to the federal government, claims for payment that are false or fraudulent or making a false statement to avoid, decrease or conceal an obligation to pay money to the federal government. The government and qui tam relators have brought False Claims Act actions against pharmaceutical companies on the theory that their practices have caused false claims to be presented to the government. There is also a separate false claims provision imposing criminal penalties. Other federal healthcare fraud-related laws also provide criminal liability for violations. The Criminal Healthcare Fraud statute (18 U.S.C. §1347) prohibits knowingly and willfully executing a scheme to defraud any healthcare benefit program, including private third-party payers. Federal criminal law at 18 U.S.C. §1001, among other sections, prohibits knowingly and willfully falsifying, concealing or covering up a material fact or making any materially false, fictitious or fraudulent statement in connection with the delivery of or payment for healthcare benefits, items or services. Other Laws The Health Insurance Portability and Accountability Act of 1996, or HIPAA, as amended by the Health Information Technology for Economic and Clinical Health Act, imposes criminal liability for executing a scheme to defraud any healthcare benefit program and also imposes obligations, including mandatory contractual terms, with respect to safeguarding the privacy, security and transmission of individually identifiable health information. HIPAA also imposes criminal liability for knowingly and willfully falsifying, concealing or covering up a material fact or making any materially false statements in connection with the delivery of or payment for healthcare benefits, items or services. The federal Physician Payments Sunshine Act requirements under the ACA, and its implementing regulations, require manufacturers of drugs, devices, biologics and medical supplies to report to the Department of Health and Human Services information related to payments and other transfers of value made to covered recipients, such as physicians and teaching hospitals, and physician ownership and investment interests. Payments made to physicians and research institutions for clinical trials are included within the ambit of this law. A number of states also have statutes or regulations similar to the federal Anti-Kickback Statute and False Claims Act that apply to items and services reimbursed under Medicaid and other state programs. Some state anti-kickback statutes apply not just to government payors, but to all payors, including commercial payors. Employees As of December 31, 2016, we had 456 employees, including 169 in research and development, 137 in our commercial organization, 66 in our medical affairs group and 84 in general and administrative supporting functions. Geographically, 332 of our employees were based in the United States and 124 were based outside the United States. None of our employees are represented by a labor union and we consider our employee relations to be good. Corporate Information We were incorporated in the State of Delaware on September 4, 2002. Our principal executive offices are located at 450 West 15th Street, Suite 505, New York, NY 10011, and our telephone number is (646) 747-1000. We also have administrative offices in San Diego, California and London, United Kingdom. Our corporate website address is www.interceptpharma.com. Our Annual Report on Form 10-K, Quarterly Reports on Form 10-Q, Current Reports on Form 8-K, and amendments to reports filed pursuant to Sections 13(a) and 15(d) of the Securities Exchange Act of 1934, as amended, or the Exchange Act, are available free of charge on our website as soon as reasonably practicable after we electronically file such material with, or furnish it to, the Securities and Exchange Commission. The Securities and Exchange Commission maintains an internet site that contains our public filings with the Securities and Exchange\n",
      "\n",
      "----------------------------------- 0001144204-17-010224 -----------------------------------\n",
      "len  :3972\n",
      "\n",
      "emp_header  :False\n",
      "\n",
      "first_emp_head_block  :False\n",
      "\n",
      "para_text  :TABLE OF CONTENTS Contracted Power - (continued) Seasonality Each CP facility has unique seasonality based on local market factors including weather and energy demand. The solar projects generate a disproportionate amount of their revenue in the summer months when insolation is highest, while wind energy revenues vary by season depending on the location of the specific project. Other CP projects may also exhibit seasonality based on the respective market's varying electricity demand. For example, BEC is a peaking power plant that is dispatched when energy demand increases above a base load, therefore BEC is dispatched more often in the summer and winter months. Competition The contracted portion of CP's business is not subject to substantial direct competitive price pressure due to the long-term nature of the PPAs and tolling agreements with creditworthy off-takers. Our BEC facility has incremental generating capacity which is available for sale in the spot power market. After the expiration of the PPAs or tolling agreements, our facilities could be subject to greater competition. Regulation The power and energy sectors are the subject of substantial and complex laws, rules and regulations. Sector regulators include the Federal Energy Regulatory Commission (FERC) and the North American Electric Reliability Corporation (NERC). The FERC has jurisdiction over the transmission and wholesale sale of electricity in interstate commerce and over the transportation, storage and certain sales of natural gas in interstate commerce, including the rates, charges and other terms and conditions for such services. The NERC serves to establish and enforce reliability standards applicable to all users, owners and operators of the bulk power system. The U.S. Environmental Protection Agency also oversees certain environmental matters related to the construction and operations of CP's electric generating facilities. The New York competitive wholesale electricity market in which BEC participates is administered by the New York Independent System Operator (NYISO). NYISO is a not-for-profit agency that serves to ensure regional grid reliability, comprehensive planning, and open and efficient markets. BEC is also subject to certain New York State and New York City regulations and to environmental regulations by the State of New Jersey's Department of Environmental Protection. Each of the solar and wind facilities comprising CP is subject to regulation and oversight in the jurisdictions in which they operate. Employees and Management At December 31, 2016, the CP businesses had seven employees and two contractors that provide certain management oversight, while O&M is provided by third parties. MIC Hawaii Industry Overview Beginning in 2008 with the Hawaii Clean Energy Initiative, Hawaii implemented a RPS that required increasing levels of electricity be derived from renewable resources with the goal of generating cleaner and lower cost electricity. In 2015, Hawaii accelerated its RPS and currently targets 100% of its electricity generated from renewable resources by 2045 with interim milestones in 2020, 2030 and 2040. MIC Hawaii's businesses support the State's clean energy goals of lowering emissions, increasing energy security and decreasing costs for all energy users in Hawaii. For example, at Hawaii Gas, we are focused on diversifying our feedstock to include fuels which are cleaner burning and have the potential to lower energy costs for our customers, such as renewable natural gas (RNG) and liquefied natural gas (LNG). Further, natural gas-fired generation better complements Hawaii's growing renewable power generation sources given its efficiency and operating flexibility. At Waihonu Solar, we are generating renewable energy that serves customers on Oahu. At Critchfield Pacific, Inc. (CPI), we work with customers to lower their overall energy use through the design and construction of energy efficient infrastructure.\n",
      "\n",
      "----------------------------------- 0001144204-17-021318 -----------------------------------\n",
      "len  :3904\n",
      "\n",
      "emp_header  :False\n",
      "\n",
      "first_emp_head_block  :False\n",
      "\n",
      "para_text  :Members Name (age), position  Member since  Audit Committee  Personnel Committee  Related Parties Committee  Nomination and Corporate Governance Committee  Risk and Capital Management Committee  Strategy Committee  Compensation Committee(1)  International Advisory Council(2)   Board of Directors (3) (12 members)                         Pedro Moreira Salles (57), Chairman  08/2009                     Alfredo Egydio Arruda Villela Filho (47), Vice Chairman  03/2003                     Roberto Egydio Setubal (62), Vice Chairman  03/2003                     Alfredo Egydio Setubal (58), Member  06/2007                     Candido Botelho Bracher (58), Member  02/2009                     Demosthenes Madureira de Pinho Neto (56), Member  05/2012                     Fábio Colletti Barbosa (62), Independent Member  07/2015                     Gustavo Jorge Laboissière Loyola (64), Independent Member  07/2006                     José Gall  (65), Independent Member  04/2016                     Nildemar Secches (68), Independent Member  05/2012                     Pedro Luiz Bodin de Moraes (60), Independent Member  02/2009                     Ricardo Villela Marino (42), Member  06/2008                                          Board of Officers (24 members)                         Roberto Egydio Setubal (62), Chief Executive Officer  11/1995                     Candido Botelho Bracher (58), General Manager  08/2005                     Márcio de Andrade Schettini (52), General Manager  07/2015                     Marco Ambrogio Crespi Bonomi (60), General Manager  07/2015                     Eduardo Mazzilli de Vassimon (58), General Manager  03/2013                     André Sapoznik (44), Vice President  12/2016                     Caio Ibrahim David (48), Chief Financial Officer and Vice President  12/2016                     Claudia Politanski (46), Vice President  11/2008                     Alexsandro Broedel Lopes (42), Executive Officer  08/2012                     Fernando Barçante Tostes Malta (48), Executive Officer  04/2016                     Leila Cristiane Barboza Braga de Melo (45), Executive Officer  04/2015                     Paulo Sergio Miron (50), Executive Officer  07/2015                     Adriano Cabral Volpini (44), Officer  02/2015                     Álvaro Felipe Rizzi Rodrigues (39), Officer  04/2015                     Atilio Luiz Magila Albiero Junior (39), Officer  04/2016                     Eduardo Hiroyuki Miyaki (44), Officer  08/2011                     Emerson Macedo Bortoloto (39), Officer  11/2011                     Gilberto Frussa (50), Officer  04/2016                     José Virgilio Vita Neto (38), Officer  04/2015                     Marcelo Kopel (52), Officer and Investor Relations Officer  06/2014                     Matias Granata (42), Officer  07/2014                     Rodrigo Luis Rosa Couto (41), Officer  01/2012                     Sergio Mychkis Goldstein (39), Officer  04/2016                     Wagner Bettini Sanches (45), Officer  06/2014                                          Audit Committee(3) (5 members)                         Antonio Francisco de Lima Neto (51), Independent Member  07/2015                     Diego Fresco Gutierrez (46), Independent Member and Financial Expert  04/2014                     Geraldo Travaglia Filho (65), Independent Member  03/2013                     Maria Helena dos Santos Fernandes de Santana (57), Independent Member  06/2014                     Rogério Paulo Calder n Peres (54), Independent Member  11/2016                                          Fiscal Council(3) (3 members)                         Alkimar Ribeiro Moura (75), Independent Member  04/2016                     Carlos Roberto de Albuquerque Sá (66), Independent Member  04/2016                     José Caruso Cruz Henriques (69), Independent Member  04/2016\n",
      "\n",
      "----------------------------------- 0001144204-17-010224 -----------------------------------\n",
      "len  :3878\n",
      "\n",
      "emp_header  :False\n",
      "\n",
      "first_emp_head_block  :False\n",
      "\n",
      "para_text  :TABLE OF CONTENTS IMTT - (continued) Regulation The rates that IMTT charges for its services are not subject to regulation. However, a number of regulatory bodies oversee IMTT's operations. IMTT must comply with numerous federal, state and local environmental, occupational health and safety, security, tax and planning statutes and regulations. These regulations require IMTT to obtain and maintain permits to operate its facilities and impose standards that govern the way IMTT operates its business. If IMTT does not comply with the relevant regulations, it could lose its operating permits and/or incur fines and increased liability. As a result, IMTT has developed environmental and health and safety compliance functions overseen by terminal managers at the terminal level, as well as by IMTT's Vice President of Environmental, Health, Safety and Security, Chief Operating Officer and Chief Executive Officer. While changes in environmental, health and safety regulations pose a risk of higher operating costs, such changes are generally phased in over time to manage the impact on both the industry and the business. The Bayonne terminal has significant environmental remediation requirements that were partially assumed at the time of purchase from the various former owners. One former owner retained environmental remediation responsibilities for a purchased site as well as responsibility for sharing other remediation costs. Remediation efforts entail removal of the free product, groundwater control and treatment, soil treatment, repair/replacement of sewer systems, and the implementation of containment and monitoring systems. These remediation activities are expected to continue for an additional ten to twenty years. See \"Legal Proceedings\" in Part I, Item 3, for further discussions. The Lemont terminal has entered into a consent order with the State of Illinois to remediate contamination at the site that pre-dated IMTT's ownership. This remediation effort, including the implementation of extraction and monitoring wells and soil treatment, is estimated to continue for an additional ten to twenty years. Employees and Management As of December 31, 2016, IMTT (excluding the Newfoundland terminal) had a total of 1,052 employees, of which 230 employees were unionized. In January 2017, IMTT agreed to a new, five year collective bargaining agreement with its Bayonne employees represented by the United Steel Workers. This agreement is expected to be executed within the quarter ended March 31, 2017. We believe employee relations at IMTT are good. The day-to-day operations of IMTT are managed by individual terminal managers who are responsible for most aspects of the operations at their terminals. IMTT's operations are overseen by senior personnel with significant experience in the bulk liquid storage industry. Management of the business is headquartered in New Orleans, Louisiana. Atlantic Aviation Industry Overview Fixed based operations (FBOs) primarily service the GA corporate and leisure flying segment of the air transportation industry. Local airport authorities own the airport properties and grant FBO operators the right to provide fueling and other services pursuant to long-term ground leases. Fueling services provide the majority of an FBO's revenue and gross margin. FBOs often operate in environments with high barriers to entry. Airports tend to have limited physical space for additional FBOs. Airport authorities generally do not have an incentive to add additional FBOs unless there is a significant demand for additional services. Government approvals and design and construction of a new FBO can also take significant time and require significant capital expenditures. Furthermore, airports typically impose minimum standards with respect to the experience, capital investment and breadth of services provided by the FBO.\n",
      "\n",
      "----------------------------------- 0000930413-17-000981 -----------------------------------\n",
      "len  :3862\n",
      "\n",
      "emp_header  :False\n",
      "\n",
      "first_emp_head_block  :False\n",
      "\n",
      "para_text  :Form 20-F Item Number and Caption Location Page   Part I      Item 1. Identity of Directors, Senior Management and Advisers      A. Directors and senior management Not applicable.     B. Advisers Not applicable.     C. Auditors Not applicable.    Item 2. Offer Statistics and Expected Timetable      A. Offer statistics Not applicable.     B. Method and expected timetable Not applicable.    Item 3. Key Information      A. Selected financial data \"Selected consolidated financial data\" 3     \"Exchange rates\" 4    B. Capitalisation and indebtedness Not applicable.     C. Reason for the offer and use of proceeds Not applicable.     D. Risk factors \"Risk factors\" 197-214   Item 4. Information on the Company      A. History and development of the company \"Business overview\" 2     \"Business - History and development of Lloyds Banking Group\" 4     \"Business - Legal actions and regulatory matters\" 6-9     \"Operating and financial review and prospects - Line of business information\" 24-27    B. Business overview \"Business overview\" 2     \"Business - Legal actions and regulatory matters\" 6-9     \"Operating and financial review and prospects - Line of business information\" 24-27     \"Regulation\" 178-181    C. Organisational structure \"Lloyds Banking Group structure\" 216    D. Property, plant and equipment \"Business - Properties\" 6   Item 4A. Unresolved Staff Comments Not applicable.    Item 5. Operating and Financial Review and Prospects      A. Operating results \"Operating and financial review and prospects\" 10-116     \"Operating and financial review and prospects - Credit risk\" 53-85     \"Regulation\" 178-181     \"Operating and financial review and prospects - Market Risk\" 87-92    B. Liquidity and capital resources \"Operating and financial review and prospects - Risk elements in the loan portfolio - Cross border outstandings\" 85     \"Operating and financial review and prospects - Funding and Liquidity Risk\" 95-100     \"Operating and financial review and prospects - Capital risk\" 101-109     \"Operating and financial review and prospects - Investment portfolio, maturities, deposits, short-term borrowings\" 113-116     \"Dividends\" 185     \"Notes to the consolidated financial statements - note 48\" F-70    C. Research and development, patents and licenses, etc. Not applicable.     D. Trend information \"Operating and financial review and prospects - Overview and trend information\" 11-12    E. Off-balance sheet arrangements \"Operating and financial review and prospects - Funding and liquidity risk - Off balance sheet arrangements\" - also refer to financial notes 100    F. Tabular disclosure of contractual obligations \"Operating and financial review and prospects - Funding and liquidity risk - Contractual cash obligations\" 100    G. Safe harbor \"Forward looking statements\" 215   Item 6. Directors, Senior Management and Employees      A. Directors and senior management \"Management and employees - Directors and senior management\" 117-120    B. Compensation \"Compensation\" - also refer to financial notes 120-151    C. Board practices \"Management and employees\" 117-120     \"Articles of association of Lloyds Banking Group plc\" 190     \"Management and employees - Employees\" 120     \"Compensation - Service agreements\" 131     \"Corporate governance - Leadership\" 153     \"Corporate governance - the Board in 2016\" 154-161     \"Corporate governance - Audit Committee Report\" 167-170     \"Compensation - Annual report on remuneration - Remuneration Committee\" 150-151    D. Employees \"Management and employees - Employees\" 120    E. Share ownership \"Compensation - Outstanding share awards\" 145-149     \"Notes to the consolidated financial statements - note 2\" F-15   Item 7. Major Shareholders and Related Party Transactions      A. Major shareholders \"Major shareholders and related party transactions - Major shareholders\" 177\n",
      "\n",
      "----------------------------------- 0001144204-17-010224 -----------------------------------\n",
      "len  :3795\n",
      "\n",
      "emp_header  :False\n",
      "\n",
      "first_emp_head_block  :False\n",
      "\n",
      "para_text  :TABLE OF CONTENTS Atlantic Aviation - (continued) Competition Atlantic Aviation directly competes with other FBO operators at more than half of its locations. The FBOs compete on the basis of location of the facility relative to runways and street access, service, safety, value-added features, reliability and price. Each FBO also faces competitive pressure from the fact that aircraft may take on sufficient fuel at one location and not need to refuel at a specific destination. FBO operators also face indirect competition from facilities located at nearby airports. Atlantic Aviation's main competitors are Signature Flight Support, Jet Aviation, Million Air, Sheltair Aviation and TAC Air. In February 2016, Signature Flight Support acquired Landmark Aviation, resulting in the combination of two of Atlantic Aviation's biggest competitors. Signature Flight Support competes with Atlantic at 22 airports, but only three of these locations were added as a result of the merger. Atlantic Aviation has not noticed a significant shift in customer or competitor behavior and remains confident in its ability to compete effectively. To our knowledge, other than the competitors listed, no other competitor operated more than 20 FBOs in the U.S. at December 31, 2016. Customers Atlantic Aviation does not depend on a single customer, the loss of which would have a material adverse effect on the business. Regulation The aviation industry is overseen by a number of regulatory bodies, but its primary regulator is the FAA. In addition, local airport authorities also regulate the FBOs. The business must comply with federal, state and local environmental statutes and regulations associated in part with the operation of fuel storage tank systems and mobile fueling vehicles. These requirements include, among other things, tank and pipe testing for tightness, soil sampling for evidence of leaking and remediation of detected leaks and spills. Atlantic Aviation's FBOs are subject to regular inspection by federal and local environmental agencies as well as local fire departments and other agencies. The business does not expect that compliance and related remediation work, if any, will have a material negative impact on earnings or the competitive position of Atlantic Aviation. The business has not received notice requiring it to cease operations at any location or of any abatement proceeding by any government agency as a result of failure to comply with applicable environmental laws and regulations. Employees and Management As of December 31, 2016, the business employed 1,956 people, of which 190 employees were subject to collective bargaining agreements. We believe relations with union and non-union employees at Atlantic Aviation are good. The day-to-day operations of Atlantic Aviation are managed by individual site managers who are responsible for most aspects of the operations at their site. Atlantic Aviation's operations are overseen by senior personnel with significant experience in the aviation industry. Contracted Power Industry Overview The power industry represents a large and critical infrastructure market, both in terms of the number and value of facilities as well as their contribution to overall economic activity. In developed economies, capital spending in the sector is driven by aging infrastructure, new technologies, increased legislation regarding emissions, the use of renewable energy and modest demand growth. The Energy Information Administration forecasts the demand for electricity in the U.S. to grow at a compound annual rate of approximately 0.7% over the next twenty years. As aging and inefficient generating capacity is retired or replaced, opportunities for deployment of capital in the growth of our CP segment are expected to increase.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print_row_detail(df=train_df, nrow=10, header_list = ['acc_id'  ],\n",
    "                    detail_list = ['len', 'emp_header', 'first_emp_head_block', 'para_text'],\n",
    "                    sortby=['len', 'acc_id'], ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-30T16:58:53.330069Z",
     "start_time": "2018-06-30T16:58:49.617195Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index number: \n",
      "734\n",
      "File path: \n",
      "c:/projects/DSBC/capstone/sec_employee_information_extraction/../employee_filings/0001047469-16-014916.html\n",
      "No Employees header\n",
      "Para number: 0\n",
      "Original match has no name\n",
      "Original match:  We currently have 21 employees, 18 of whom are located in Denver, Colorado, one who is located\n",
      "in Zug, Switzerland, and two who are located in Toronto, Canada. Our employees are not subject to a labor\n",
      "contract or a collective bargaining agreement. We consider our employee relations to be good.\n",
      "Block match name:  p\n",
      "Block:  \n",
      "<p class=\"c15\">We currently have 21 employees, 18 of whom are located in Denver, Colorado, one who is located\n",
      "in Zug, Switzerland, and two who are located in Toronto, Canada. Our employees are not subject to a labor\n",
      "contract or a collective bargaining agreement. We consider our employee relations to be good.</p>\n",
      "<class 'bs4.element.Tag'>\n",
      "Block Text:  \n",
      "We currently have 21 employees, 18 of whom are located in Denver, Colorado, one who is located in Zug, Switzerland, and two who are located in Toronto, Canada. Our employees are not subject to a labor contract or a collective bargaining agreement. We consider our employee relations to be good.\n"
     ]
    }
   ],
   "source": [
    "print_parse_objects(file='0001047469-16-014916')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}