demo/BulkDataDemo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example bulk data client interaction\n",
    "\n",
    "Implements a simple [FHIR bulk data client](http://wiki.hl7.org/index.php?title=201801_Bulk_Data) that invokes the `$export` operation and converts the resulting data stream to a generator.\n",
    "\n",
    "### Config\n",
    "\n",
    "Start by reading required config parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "with open('config.yaml') as f:\n",
    "    config = yaml.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup\n",
    "\n",
    "Construct the client. First we request the remote server to prepare a dataset using ``client.provision()``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check for successful provisioning of the list of data files using a property of the client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fhir.client import BulkDataClient\n",
    "\n",
    "client = BulkDataClient(**config)\n",
    "client.provision(_type='Patient')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.provisioned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://s3.us-south.cloud-object-storage.appdomain.cloud/fhir-r4-connectathon/job74/Patient_0.ndjson']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.manifest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Consume data\n",
    "\n",
    "Once the dataset is provisioned we can iterate through the resources in the bulk data set. The client leaves most data  on the remote server and waits to retrieve subsequent files until the previous file in the manifest is consumed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'resourceType': 'Patient',\n",
       " 'id': 'db049c87-0b30-447f-b995-3d700b61f36d',\n",
       " 'meta': {'versionId': '1', 'lastUpdated': '2019-09-13T19:26:19.6Z'},\n",
       " 'active': True,\n",
       " 'name': [{'family': 'Ortiz', 'given': ['David']}],\n",
       " 'gender': 'male'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = client.iter_json()\n",
    "next(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleanup\n",
    "\n",
    "We should close the connection in the underlying session prior to releasing the client. Alternatively the client also functions as a context manager for simplicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['https://s3.us-south.cloud-object-storage.appdomain.cloud/fhir-r4-connectathon/job75/Patient_0.ndjson']\n"
     ]
    }
   ],
   "source": [
    "client.session.close()\n",
    "\n",
    "with BulkDataClient(**config) as newclient:\n",
    "    newclient.provision(_type='Patient')\n",
    "    print(newclient.manifest)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}