Leçon 2 - La préparation de données par programmation avec Python et Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# _Leçon 2_: La préparation de données par programmation avec Python et Pandas\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <b>A la fin de la formation, vous pourrez:</b>\n",
    "    <br>\n",
    "    <ul>\n",
    "        <li> Comprendre et avoir une expérience pratique de la structure et de la conception des fichiers de données exploitables par une machine.</li>\n",
    "        <li>Intégrer et appliquer les méthodes des bibliothèques d'analyse de données de base de Numpy, Pandas et Matplotlib pour étudier et manipuler les données sources.</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Introduction\n",
    "\n",
    "L'expression \"données ouvertes\" s'entend généralement des données qui sont mises à la disposition du public gratuitement, sans enregistrement ni licence restrictive, à quelque fin que ce soit (y compris à des fins commerciales), dans des formats électroniques exploitables par une machine qui garantissent que les données sont faciles à trouver, à télécharger et à utiliser. \n",
    " \n",
    "Les initiatives en matière de données ouvertes prises par les institutions publiques, telles que les gouvernements et les organisations intergouvernementales, reconnaissent que ces données sont produites avec des fonds publics et doivent donc, à quelques exceptions près, être traitées comme des biens publics.\n",
    " \n",
    "La réutilisation des données, tant par les experts en données que par le grand public, est essentielle pour créer de nouvelles opportunités et de nouvelles connaissances à partir des données gouvernementales. La réutilisation des données ouvertes requiert deux critères de base :\n",
    " \n",
    "1. __Les données doivent être légalement ouvertes__, ce qui signifie qu'elles sont placées dans le domaine public ou dans des conditions d'utilisation ouverte avec un minimum de restrictions. Cela permet de garantir que les politiques gouvernementales ne créent pas d'obstacles ou d'ambiguïtés quant à la manière dont les données peuvent être utilisées.\n",
    "2. __Les données doivent être techniquement ouvertes__, ce qui signifie qu'elles sont publiées dans des formats électroniques exploitables par une machine et non propriétaires. Cela garantit que les citoyens ordinaires peuvent accéder aux données et les utiliser à peu de frais, voire gratuitement, en utilisant des outils logiciels courants.\n",
    "\n",
    "L'objectif de ce programme de formation en _Analyse et validation de données ouvertes_ est de guider les participants et les aider à fournir des données techniquement ouvertes : des données bien structurées, exploitables par une machine, validées selon un schéma de métadonnées défini et standard.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 Le cycle de vie de la gestion des données\n",
    "\n",
    "Si le processus de création, de maintenance et d'exploitation de nouvelles données est souvent présenté comme un cycle, il s'agit plutôt d'une spirale. Chaque cycle monte en spirale, et donne lieu à plus d'informations. Cependant, l'efficacité de chaque étape est définie par les besoins de ses utilisateurs, et sa pertinence par rapport au processus ou aux événements qu'elle reflète.\n",
    "\n",
    "![Cycle de vie des données](images/data-lifecycle-fr.jpg)\n",
    "\n",
    "### 2.2.3 Collecte et création\n",
    "\n",
    "Avant de pouvoir collecter des données, il faut connaître toute une série de choses :\n",
    "\n",
    "- pourquoi collectons-nous les données ?\n",
    "- à quoi servent-elles ?\n",
    "- avons-nous le consentement et/ou l'autorisation légale de collecter ces données ?\n",
    "- existe-t-il une série de données existantes que ces données complètent ou élargissent ?\n",
    "- comment les données seront-elles collectées ?\n",
    "- qui sera responsable de la qualité des données et comment cette qualité sera-t-elle mesurée ?\n",
    "- qui aura accès aux données et quel est le degré de sensibilité de ces données (par exemple, identification personnelle) ?\n",
    "- utilisons-nous une classification ou un format de métadonnées standardisé et convenu ?\n",
    "\n",
    "### 2.2.4 Classification et traitement\n",
    "\n",
    "Le créateur des données est le mieux placé pour savoir de quoi il s'agit et devrait attribuer des mots-clés comme descripteurs. Ces données sur les données sont appelées métadonnées. Le terme est ambigu, car il est utilisé pour deux concepts fondamentalement différents :\n",
    " \n",
    "- __Les métadonnées structurelles__ correspondent aux métadonnées internes (c'est-à-dire les métadonnées concernant la structure des objets de la base de données tels que les tables, les colonnes, les clés et les index) ;\n",
    "- __Les métadonnées descriptives__, qui correspondent à des métadonnées externes. (c'est-à-dire des métadonnées typiquement utilisées pour la découverte et l'identification, comme des informations utilisées pour rechercher et localiser un objet tel que le titre, l'auteur, les sujets, les mots clés, l'éditeur) ;\n",
    " \n",
    "Les métadonnées descriptives permettent la découverte de l'objet. Les métadonnées structurelles permettent d'appliquer, d'interpréter, d'analyser, de restructurer les données et de les relier à d'autres ensembles de données similaires.\n",
    " \n",
    "Les métadonnées peuvent permettre l'interopérabilité entre différents systèmes. Une structure convenue pour interroger la \"fiabilité\" d'une série de données peut permettre à des systèmes logiciels indépendants de trouver et d'utiliser des données à distance.\n",
    " \n",
    "Au-delà des métadonnées, il existe également des mécanismes permettant de structurer les relations entre les hiérarchies de mots clés. Ces mécanismes sont connus sous le nom d'ontologies et, avec les métadonnées, peuvent être utilisés pour définir avec précision et permettre la découverte de données.\n",
    " \n",
    "L'ajout de métadonnées aux ressources de données existantes peut être un processus coûteux et exigeant en main-d'œuvre. Cela peut devenir un obstacle à la mise en œuvre d'un système complet de gestion des connaissances.\n",
    "\n",
    "Les types de métadonnées descriptives que nous devrions inclure (et les termes que nous devrions utiliser pour les désigner) :\n",
    "\n",
    "| OBLIGATOIRE              | RECOMMANDÉ               | FACULTATIF               |\n",
    "|:-------------------------|:-------------------------|:-------------------------|\n",
    "| Titre                    | Tag(s)                   | Dernière mise à jour     |\n",
    "| Description              | Conditions d'utilisation | Fréquence de mise à jour |\n",
    "| Thème(s)                 | Courriel de contact      | Couverture géographique  |\n",
    "| Organisme de publication |                          | Couverture temporelle    |\n",
    "|                          |                          | Validité                 |\n",
    "|                          |                          | Ressources connexes      |\n",
    "|                          |                          | Règlements               |\n",
    "\n",
    "### 2.2.5 Manipulation, conversion ou altération\n",
    "\n",
    "Cette partie du processus est celle où les données sont transcrites, traduites, vérifiées, validées, nettoyées et gérées.\n",
    "\n",
    "C'est ce qui présente le plus grand risque pour la cohérence des données. Tout changement de format ou toute manipulation, ou même la copie d'un fichier d'un système à un autre, introduit un risque de corruption des données. De même, cela augmente également le risque que des données - qu'elles soient erronées ou non - soient accidentellement communiquées aux utilisateurs ou au public avant qu'elles ne soient prêtes.\n",
    "\n",
    "C'est ce que l'on appelle le __data wrangling__ ou préparation des données (il n'y a pas de traduction parfaite pour ce terme et vous le retrouverez souvent sous forme anglaise).\n",
    "\n",
    "### 2.2.6 Analyse et présentation\n",
    "\n",
    "L'analyse ne fait peut-être pas toujours partie du rôle du gestionnaire des données, mais c'est certainement la raison pour laquelle les données sont collectées.\n",
    "\n",
    "C'est là que les données sont interprétées, combinées à d'autres ensembles de données pour produire une méta-analyse, et que l'analyse devient l'histoire que vous souhaitez raconter à partir des données.\n",
    "\n",
    "La qualité de l'histoire dépend de celle de vos données et la recherche n'est considérée comme valable que si les données qui l'alimentent sont également publiées.\n",
    "\n",
    "L'objectif de l'analyse est d'informer le comportement collectif ou individuel, d'influencer la politique ou de soutenir l'activité économique, parmi beaucoup d'autres. La confiance peut être obtenue en \"montrant votre fonctionnement\" (c'est à dire comment vous avez analysé les données), ce qui inclut la publication des données.\n",
    "\n",
    "### 2.2.7 Préservation et stockage\n",
    "\n",
    "Les données doivent être préservées de la corruption et être disponibles pour une utilisation ultérieure une fois l'analyse initiale terminée. Il est également nécessaire de préserver les données en cas de questions concernant l'analyse.\n",
    "\n",
    "Le stockage à long terme exige que les __métadonnées__ soient bien définies et soient vraiment utiles pour garantir que la compréhension de ce que les données décrivent est encore possible longtemps après leur collecte initiale.\n",
    "\n",
    "Les données peuvent finir par être stockées dans plusieurs formats ou sur plusieurs systèmes. Il est essentiel que la primauté soit établie (c'est-à-dire quel ensemble de données est la version originale et prime sur les autres) et que les différents formats soient maintenus en alignement.\n",
    "\n",
    "### 2.2.8 Publication et accès\n",
    "\n",
    "Même lorsque les données ne sont diffusées qu'au sein du gouvernement ou d'une organisation - et non pour le public - il y aura toujours d'autres personnes qui voudront utiliser vos données, ou qui en tireront profit si elles savent qu'elles existent. La plus grande inefficacité dans la gestion des données survient lorsque la recherche est répétée parce qu'un autre ministère, un autre département, une autre équipe de l'organisation a besoin des mêmes données sans savoir qu'elles existent déjà.\n",
    "\n",
    "La diffusion ne consiste pas simplement à mettre les données à la disposition du public, mais aussi à créer un processus prévisible pour cette diffusion.\n",
    "\n",
    "Les données régulièrement collectées (comme les taux d'inflation) ont besoin d'un cycle de diffusion prévisible, car de nombreuses entreprises fondent leurs décisions d'investissement sur la disponibilité de ces informations.\n",
    "\n",
    "Publier un calendrier de diffusion pour vos consomateurs de données (et s'y tenir) leur permet de planifier leur propre analyse, ou de réagir à votre analyse.\n",
    "\n",
    "L'accès implique que vous ayez besoin d'une base de données centralisée qui soit accessible à vos utilisateur. Dans le cas de données ouvertes, comment les données seront-elles déplacées des serveurs internes vers un dépôt public ?\n",
    "\n",
    "La responsabilité de ce processus doit être attribuée et mesurée.\n",
    "\n",
    "### 2.2.9 Conservation et réutilisation\n",
    "\n",
    "Une fois les données publiées, la question se pose de savoir pendant combien de temps elles seront disponibles. Les données de recherche devraient, en règle générale, être disponibles à perpétuité.\n",
    "\n",
    "Les données chronologiques sont d'autant plus utiles qu'elles ont été collectées longtemps. La suppression soudaine de données peut provoquer d'énormes perturbations.\n",
    "\n",
    "Si des systèmes et un soutien appropriés n'ont pas été mis en place, la conservation peut devenir un problème très coûteux.\n",
    "\n",
    "Il est important, pour favoriser la réutilisation, de définir clairement les droits d'auteur et les licences qui permettent de réutiliser librement les données à n'importe quelle fin.\n",
    "\n",
    "### 2.2.10 Archivage\n",
    "\n",
    "Les ensembles de données peuvent devenir très volumineux et ne peuvent être consultés que rarement. Cela peut devenir problématique pour le stockage à long terme.\n",
    "\n",
    "Un processus d'archivage - où les données peuvent être stockées à moindre coût tout en restant accessibles - peut devoir être envisagé.\n",
    "\n",
    "<div class=\"alert alert-block alert-danger\">\n",
    "    <b>Actions:</b>\n",
    "    <br>\n",
    "    <ul>\n",
    "        <li>S'assurer qu'il y a un propriétaire des données qui est responsable du cycle de vie des données, y compris de leur publication et de la réponse aux commentaires ou aux questions;</li>\n",
    "        <li>Préparer un plan de collecte de données, en veillant à ce que les définitions et la structure des données soient conformes aux normes internationales;</li>\n",
    "        <li>S'assurer que les métadonnées sont convenues avec les parties prenantes et qu'elles sont utiles et normalisées;</li>\n",
    "        <li>S'assurer que les données font l'objet d'une licence appropriée pour garantir leur diffusion et leur réutilisation;</li>\n",
    "    </ul>\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Références:</b>\n",
    "    <br>\n",
    "    <ul>\n",
    "        <li><a href=\"http://www.bu.edu/datamanagement/background/data-life-cycle/\">Cycle de vie des données de l'université de Boston</a></li>\n",
    "        <li><a href=\"http://searchsecurity.techtarget.com/magazineContent/Data-Lifecycle-Management-Model-Shows-Risks-and-Integrated-Data-Flow\">Tech Target Data Lifecycle Management</a></li>\n",
    "        <li><a href=\"http://en.wikipedia.org/wiki/Dublin_Core\">Norme de métadonnées de Dublin Core</a></li>\n",
    "        <li><a href=\"http://www.w3.org/TR/vocab-dcat/\">Norme de métadonnées DCAT</a></li>\n",
    "        <li><a href=\"http://en.wikipedia.org/wiki/Seven_Basic_Tools_of_Quality\">Sept outils de base de qualité</a></li>\n",
    "    </ul>\n",
    "</div>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Préparation de données et préparation de la publication\n",
    "\n",
    "Il existe un certain nombre de formats communs pour la distribution des données. Certains sont considérés comme \"ouverts\" (tels que CSV, XML, texte et autres) et d'autres comme propriétaires (SAS, STATA, SPSS, etc.). XLS et XLSX, associés à Microsoft Excel, sont des formats relativement ouverts et un certain nombre de systèmes logiciels peuvent interpréter les données.\n",
    " \n",
    "Les formats propriétaires sont légitimes, même dans le cadre d'une initiative de données ouvertes, car ce sont les systèmes logiciels utilisés par de nombreux utilisateurs professionnels de données. Cependant, comme ces formats ne sont souvent pas interopérables, le potentiel de réutilisation des données est limité, à moins que les formats ouverts ne soient également pris en charge. La diffusion de données dans des formats propriétaires n'exclut pas la diffusion dans des formats ouverts et vice versa.\n",
    "\n",
    "Les tableurs et les systèmes de données distribués ne disposent souvent pas d'une structure de données convenue. Un chercheur qui souhaite combiner ces données avec d'autres données doit d'abord les normaliser, puis décider de termes normalisés pour définir les colonnes et les types de données dans ces colonnes.\n",
    "\n",
    "![Un tableau de données structuré \"lisible par l'homme\" typique](images/01-01-human-readable-data.jpg)\n",
    "\n",
    "La conversion de données tabulaires semi-structurées en un format typique exploitable par une machine donne le fichier CSV (comma-separated-value en français données séparées par des virgules). Ce sont des fichiers tabulaires avec une ligne d'en-tête qui définit chacune des données dans les colonnes et lignes ci-dessous.\n",
    "\n",
    "![Un tableau de données structuré en CSV](images/01-01-machine-readable-data.jpg)\n",
    "\n",
    "Sans tenir compte du respect d'autres normes, les fichiers CSV peuvent être organisés de telle sorte qu'ils soient \"joints\" sur une colonne commune. Par exemple, un ensemble de coordonnées géospatiales peut être utilisé pour relier plusieurs fichiers similaires couvrant différentes séries de données.\n",
    "\n",
    "### 2.3.1 Considérations relatives aux feuilles de calcul exploitables par une machine\n",
    "\n",
    "Data.gov dispose d'un [Abécédaire de la lisibilité des documents et données en ligne](https://www.data.gov/developers/blog/primer-machine-readability-online-documents-and-data) et un fil de discussion sur Data.gov.uk propose les conseils suivants :\n",
    "\n",
    "- Un demandeur peut ouvrir le fichier de données dans des progiciels gratuits et largement accessibles - cela signifie que des formats tels que CSV doivent être préférés ou proposés en plus de formats tels qu'Excel, et que les formats propriétaires qui ne peuvent être ouverts qu'avec des logiciels commerciaux ou spécialisés doivent être évités. \n",
    "- Il est possible de traiter les données directement, en y effectuant toutes les opérations appropriées telles que le tri des colonnes, le filtrage des lignes, l'exécution d'agrégats de valeurs - cela nécessite des données bien structurées. Dans la mesure du possible, la signification des données ne doit pas être contenue dans la présentation. \n",
    "- Les éléments communs de l'ensemble de données sont exprimés de manière uniforme - par exemple, les dates sont toujours dans le même format, les codes ou les noms sont toujours dans le même cas, et les nombres sont exprimés de manière cohérente (par exemple 1,000 ou 1000 mais pas un mélange des deux). \n",
    "- La signification des champs et des valeurs est clairement documentée - soit par une désignation claire des champs, soit par des descriptions accompagnant les données.  \n",
    "\n",
    "En outre, dans la mesure du possible, l'exploitabilité par une machine est améliorée si :\n",
    "\n",
    "- L'ensemble de données utilise des normes communes lorsqu'elles existent - y compris des identificateurs et des noms de champs standard. Il peut s'agir de normes telles que le vocabulaire des dépenses publiques développé pour le gouvernement, ou de normes tierces telles que KML pour l'indication des \"points d'intérêt\".\n",
    "\n",
    "Il est également important de noter qu'il peut être possible de fournir des données dans divers formats exploitables par une machine et, dans la mesure du possible, le fournisseur de données doit échanger avec les réutilisateurs pour identifier le meilleur format. Par exemple, les points d'intérêt pourraient être fournis sous forme de feuille de calcul CSV ou KML. L'idéal serait que les deux soient fournis : toutefois, selon le contexte et le réutilisateur, l'un peut être plus approprié que l'autre. \n",
    "  \n",
    "### 2.3.2 Données _longues_ et données _larges_\n",
    "\n",
    "Toute série de données est constituée de valeurs numériques (généralement) décrites par des termes de métadonnées normalisés (temps, zone, description spécifique, etc.). Il y a deux façons principales de présenter ces données lisibles par machine, qui peuvent être résumées en _largeur_ ou _longueur_. Vous devez faire un choix délibéré quant au format que vous choisirez, et chacun a ses propres forces et faiblesses :\n",
    "\n",
    "- __Les données larges__ présentent des données numériques en plusieurs colonnes. Soit sous forme de catégories (par exemple, chaque pays est présenté dans sa propre colonne), soit par date (par exemple, chaque mise à jour annuelle donne lieu à une nouvelle colonne). Les nouvelles données traversent l'écran de gauche à droite :\n",
    "\n",
    "![Données larges](images/01-01-wide-data.jpg)\n",
    "\n",
    "__Les données larges__ sont souvent utilisés pour la visualisation et le traitement des données, car les données peuvent facilement être regroupées dans les axes nécessaires aux bibliothèques de cartes. Cependant, il s'agit d'un format d'archivage difficile car la mise à jour d'une telle série de données nécessite l'équivalent de la création d'un nouveau champ (le _année_ dans les champs ci-dessus) et ensuite la mise à jour de chaque ligne avec les informations appropriées. Cela peut être une opération coûteuse dans une grande base de données, et signifie également que l'écriture d'une méthode programmatique pour interroger vos données est plus difficile.\n",
    "\n",
    "- __Les données longues__ présente les données numériques sur plusieurs lignes avec une seule colonne pour les valeurs. Les nouvelles données descendent l'écran de haut en bas :\n",
    "\n",
    "![Données longues](images/01-01-narrow-data.jpg)\n",
    "\n",
    "__Les données longues__ sont les meilleures pour l'archivage et pour représenter la structure que vous trouverez habituellement dans une base de données. Chaque ligne d'une _longue_ série de données représente une ligne dans une base de données. L'ajout de nouvelles informations est relativement simple puisque vous ne devez mettre à jour qu'une seule ligne à la fois. En termes de base de données, vous créez une seule entrée de base de données.\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <p>La préférence dans la publication de données ouvertes est pour le format <b>long</b>, et ce sera la méthode habituellement recommandée pour la publication. Cela dit, la conversion entre ces deux formats - pour autant que les données soient exploitables par une machine avec des métadonnées bien définies - est simple.\n",
    "</div>\n",
    "\n",
    "### 2.3.3 Définir les métadonnées et le schéma de destination\n",
    "\n",
    "La création d'un `schéma` est la première partie du processus de préparation des données. Votre schéma définit la structure cible de métadonnées pour votre processus de préparation. Ce n'est pas le format dans lequel vos données d'entrée arrivent, mais c'est ce à quoi vous voulez qu'elle ressemble quand vous avez terminé.\n",
    "\n",
    "Votre schéma définit les exigences, les contraintes et les valeurs par défaut raisonnables disponibles pour la saisie de données dans les champs définis par le schéma. Une fois terminé, vous devrez procéder à un nettoyage et à une validation supplémentaires.\n",
    "\n",
    "En termes simples, les colonnes d'un fichier CSV ou Excel d'entrée seront restructurées en de nouvelles colonnes définies par les champs de votre schéma. Ces champs cibles sont susceptibles d'être ceux d'une base de données. Tant que vos données d'entrée ne sont pas conformes à cette structure, vos données ne doivent pas être publiées en tant que données ouvertes.\n",
    "\n",
    "Nous utiliserons les définitions et le schéma de [Frictionless Data](https://specs.frictionlessdata.io/table-schema/#field-descriptors) pour définir les champs de métadonnées structurelles. \n",
    "\n",
    "#### ___`type` et `format`___\n",
    "\n",
    "Le `type` définit le type de données du champ, tandis que le `format` affine les propriétés spécifiques du `type`. Ce sont les types de base que vous êtes susceptible d'utiliser, avec des indentations pour les formats :\n",
    "\n",
    "- `string` : toute chaîne basée sur du texte.\n",
    "  - `default` : toute chaîne de caractères\n",
    "  - `email` : une adresse électronique\n",
    "  - `uri` : toute adresse web / URI\n",
    "- `number` : toute valeur basée sur un nombre, y compris les nombres entiers et les nombres flottants.\n",
    "- `integer` : toute valeur basée sur un nombre entier.\n",
    "- `boolean`: une valeur booléenne [`true`, `false`]. Peut définir des contraintes de catégorie pour fixer le terme utilisé.\n",
    "- `object` : toute donnée JSON valide.\n",
    "- `array` : toute donnée valide basée sur un tableau.\n",
    "- `date` : toute date sans heure. Doit être au format ISO8601, `YYYY-MM-DD`.\n",
    "- `datetime` : toute date avec une heure. Doit être au format ISO8601, avec l'heure UTC spécifiée (en option) comme `AAAA-MM-JJ hh:mm:ss Zz`.\n",
    "- `year` : toute année, formatée comme `YYYY`.\n",
    "\n",
    "#### ___ `schéma` design___\n",
    "\n",
    "Consultez les données ouvertes présentées sur le site [Données hospitalières relatives à l'épidémie de COVID-19](https://www.data.gouv.fr/fr/datasets/donnees-hospitalieres-relatives-a-lepidemie-de-covid-19/). Prévisualisez le fichier de schéma des métadonnées pour l'hospitalisation :\n",
    "\n",
    "![Métadonnées structurelles sur l'hospitalisation](images/01-01-structural-metadata-covid-hospitalisation.jpg)\n",
    "\n",
    "Le premier terme de chaque ligne définit le nom de la colonne de destination (les termes des métadonnées structurelles), ainsi que son type de données, les descriptions en anglais et en français et un exemple de valeur attendue pour ce champ.\n",
    "\n",
    "Avant de commencer à restructurer des fichiers de données désordonnés, créez une telle définition de métadonnées structurelles pour vos données. Et la première chose à faire lorsque vous commencez est simplement de regarder vos données et de comprendre comment elles sont actuellement structurées. L'objectif que vous poursuivez en passant de données désordonnées à des données structurées est simple :\n",
    "\n",
    " Préserver tout ce qui est connu de vos données sources lorsque vous les restructurez.\n",
    "\n",
    "Voici un résumé de ces exigences en matière de conception de schémas :\n",
    "\n",
    "- Etablir une convention standard pour nommer les champs de colonnes, et s'y tenir ; la convention se réfère à la structure physique du mot, comme `startDate` ou `place_name`. Une convention est `casCamel`, une autre est `séparation_substantielle`. Ne les mélangez pas car c'est frustrant pour les utilisateurs.\n",
    "- Chaque valeur de chaque colonne de chaque table ne doit représenter qu'une seule chose ; si vous regardez la colonne `région` ci-dessous, vous verrez qu'elle doit être divisée en deux colonnes séparées, `région` et `district`, pour la Côte d'Ivoire :\n",
    "\n",
    "| description | région                   | année | valeur    |\n",
    "|:------------|:-------------------------|:------|:----------|\n",
    "| Population  | Gbôklé, Bas-Sassandra    | 2014  | 400,798   |\n",
    "| Population  | Nawa, Bas-Sassandra      | 2014  | 1,053,084 |\n",
    "| Population  | San-Pédro, Bas-Sassandra | 2014  | 826,666   |\n",
    "| Population  | Indénié-Djuablin, Comoé  | 2014  | 560,432   |\n",
    "| Population  | Sud-Comoé, Comoé         | 2014  | 642,620   |\n",
    "\n",
    "- Définissez le `type`, le `format` et la `définition` pour chaque nom de colonne ; la définition du schéma ci-dessous est la forme __longues__. Si chaque `année` avait sa propre colonne, vous auriez alors - au lieu de `année` - une ligne pour `2014`, alors la prochaine version pourrait inclure `2016` et ainsi de suite. Vous pouvez voir que la mise à jour simultanée des données et du schéma représente plus de travail pour vous et vos utilisateurs que la simple mise à jour des données :\n",
    "\n",
    "| colonne     | type             | description                                      | exemple    |\n",
    "|:------------|:-----------------|:-------------------------------------------------|:-----------|\n",
    "| description | string           | Définition de la série de données démographiques | Population |\n",
    "| région      | string           | Nom d'une des 31 régions de Côte d'Ivoire        | Sud-Comoé  |\n",
    "| district    | string           | Nom d'un des 14 districts de la Côte d'Ivoire    | Comoé      |\n",
    "| année       | date             | L'année de la série de données                   | 2014       |\n",
    "| valeur      | integer          | La valeur pour la série de donnée                | 560432     |\n",
    "\n",
    "Il existe des exigences supplémentaires pour valider vos données - par exemple, noter que les virgules entre les chiffres doivent être supprimées - et nous allons les passer en revue ci-dessous.\n",
    "\n",
    "#### ___Conception de `métadonnées descriptives`___\n",
    "\n",
    "Avant de commencer à débattre de nos données, nous devrions d'abord examiner le tableau suivant des termes formels qui décrivent les données :\n",
    "\n",
    "| OBLIGATOIRE              | RECOMMANDÉ               | FACULTATIF               |\n",
    "|:-------------------------|:-------------------------|:-------------------------|\n",
    "| Titre                    | Tag(s)                   | Dernière mise à jour     |\n",
    "| Description              | Conditions d'utilisation | Fréquence de mise à jour |\n",
    "| Thème(s)                 | Courriel de contact      | Couverture géographique  |\n",
    "| Organisme de publication |                          | Couverture temporelle    |\n",
    "|                          |                          | Validité                 |\n",
    "|                          |                          | Ressources connexes      |\n",
    "|                          |                          | Règlements               |\n",
    "\n",
    "Ce sont les définitions et les termes de haut niveau pour décrire les données que vous restructurez. Une personne qui ouvre une feuille de calcul a très peu de contexte quant à ce qu'elle lit, et ces métadonnées fournissent un contexte bien nécessaire.\n",
    "\n",
    "La prochaine étape consiste à commencer à restructurer vos données désordonnées pour les rendre conformes au schéma que vous venez de créer.\n",
    "\n",
    "### 2.3.4 Les habitudes qui rendent les feuilles de calcul inutilisables\n",
    "\n",
    "Dans son excellent essai [The Art of Spreadsheets] (http://john.raffensperger.org/ArtOfTheSpreadsheet/Chapter09_ShowAllTheInformation.html) John Raffensperger énumère 37 façons de cacher des données dans un tableur. En voici 10 :\n",
    "\n",
    "- Ne pas partager le fichier. C'est la façon la plus courante de cacher des informations, et la plus efficace.\n",
    "- Cacher la feuille. Il vous faut donc d'abord au moins deux feuilles : Format, Feuille, Cacher.\n",
    "- Cachez la ligne : Format, Rangée, Cacher.\n",
    "- Cachez la colonne : Formater, Colonne, Cacher.\n",
    "- Cacher la cellule et protéger la feuille : Format, Cellules, Protection, Caché, puis Outils, Protection. Ceci affiche un écran, mais cache la formule : =if(1, \"Paix !\", \"Attaque à l'aube.\").\n",
    "- Rendre la colonne trop étroite : Format, Colonne, Largeur, 0.\n",
    "- Pour les formules susceptibles d'être nulles, utilisez Outils, Options, Affichage et désactivez la case des valeurs nulles. Par exemple : =IF(1, 0, \"Attaque à l'aube.\").\n",
    "- Utilisez une formule qui renvoie un blanc : =IF(1, \" \", \"Attaque à l'aube.\").\n",
    "- Créez une formule compliquée qui affiche les informations, mais formatez-les sous forme de texte (avec Format, Cellules, Nombre, Texte, ou commencez simplement la cellule par un simple guillemet), de sorte que la formule soit affichée plutôt que la sortie.\n",
    "- Formatez la police avec Wingdings : Format, Cells, Font, Wingdings. Cela permet d'afficher des caractères inintelligibles.\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "    <b>Exercice:</b>\n",
    "    <br>\n",
    "    <p><a href=\"https://www.data.gouv.fr/fr/datasets/donnees-hospitalieres-relatives-a-lepidemie-de-covid-19/\">Santé publique France</a> produit des mises à jour quotidiennes des données d'hospitalisation COVID-19 de la France. En vous inspirant de la liste de John Raffensperger, votre tâche consiste à brouiller autant que possible les données <a href=\"https://www.data.gouv.fr/fr/datasets/r/63352e38-d353-4b54-bfd1-f1b3ee1cabd7\">donnees-hospitalieres-covid19.csv</a>.\n",
    "    <p>Les notes seront attribuées pour:</p>\n",
    "    <ul>\n",
    "        <li> en rendant la présentation juste assez mauvaise pour que quelqu'un qui utilise les données soit tenté de penser qu'il peut encore les utiliser !\n",
    "        <li> l'utilisation d'effets de couleurs et de polices de caractères d'une manière qui offense vraiment l'œil</li>\n",
    "        <li> ingéniosité pour cacher des données à la vue de tous.</li>\n",
    "    </ul>\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Tutoriels et références:</b>\n",
    "    <br>\n",
    "    <ul>\n",
    "        <li><a href=\"http://blog.ouseful.info/2012/11/27/when-machine-readable-data-still-causes-issues-wrangling-dates/\">Quand les données lisibles par machine posent encore des \"problèmes\" - Les dates de dispute...</a></li>\n",
    "        <li><a href=\"http://www.opendataimpacts.net/2012/11/more-than-csv/\">Paysages numériques : des données ouvertes efficaces nécessitent plus qu'un simple CSV</a></li>\n",
    "        <li><a href=\"https://www.mikealche.com/software-development/a-humble-guide-to-database-schema-design\">Un guide pour la conception de schémas de bases de données</a></li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 Explorer et manipuler les données avec Pandas\n",
    "\n",
    "Ce serait bien si vous pouviez importer des données désordonnées et avoir un programme qui les restructure automatiquement pour vous dans le schéma de métadonnées que vous avez choisi. Il n'existe actuellement aucun mécanisme pour faire cela, et les préparations de données par programmation ne réduisent la complexité du processus que de façon limitée. Ca apporte seulement de réduire les tâches répétitives, comme le copier-coller, et limiter de façon important le risque d'introduire des erreurs liées à la copie.\n",
    "\n",
    "### Configurer votre environnement de développement et installer des paquets tiers\n",
    "\n",
    "Python est un langage de programmation. Au cœur de ce langage se trouvent les fonctions et les définitions que vous avez apprises dans la leçon 1.2.1. Cependant, Python est aussi une communauté de développeurs passionnés - tout comme vous - qui apportent chacun au langage de programmation leur intérêt particulier. Il peut s'agir de résoudre des mathématiques complexes, ou de dessiner des graphiques et des visualisations saisissantes, de créer des jeux informatiques, ou même de travailler avec des données structurées. Ce faisant, ils créent de nouvelles fonctions qu'ils peuvent regrouper sous forme de bibliothèques pour les mettre à la disposition des autres. La communauté Python a produit des dizaines de milliers de ces paquets, et nous allons en installer certains maintenant.\n",
    "\n",
    "Si vous ne l'avez pas encore fait, lancez votre navigateur Anaconda : \n",
    "\n",
    "![Jupyter menu (base)](images/jupyter-home-base.jpg)\n",
    "\n",
    "Sélectionnez Environnements dans le menu de gauche :\n",
    "\n",
    "![Jupyter menu](images/jupyter-menu.jpg)\n",
    "\n",
    "Anaconda vous permet de créer des environnements de développement isolés. L'important est de comprendre qu'il conserve votre logiciel de programmation dans sa propre \"zone de jeu\", ce qui le rend facile à étendre et à supprimer. Créez un nouvel environnement en cliquant sur le bouton \"Create\" au bas de l'écran : \n",
    "\n",
    "![Jupyter environments](images/jupyter-environments.JPG \"Jupyter environments\")\n",
    "\n",
    "Donnez à votre environnement le nom que vous voulez, mais vous pouvez l'appeler `datascience` pour qu'il soit facile à trouver. Maintenant, vous devez installer les paquets dont nous avons besoin pendant ce cours. Il y a deux façons de le faire, l'une pour les paquets que Conda connaît déjà, et l'autre à partir du terminal du système d'exploitation. Nous allons maintenant appliquer la seconde méthode qui nous permettra d'installer tout ce dont nous avons besoin.\n",
    "\n",
    "Faites un clic gauche sur la \"flèche\" à côté de `datascience` et choisissez `Open Terminal`. Assurez-vous que vous cliquez sur le bon environnement ; pas sur `base` ou`root`, mais sur le nom de l'environnement que vous venez de créer : \n",
    "\n",
    "![Jupyter terminal](images/jupyter-terminal.JPG \"Jupyter terminal\")\n",
    "\n",
    "Une nouvelle fenêtre s'ouvrira avec une invite de commande. Tapez ce qui suit et entrez.\n",
    "\n",
    "    pip install whyqd\n",
    "    \n",
    "![Jupyter terminal window](images/jupyter-terminal-window.JPG \"Jupyter terminal window\")\n",
    "  \n",
    "[Whyqd](https://github.com/whythawk/whyqd/) comporte des dépendances que nous utiliserons lors des prochaines leçons, dont [Pandas](https://pandas.pydata.org/pandas-docs/stable/), qui seront également installées automatiquement. Nous n'utiliserons pas `whyqd` avant la prochaine leçon, mais c'est un bon moment pour s'assurer que nous l'avons. Faites maintenant la même chose pour `matplotlib`, une bibliothèque de graphiques :\n",
    "\n",
    "    pip install matplotlib\n",
    "    \n",
    "Une fois l'installation terminée (et il se peut que vous deviez répondre \"oui\" à l'invite), retournez à la page d'accueil et cliquez sur pour installer Jupyter Notebook.\n",
    "\n",
    "![Jupyter module launcher](images/jupyter-datascience-home.JPG \"Jupyter Module Launcher\")\n",
    "\n",
    "Une fois installé, lancez-le : \n",
    "\n",
    "![Jupyter launch](images/jupyter-launch.JPG \"Launch Jupyter Notebook\")\n",
    "\n",
    "Vous devrez lancer Jupyter Notebook chaque fois que vous voudrez faire un tutoriel ou modifier vos carnets. Heureusement, Anaconda crée un raccourci Windows pour que vous puissiez le trouver facilement :\n",
    "\n",
    "![Jupyter Windows launcher](images/jupyter-windows-launcher.JPG \"Jupyter Windows launcher\")\n",
    "\n",
    "Si vous êtes bloqué, [lisez les documents](https://docs.anaconda.com/) ou cherchez des solutions sur [Stack Overflow](https://stackoverflow.com/).\n",
    "\n",
    "Il existe une multitude de tutoriels et de ressources en ligne qui vous permettront d'approfondir les sujets abordés dans cette leçon, et vous pourrez probablement les trouver vous-même. La documentation des bibliothèques que nous utilisons est également une ressource essentielle et vous familiariser avec la façon de la lire et de l'utiliser vous rapportera énormément :\n",
    " \n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Références aux bibliothèques installées pour ce cours:</b>\n",
    "    <br>\n",
    "    <ul>\n",
    "        <li><a href=\"https://docs.anaconda.com/\">Anaconda</a></li>\n",
    "        <li><a href=\"http://jupyter.org/documentation.html\">Jupyter Notebook</a></li>\n",
    "        <li><a href=\"https://pandas.pydata.org/pandas-docs/stable/\">Pandas</a></li>\n",
    "        <li><a href=\"https://docs.scipy.org/doc/numpy-1.13.0/user/\">Numpy</a></li>\n",
    "        <li><a href=\"https://matplotlib.org/\">Matplotlib</a></li>\n",
    "        <li><a href=\"https://python-poetry.org/docs/\">Poetry</a></li>\n",
    "        <li><a href=\"https://whyqd.readthedocs.io/en/latest/\">Whyqd</a></li>\n",
    "    </ul>\n",
    "</div>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exemple de préparation de données\n",
    "\n",
    "Travailler avec des données par programme signifie travailler de manière symbolique. La plupart du temps, vous ne regarderez pas les colonnes et les lignes de données comme vous le feriez dans un tableur. L'avantage est que vous n'imaginerez pas de modèles qui ne se trouvent pas dans les données, et que vous pouvez travailler avec de très grands ensembles de données. L'inconvénient est qu'il n'est pas toujours possible de savoir quand quelque chose va mal.\n",
    "\n",
    "La première étape reste la nécessité que les données soient lisibles par la machine. Cela signifie que vous devez être bon dans les compétences présentées dans la première leçon. Le logiciel que nous allons utiliser ne peut pas ouvrir des fichiers qui ne sont pas préparés correctement.\n",
    "\n",
    "Avant de pouvoir explorer les données, vous devez vous assurer qu'il y a une _ligne d'en-tête_ et qu'il n'y a _pas de colonnes ou de champs cachés_. Veuillez revoir la leçon 1.1 si vous n'êtes pas absolument sûr de ce que cela signifie.\n",
    "\n",
    "Commençons à travailler avec le module que vous utiliserez et que vous apprendrez pour le reste de ce cours : `pandas`.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Syntaxe</b>\n",
    "    <br>\n",
    "    <ul>\n",
    "        <li>Après avoir importé un module, <code>dir(module)</code> vous permet de voir une liste de toutes les fonctions implémentées dans cette bibliothèque.</li>\n",
    "        <li>Vous pouvez également lire l'aide du module avec <code>help(module)</code></li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on package pandas:\n",
      "\n",
      "NAME\n",
      "    pandas\n",
      "\n",
      "DESCRIPTION\n",
      "    pandas - a powerful data analysis and manipulation library for Python\n",
      "    =====================================================================\n",
      "    \n",
      "    **pandas** is a Python package providing fast, flexible, and expressive data\n",
      "    structures designed to make working with \"relational\" or \"labeled\" data both\n",
      "    easy and intuitive. It aims to be the fundamental high-level building block for\n",
      "    doing practical, **real world** data analysis in Python. Additionally, it has\n",
      "    the broader goal of becoming **the most powerful and flexible open source data\n",
      "    analysis / manipulation tool available in any language**. It is already well on\n",
      "    its way toward this goal.\n",
      "    \n",
      "    Main Features\n",
      "    -------------\n",
      "    Here are just a few of the things that pandas does well:\n",
      "    \n",
      "      - Easy handling of missing data in floating point as well as non-floating\n",
      "        point data.\n",
      "      - Size mutability: columns can be inserted and deleted from DataFrame and\n",
      "        higher dimensional objects\n",
      "      - Automatic and explicit data alignment: objects can be explicitly aligned\n",
      "        to a set of labels, or the user can simply ignore the labels and let\n",
      "        `Series`, `DataFrame`, etc. automatically align the data for you in\n",
      "        computations.\n",
      "      - Powerful, flexible group by functionality to perform split-apply-combine\n",
      "        operations on data sets, for both aggregating and transforming data.\n",
      "      - Make it easy to convert ragged, differently-indexed data in other Python\n",
      "        and NumPy data structures into DataFrame objects.\n",
      "      - Intelligent label-based slicing, fancy indexing, and subsetting of large\n",
      "        data sets.\n",
      "      - Intuitive merging and joining data sets.\n",
      "      - Flexible reshaping and pivoting of data sets.\n",
      "      - Hierarchical labeling of axes (possible to have multiple labels per tick).\n",
      "      - Robust IO tools for loading data from flat files (CSV and delimited),\n",
      "        Excel files, databases, and saving/loading data from the ultrafast HDF5\n",
      "        format.\n",
      "      - Time series-specific functionality: date range generation and frequency\n",
      "        conversion, moving window statistics, date shifting and lagging.\n",
      "\n",
      "PACKAGE CONTENTS\n",
      "    _config (package)\n",
      "    _libs (package)\n",
      "    _testing\n",
      "    _typing\n",
      "    _version\n",
      "    api (package)\n",
      "    arrays (package)\n",
      "    compat (package)\n",
      "    conftest\n",
      "    core (package)\n",
      "    errors (package)\n",
      "    io (package)\n",
      "    plotting (package)\n",
      "    testing\n",
      "    tests (package)\n",
      "    tseries (package)\n",
      "    util (package)\n",
      "\n",
      "SUBMODULES\n",
      "    _hashtable\n",
      "    _lib\n",
      "    _tslib\n",
      "    offsets\n",
      "\n",
      "FUNCTIONS\n",
      "    __getattr__(name)\n",
      "\n",
      "DATA\n",
      "    IndexSlice = <pandas.core.indexing._IndexSlice object>\n",
      "    NA = <NA>\n",
      "    NaT = NaT\n",
      "    __docformat__ = 'restructuredtext'\n",
      "    __git_version__ = '3adf3340453d6704d4a2cb47058214cc697a7d29'\n",
      "    describe_option = <pandas._config.config.CallableDynamicDoc object>\n",
      "    get_option = <pandas._config.config.CallableDynamicDoc object>\n",
      "    options = <pandas._config.config.DictWrapper object>\n",
      "    reset_option = <pandas._config.config.CallableDynamicDoc object>\n",
      "    set_option = <pandas._config.config.CallableDynamicDoc object>\n",
      "\n",
      "VERSION\n",
      "    1.0.3\n",
      "\n",
      "FILE\n",
      "    c:\\users\\turukawa\\.conda\\envs\\datascience\\lib\\site-packages\\pandas\\__init__.py\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "help(pd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous importons un module avec `import` et pouvons le renommer en utilisant le terme `as` :\n",
    "\n",
    "    import pandas as pd\n",
    "    import numpy as np\n",
    "    \n",
    "Il n'y a rien de spécial avec `pd` ou `np`, c'est juste une convention, mais nous utiliserons ___beaucoup___ ces modules, donc raccourcir les pandas en pd et numpy en np rend la frappe légèrement plus facile. Nous allons maintenant importer le même fichier Excel que nous avons utilisé dans la leçon 1.1 et faire exactement le même tutoriel qu'auparavant, mais cette fois-ci en utilisant  `pandas` et  `numpy`.\n",
    "\n",
    "Nous commençons par importer le fichier Excel. Dans pandas, une feuille de calcul est appelée `dataframe`, ou `df` en abrégé."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Data Source</th>\n",
       "      <th>World Development Indicators</th>\n",
       "      <th>Unnamed: 2</th>\n",
       "      <th>Unnamed: 3</th>\n",
       "      <th>Unnamed: 4</th>\n",
       "      <th>Unnamed: 5</th>\n",
       "      <th>Unnamed: 6</th>\n",
       "      <th>Unnamed: 7</th>\n",
       "      <th>Unnamed: 8</th>\n",
       "      <th>Unnamed: 9</th>\n",
       "      <th>...</th>\n",
       "      <th>Unnamed: 54</th>\n",
       "      <th>Unnamed: 55</th>\n",
       "      <th>Unnamed: 56</th>\n",
       "      <th>Unnamed: 57</th>\n",
       "      <th>Unnamed: 58</th>\n",
       "      <th>Unnamed: 59</th>\n",
       "      <th>Unnamed: 60</th>\n",
       "      <th>Unnamed: 61</th>\n",
       "      <th>Unnamed: 62</th>\n",
       "      <th>Unnamed: 63</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Last Updated Date</td>\n",
       "      <td>2020-04-09 00:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Country Name</td>\n",
       "      <td>Country Code</td>\n",
       "      <td>Indicator Name</td>\n",
       "      <td>Indicator Code</td>\n",
       "      <td>1960.0</td>\n",
       "      <td>1961.0</td>\n",
       "      <td>1962.0</td>\n",
       "      <td>1963.0</td>\n",
       "      <td>1964.0</td>\n",
       "      <td>1965.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>2011.0</td>\n",
       "      <td>2012.0</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2017.0</td>\n",
       "      <td>2018.0</td>\n",
       "      <td>2019.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Aruba</td>\n",
       "      <td>ABW</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>27526.0</td>\n",
       "      <td>28141.0</td>\n",
       "      <td>28532.0</td>\n",
       "      <td>28761.0</td>\n",
       "      <td>28924.0</td>\n",
       "      <td>29082.0</td>\n",
       "      <td>...</td>\n",
       "      <td>43778.0</td>\n",
       "      <td>43819.0</td>\n",
       "      <td>44057.0</td>\n",
       "      <td>44348.0</td>\n",
       "      <td>44665.0</td>\n",
       "      <td>44979.0</td>\n",
       "      <td>45296.0</td>\n",
       "      <td>45616.0</td>\n",
       "      <td>45948.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>AFG</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>755836.0</td>\n",
       "      <td>796272.0</td>\n",
       "      <td>839385.0</td>\n",
       "      <td>885228.0</td>\n",
       "      <td>934135.0</td>\n",
       "      <td>986074.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6927764.0</td>\n",
       "      <td>7212518.0</td>\n",
       "      <td>7528588.0</td>\n",
       "      <td>7865067.0</td>\n",
       "      <td>8204877.0</td>\n",
       "      <td>8535606.0</td>\n",
       "      <td>8852859.0</td>\n",
       "      <td>9164841.0</td>\n",
       "      <td>9477100.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 64 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Data Source World Development Indicators        Unnamed: 2  \\\n",
       "0  Last Updated Date          2020-04-09 00:00:00               NaN   \n",
       "1                NaN                          NaN               NaN   \n",
       "2       Country Name                 Country Code    Indicator Name   \n",
       "3              Aruba                          ABW  Urban population   \n",
       "4        Afghanistan                          AFG  Urban population   \n",
       "\n",
       "       Unnamed: 3  Unnamed: 4  Unnamed: 5  Unnamed: 6  Unnamed: 7  Unnamed: 8  \\\n",
       "0             NaN         NaN         NaN         NaN         NaN         NaN   \n",
       "1             NaN         NaN         NaN         NaN         NaN         NaN   \n",
       "2  Indicator Code      1960.0      1961.0      1962.0      1963.0      1964.0   \n",
       "3     SP.URB.TOTL     27526.0     28141.0     28532.0     28761.0     28924.0   \n",
       "4     SP.URB.TOTL    755836.0    796272.0    839385.0    885228.0    934135.0   \n",
       "\n",
       "   Unnamed: 9  ...  Unnamed: 54  Unnamed: 55  Unnamed: 56  Unnamed: 57  \\\n",
       "0         NaN  ...          NaN          NaN          NaN          NaN   \n",
       "1         NaN  ...          NaN          NaN          NaN          NaN   \n",
       "2      1965.0  ...       2010.0       2011.0       2012.0       2013.0   \n",
       "3     29082.0  ...      43778.0      43819.0      44057.0      44348.0   \n",
       "4    986074.0  ...    6927764.0    7212518.0    7528588.0    7865067.0   \n",
       "\n",
       "   Unnamed: 58  Unnamed: 59  Unnamed: 60  Unnamed: 61  Unnamed: 62  \\\n",
       "0          NaN          NaN          NaN          NaN          NaN   \n",
       "1          NaN          NaN          NaN          NaN          NaN   \n",
       "2       2014.0       2015.0       2016.0       2017.0       2018.0   \n",
       "3      44665.0      44979.0      45296.0      45616.0      45948.0   \n",
       "4    8204877.0    8535606.0    8852859.0    9164841.0    9477100.0   \n",
       "\n",
       "   Unnamed: 63  \n",
       "0          NaN  \n",
       "1          NaN  \n",
       "2       2019.0  \n",
       "3          NaN  \n",
       "4          NaN  \n",
       "\n",
       "[5 rows x 64 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "source = \"http://api.worldbank.org/v2/en/indicator/SP.URB.TOTL?downloadformat=excel\"\n",
    "# read_excel est une fonction de pandas qui vous ouvrira un fichier Excel\n",
    "# la \"source\" peut être n'importe où, soit dans un fichier local, soit dans une ressource en ligne distante\n",
    "df = pd.read_excel(source)\n",
    "# head imprimera les x premières lignes du fichier de données pour que nous puissions voir à quoi il ressemble\n",
    "df.head(5) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "C'est un problème! mais la ligne d'en-tête n'est pas dans la première rangée, elle est en fait dans la quatrième. Réimportons notre fichier et assurons-nous d'ignorer les trois premières rangées :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country Name</th>\n",
       "      <th>Country Code</th>\n",
       "      <th>Indicator Name</th>\n",
       "      <th>Indicator Code</th>\n",
       "      <th>1960</th>\n",
       "      <th>1961</th>\n",
       "      <th>1962</th>\n",
       "      <th>1963</th>\n",
       "      <th>1964</th>\n",
       "      <th>1965</th>\n",
       "      <th>...</th>\n",
       "      <th>2010</th>\n",
       "      <th>2011</th>\n",
       "      <th>2012</th>\n",
       "      <th>2013</th>\n",
       "      <th>2014</th>\n",
       "      <th>2015</th>\n",
       "      <th>2016</th>\n",
       "      <th>2017</th>\n",
       "      <th>2018</th>\n",
       "      <th>2019</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Aruba</td>\n",
       "      <td>ABW</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>27526.0</td>\n",
       "      <td>28141.0</td>\n",
       "      <td>28532.0</td>\n",
       "      <td>28761.0</td>\n",
       "      <td>28924.0</td>\n",
       "      <td>29082.0</td>\n",
       "      <td>...</td>\n",
       "      <td>43778.0</td>\n",
       "      <td>43819.0</td>\n",
       "      <td>44057.0</td>\n",
       "      <td>44348.0</td>\n",
       "      <td>44665.0</td>\n",
       "      <td>44979.0</td>\n",
       "      <td>45296.0</td>\n",
       "      <td>45616.0</td>\n",
       "      <td>45948.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>AFG</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>755836.0</td>\n",
       "      <td>796272.0</td>\n",
       "      <td>839385.0</td>\n",
       "      <td>885228.0</td>\n",
       "      <td>934135.0</td>\n",
       "      <td>986074.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6927764.0</td>\n",
       "      <td>7212518.0</td>\n",
       "      <td>7528588.0</td>\n",
       "      <td>7865067.0</td>\n",
       "      <td>8204877.0</td>\n",
       "      <td>8535606.0</td>\n",
       "      <td>8852859.0</td>\n",
       "      <td>9164841.0</td>\n",
       "      <td>9477100.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Angola</td>\n",
       "      <td>AGO</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>569222.0</td>\n",
       "      <td>597288.0</td>\n",
       "      <td>628381.0</td>\n",
       "      <td>660180.0</td>\n",
       "      <td>691532.0</td>\n",
       "      <td>721552.0</td>\n",
       "      <td>...</td>\n",
       "      <td>13963065.0</td>\n",
       "      <td>14660282.0</td>\n",
       "      <td>15383127.0</td>\n",
       "      <td>16130304.0</td>\n",
       "      <td>16900847.0</td>\n",
       "      <td>17691524.0</td>\n",
       "      <td>18502165.0</td>\n",
       "      <td>19332881.0</td>\n",
       "      <td>20184707.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Albania</td>\n",
       "      <td>ALB</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>493982.0</td>\n",
       "      <td>513592.0</td>\n",
       "      <td>530766.0</td>\n",
       "      <td>547928.0</td>\n",
       "      <td>565248.0</td>\n",
       "      <td>582374.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1519519.0</td>\n",
       "      <td>1546929.0</td>\n",
       "      <td>1575788.0</td>\n",
       "      <td>1603505.0</td>\n",
       "      <td>1630119.0</td>\n",
       "      <td>1654503.0</td>\n",
       "      <td>1680247.0</td>\n",
       "      <td>1706345.0</td>\n",
       "      <td>1728969.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Andorra</td>\n",
       "      <td>AND</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>7839.0</td>\n",
       "      <td>8766.0</td>\n",
       "      <td>9754.0</td>\n",
       "      <td>10811.0</td>\n",
       "      <td>11915.0</td>\n",
       "      <td>13067.0</td>\n",
       "      <td>...</td>\n",
       "      <td>75007.0</td>\n",
       "      <td>74305.0</td>\n",
       "      <td>73056.0</td>\n",
       "      <td>71515.0</td>\n",
       "      <td>70057.0</td>\n",
       "      <td>68919.0</td>\n",
       "      <td>68213.0</td>\n",
       "      <td>67876.0</td>\n",
       "      <td>67813.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 64 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  Country Name Country Code    Indicator Name Indicator Code      1960  \\\n",
       "0        Aruba          ABW  Urban population    SP.URB.TOTL   27526.0   \n",
       "1  Afghanistan          AFG  Urban population    SP.URB.TOTL  755836.0   \n",
       "2       Angola          AGO  Urban population    SP.URB.TOTL  569222.0   \n",
       "3      Albania          ALB  Urban population    SP.URB.TOTL  493982.0   \n",
       "4      Andorra          AND  Urban population    SP.URB.TOTL    7839.0   \n",
       "\n",
       "       1961      1962      1963      1964      1965  ...        2010  \\\n",
       "0   28141.0   28532.0   28761.0   28924.0   29082.0  ...     43778.0   \n",
       "1  796272.0  839385.0  885228.0  934135.0  986074.0  ...   6927764.0   \n",
       "2  597288.0  628381.0  660180.0  691532.0  721552.0  ...  13963065.0   \n",
       "3  513592.0  530766.0  547928.0  565248.0  582374.0  ...   1519519.0   \n",
       "4    8766.0    9754.0   10811.0   11915.0   13067.0  ...     75007.0   \n",
       "\n",
       "         2011        2012        2013        2014        2015        2016  \\\n",
       "0     43819.0     44057.0     44348.0     44665.0     44979.0     45296.0   \n",
       "1   7212518.0   7528588.0   7865067.0   8204877.0   8535606.0   8852859.0   \n",
       "2  14660282.0  15383127.0  16130304.0  16900847.0  17691524.0  18502165.0   \n",
       "3   1546929.0   1575788.0   1603505.0   1630119.0   1654503.0   1680247.0   \n",
       "4     74305.0     73056.0     71515.0     70057.0     68919.0     68213.0   \n",
       "\n",
       "         2017        2018  2019  \n",
       "0     45616.0     45948.0   NaN  \n",
       "1   9164841.0   9477100.0   NaN  \n",
       "2  19332881.0  20184707.0   NaN  \n",
       "3   1706345.0   1728969.0   NaN  \n",
       "4     67876.0     67813.0   NaN  \n",
       "\n",
       "[5 rows x 64 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_excel(source, header=3)\n",
    "df.head(5) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vous remarquez la valeur `NaN` dans la colonne `2019` ?  `NaN` est l'abréviation de `Not-a-Number`, et c'est la façon dont  pandas (et `numpy`, le logiciel sur lequel il fonctionne) garde la trace de nombres indéfinis. Dans ce cas, cela signifie qu'il n'y a pas de chiffres disponibles pour certains pays pour cette année-là. Pandas essaiera de deviner le format des colonnes pour vous, mais il arrive qu'il se trompe. Vérifions et voyons ce qu'il en pense :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 264 entries, 0 to 263\n",
      "Data columns (total 64 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   Country Name    264 non-null    object \n",
      " 1   Country Code    264 non-null    object \n",
      " 2   Indicator Name  264 non-null    object \n",
      " 3   Indicator Code  264 non-null    object \n",
      " 4   1960            258 non-null    float64\n",
      " 5   1961            258 non-null    float64\n",
      " 6   1962            258 non-null    float64\n",
      " 7   1963            258 non-null    float64\n",
      " 8   1964            258 non-null    float64\n",
      " 9   1965            258 non-null    float64\n",
      " 10  1966            258 non-null    float64\n",
      " 11  1967            258 non-null    float64\n",
      " 12  1968            258 non-null    float64\n",
      " 13  1969            258 non-null    float64\n",
      " 14  1970            258 non-null    float64\n",
      " 15  1971            258 non-null    float64\n",
      " 16  1972            258 non-null    float64\n",
      " 17  1973            258 non-null    float64\n",
      " 18  1974            258 non-null    float64\n",
      " 19  1975            258 non-null    float64\n",
      " 20  1976            258 non-null    float64\n",
      " 21  1977            258 non-null    float64\n",
      " 22  1978            258 non-null    float64\n",
      " 23  1979            258 non-null    float64\n",
      " 24  1980            258 non-null    float64\n",
      " 25  1981            258 non-null    float64\n",
      " 26  1982            258 non-null    float64\n",
      " 27  1983            258 non-null    float64\n",
      " 28  1984            258 non-null    float64\n",
      " 29  1985            258 non-null    float64\n",
      " 30  1986            258 non-null    float64\n",
      " 31  1987            258 non-null    float64\n",
      " 32  1988            258 non-null    float64\n",
      " 33  1989            258 non-null    float64\n",
      " 34  1990            260 non-null    float64\n",
      " 35  1991            260 non-null    float64\n",
      " 36  1992            259 non-null    float64\n",
      " 37  1993            259 non-null    float64\n",
      " 38  1994            259 non-null    float64\n",
      " 39  1995            260 non-null    float64\n",
      " 40  1996            260 non-null    float64\n",
      " 41  1997            260 non-null    float64\n",
      " 42  1998            261 non-null    float64\n",
      " 43  1999            261 non-null    float64\n",
      " 44  2000            261 non-null    float64\n",
      " 45  2001            261 non-null    float64\n",
      " 46  2002            261 non-null    float64\n",
      " 47  2003            261 non-null    float64\n",
      " 48  2004            261 non-null    float64\n",
      " 49  2005            261 non-null    float64\n",
      " 50  2006            261 non-null    float64\n",
      " 51  2007            261 non-null    float64\n",
      " 52  2008            261 non-null    float64\n",
      " 53  2009            261 non-null    float64\n",
      " 54  2010            261 non-null    float64\n",
      " 55  2011            261 non-null    float64\n",
      " 56  2012            260 non-null    float64\n",
      " 57  2013            260 non-null    float64\n",
      " 58  2014            260 non-null    float64\n",
      " 59  2015            260 non-null    float64\n",
      " 60  2016            260 non-null    float64\n",
      " 61  2017            260 non-null    float64\n",
      " 62  2018            260 non-null    float64\n",
      " 63  2019            0 non-null      float64\n",
      "dtypes: float64(60), object(4)\n",
      "memory usage: 132.1+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`dtypes` nous indiquent les types de données pour chacune des colonnes. `object` est n'importe quelle sorte de `string` ou de type de données qui ne correspondend pas aux types de données numériques, de catégories (on verra ca plus tard) ou de date. `float64` signifie que ces valeurs sont considérées comme des nombres à virgule flottante de 64 bits.\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <p>À partir de maintenant, je ne ferai référence à la documentation que pour chaque nouvelle fonction que j'introduirai sous forme de commentaire dans le code, avec un lien vers une ressource en ligne où vous pourrez trouver des informations supplémentaires. C'est une habitude de codage utile à prendre car vous pouvez parfois oublier exactement comment vous avez découvert cette solution ou cette référence.</p>\n",
    "    <p>Par exemple, <code># https://stackoverflow.com/a/51794989</code> est une référence à <a href=\"https://stackoverflow.com\">Stackoverflow</a> - un site de questions et réponses très populaire parmi les développeurs. Si vous visitez ce lien, vous obtiendrez une foule d'informations sur la ligne de code, ou la solution technique, qui suit. Certaines références, par exemple <code># Référence pour np.where https://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html</code>, vous mèneront à la documentation du paquet.</p>\n",
    "    <p>Prendre l'habitude de lire des documents de référence. Le développement de logiciels exige de la patience et vous serez toujours en train de chercher. Faire vos propres recherches fait de vous un bon développeur.</p>\n",
    "</div>\n",
    "\n",
    "La liste complète des colonnes peut être consultée sous forme de liste :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Country Name', 'Country Code', 'Indicator Name', 'Indicator Code',\n",
       "       '1960', '1961', '1962', '1963', '1964', '1965', '1966', '1967', '1968',\n",
       "       '1969', '1970', '1971', '1972', '1973', '1974', '1975', '1976', '1977',\n",
       "       '1978', '1979', '1980', '1981', '1982', '1983', '1984', '1985', '1986',\n",
       "       '1987', '1988', '1989', '1990', '1991', '1992', '1993', '1994', '1995',\n",
       "       '1996', '1997', '1998', '1999', '2000', '2001', '2002', '2003', '2004',\n",
       "       '2005', '2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013',\n",
       "       '2014', '2015', '2016', '2017', '2018', '2019'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le fichier est actuellement au format `large` et nous préférerions qu'il soit `long`, avec toutes les années qui s'écoulent le long de la trame de données. Pour ce faire, nous devons faire 'pivoter (instruction `pivot`) notre cadre de données et créer une nouvelle colonne `année`. Pandas dispose d'une fonction appelée `melt` pour ce faire, mais nous devons d'abord obtenir les colonnes que nous voulons utiliser :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country Name</th>\n",
       "      <th>Country Code</th>\n",
       "      <th>Indicator Name</th>\n",
       "      <th>Indicator Code</th>\n",
       "      <th>Année</th>\n",
       "      <th>Valeur</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Aruba</td>\n",
       "      <td>ABW</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>1982</td>\n",
       "      <td>30943.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>AFG</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>1982</td>\n",
       "      <td>2208967.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Angola</td>\n",
       "      <td>AGO</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>1982</td>\n",
       "      <td>2382559.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Albania</td>\n",
       "      <td>ALB</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>1982</td>\n",
       "      <td>954645.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Andorra</td>\n",
       "      <td>AND</td>\n",
       "      <td>Urban population</td>\n",
       "      <td>SP.URB.TOTL</td>\n",
       "      <td>1982</td>\n",
       "      <td>36561.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Country Name Country Code    Indicator Name Indicator Code Année     Valeur\n",
       "0        Aruba          ABW  Urban population    SP.URB.TOTL  1982    30943.0\n",
       "1  Afghanistan          AFG  Urban population    SP.URB.TOTL  1982  2208967.0\n",
       "2       Angola          AGO  Urban population    SP.URB.TOTL  1982  2382559.0\n",
       "3      Albania          ALB  Urban population    SP.URB.TOTL  1982   954645.0\n",
       "4      Andorra          AND  Urban population    SP.URB.TOTL  1982    36561.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_columns = ['Country Name', 'Country Code', 'Indicator Name', 'Indicator Code']\n",
    "year_columns = set(df.columns).difference(set(id_columns))\n",
    "\n",
    "# https://pandas.pydata.org/docs/reference/api/pandas.melt.html\n",
    "df = pd.melt(df, id_vars=id_columns, value_vars=year_columns,\n",
    "             var_name='Année', value_name='Valeur')\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15840 entries, 0 to 15839\n",
      "Data columns (total 6 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   Country Name    15840 non-null  object \n",
      " 1   Country Code    15840 non-null  object \n",
      " 2   Indicator Name  15840 non-null  object \n",
      " 3   Indicator Code  15840 non-null  object \n",
      " 4   Année           15840 non-null  object \n",
      " 5   Valeur          15291 non-null  float64\n",
      "dtypes: float64(1), object(5)\n",
      "memory usage: 742.6+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Année` est de type `object`. S'il est important de convertir cela en un type de date, alors il faudrait le faire. Cependant, le fichier convient parfaitement à nos besoins, nous pouvons donc l'enregistrer au format CSV :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "destination = \"data/lesson-programmatic/world-urban-population-data.csv\"\n",
    "# `index=False` signifie que nous n'allons pas garder cette colonne d'index - elle sera recréée lorsque nous ouvrirons le fichier à l'avenir\n",
    "df.to_csv(destination, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas peux faire bien plus que simplement déplacer des données. Vous pouvez aussi filtrer, trier et dessiner des graphiques :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x2a3268abe08>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAERCAYAAABrWly6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU1f3/8deHEAg7EnZICCCL7EsIorZSt+IGFVGk1A0qVq3VWmvV+nX9Wa1aq622FRUBsaJSFPcdFUXAhB0CskMACYSdkP3z+2MGjSGBBJLMZPJ+Ph55cHPunZnPYcKbmzvnnmPujoiIVH01Ql2AiIiUDwW6iEiEUKCLiEQIBbqISIRQoIuIRAgFuohIhAhpoJvZBDNLN7OlpTi2nZl9YmaLzewzM2tbGTWKiFQVoT5DnwgMKeWxjwGT3b0XcD/wUEUVJSJSFYU00N39C2Bn4TYz62hm75tZipnNMrOuwV3dgE+C2zOBYZVYqohI2Av1GXpxxgM3unt/4FbgX8H2RcDFwe2LgAZmFhuC+kREwlLNUBdQmJnVB04BXjOzQ821g3/eCjxlZlcBXwCbgbzKrlFEJFyFVaAT+I1ht7v3KbrD3bcAw+H74L/Y3fdUcn0iImErrC65uPteYJ2ZXQJgAb2D203N7FC9dwATQlSmiEhYCvWwxZeBr4EuZpZmZmOB0cBYM1sELOOHDz8HAyvN7FugBfBgCEoWEQlbpulzRUQiQ1hdchERkWMXsg9FmzZt6gkJCaF6eRGRKiklJWWHuzcrbl/IAj0hIYHk5ORQvbyISJVkZhtK2qdLLiIiEUKBLiISIRToIiIRIqzuFM3NzSUtLY2srKxQlxIWYmJiaNu2LdHR0aEuRUSqgLAK9LS0NBo0aEBCQgKF5nKpltydjIwM0tLSaN++fajLEZEqIKwuuWRlZREbG1vtwxzAzIiNjdVvKyJSamEV6IDCvBD9XYhIWYRdoIuIRCp358mPV7F8y94KeX4FeiGDBw/mgw8++FHbE088wfXXX1/iY+rXr1/RZYlIBCgocO6esYy/f/wtby7aUiGvoUAvZNSoUUydOvVHbVOnTmXUqFEV/tp5eVqrQyRS5eUXcOtri3hxzgbG/bQDfxrSpUJeR4FeyIgRI3j77bfJzs4GYP369WzZsoU+ffpw5pln0q9fP3r27MmMGTOKffyjjz7KgAED6NWrF/fcc8/3z9GjR4/vj3nssce49957gcBvBHfeeSenn346Tz75ZMV2TkRCIjsvnxv+O5/pCzbzh7M7c8e5XSvs87GwGrZY2H1vLSv360zdWjfkngu7l7g/NjaWpKQk3n//fYYNG8bUqVMZOXIkderU4fXXX6dhw4bs2LGDk08+maFDh/7oTfnwww9ZtWoV8+bNw90ZOnQoX3zxBfHx8Uesaffu3Xz++efl1kcRCR+ZOXlc+2IKs1bt4O4LujHmtIodgqwz9CIKX3Y5dLnF3bnzzjvp1asXZ511Fps3b2bbtm0/etyHH37Ihx9+SN++fenXrx8rVqxg1apVR329kSNHVkg/RCS09mblcuWEeXy1egePXNyrwsMcwvgM/Uhn0hXpF7/4Bbfccgvz58/n4MGD9OvXj4kTJ7J9+3ZSUlKIjo4mISHhsPHh7s4dd9zBtdde+6P2tLQ0CgoKvv++6OPq1atXcZ0RkZDYeSCHKyfMI3XrXv4xqi8X9GpdKa+rM/Qi6tevz+DBgxkzZsz3H4bu2bOH5s2bEx0dzcyZM9mw4fDZK3/+858zYcIE9u/fD8DmzZtJT0+nRYsWpKenk5GRQXZ2Nm+//Xal9kdEKlf63ixGPvM1K7ftY/wV/SstzCGMz9BDadSoUQwfPvz7Sy+jR4/mwgsvJDExkT59+tC1a9fDHnPOOeeQmprKoEGDgMB/DFOmTKF58+bcfffdDBw4kPbt2xf7WBGJDGm7Mhn93Fy278tm4tUDOKVj00p9/ZCtKZqYmOhFF7hITU3lpJNOCkk94Up/JyJVw9rt+xn93FwOZOcxcUwS/eJPqJDXMbMUd08sbl+pL7mYWZSZLTCzw64ZmFltM3vFzFab2VwzSzj2ckVEqpbUrXu59Jmvyckr4OVxJ1dYmB9NWa6h3wSklrBvLLDL3U8E/g789XgLExGpChZu2s1l4+dQs0YNXrl2EN1bNwpZLaUKdDNrC5wPPFfCIcOAScHtacCZdowj50N1CSgc6e9CJLzNWZvB6Gfn0LBOTV77zSBObB7aqUBKe4b+BHAbUFDC/jbAJgB3zwP2ALFFDzKzcWaWbGbJ27dvP+xJYmJiyMjIUJDxw3zoMTExoS5FRIoxc2U6V06YR6vGdXjt2lOIa1I31CUdfZSLmV0ApLt7ipkNLumwYtoOS2V3Hw+Mh8CHokX3t23blrS0NIoL++ro0IpFIhJe3l2ylZumLqBziwZMHpNEbP3aoS4JKN2wxVOBoWZ2HhADNDSzKe7+q0LHpAFxQJqZ1QQaATvLWkx0dLRW5xGRsDYtJY3bpi2iX/wJPH/VABrVCZ8lIo96ycXd73D3tu6eAFwGfFokzAHeBK4Mbo8IHqPrJiISUV74ah23vraIUzo2ZfLYpLAKcziOG4vM7H4g2d3fBJ4HXjSz1QTOzC8rp/pERELO3Xnyk1U88fEqhnRvyZOj+lC7ZlSoyzpMmQLd3T8DPgtu312oPQu4pDwLExEJBwUFzgPvLOeFr9Yzon9bHh7ek5pR4Tlrim79FxEpQV5+AbdPX8K0lDTGnNqeu84/iRo1wnetXwW6iEgxsvPyuenlhby/7Dt+f1ZnfnfmiWG/cLsCXUSkiP3ZeYybnMzsNRmVsjBFeVGgi4gUkrE/m6snfsOyLXv5+8jeXNS36twLokAXEQnavPsglz8/l827DjL+8v6ceVKLUJdUJgp0ERFgdfo+Ln9+Hvuz83hx7ECS2jcJdUllpkAXkWpvwcZdjJn4DVE1avDKuEF0a90w1CUdEwW6iFRrM1emc/2U+TRrUJvJY5JIaFp11/lVoItItTV9fhq3TVtM5xYNmDhmAM0bVO3ZTRXoIlItjf9iDX95dwWndIzlmcv70yAmvOZlORYKdBGpVgoKnIfeS+XZWes4v2crHh/ZOyznZTkWCnQRqTZy8gq4bdoi3li4hSsGteOeC7sTFca38peVAl1EqoV9WblcN2U+X67ewa3ndOaGn4X/rfxlpUAXkYiXvjeLq174hm+37ePREb24JDEu1CVVCAW6iES01en7uXLCPHZl5vDclYkM7tI81CVVGAW6iESslA07GTspmZo1jFfGDaJn20ahLqlCKdBFJCK9v3QrN01dSKtGMUweM5D42LqhLqnCKdBFJOJM+HIdD7yznL5xjXnuygE0qVcr1CVVCgW6iESM/ALnwXdSmfDVOn7evQVPXtaXmOjIGGNeGgp0EYkIWbn53Dw1sMLQ1acmcNf53SJqjHlpKNBFpMrbdSCHsZO+YcGm3fzfBd0YW0VWGCpvCnQRqdK27D7IFRPmsTEjk6d/2Y/zerYKdUkhU+NoB5hZjJnNM7NFZrbMzO4r5pirzGy7mS0Mfv26YsoVEfnBqm37uPjfs9m2J4tJY5KqdZhD6c7Qs4Ez3H2/mUUDX5rZe+4+p8hxr7j7b8u/RBGRw6VsCCxKUatmDV65tuouSlGejhro7u7A/uC30cEvr8iiRESO5NMV27j+pfm0bBjDi2MHEtck8seYl8ZRL7kAmFmUmS0E0oGP3H1uMYddbGaLzWyamUXmRAkiEnKvJW/imskpdGregGnXnaIwL6RUge7u+e7eB2gLJJlZjyKHvAUkuHsv4GNgUnHPY2bjzCzZzJK3b99+PHWLSDXj7vzrs9X8cdpiBnWI5eVxJ9O0fu1QlxVWShXoh7j7buAzYEiR9gx3zw5++yzQv4THj3f3RHdPbNas2TGUKyLVUUGBc99by3nk/ZUM7d2aCVcNoH5tDdIrqjSjXJqZWePgdh3gLGBFkWMKf7Q8FEgtzyJFpPrKzsvnxqkLmDh7PWNPa88TI/tQq2aZzkWrjdL8F9cKmGRmUQT+A3jV3d82s/uBZHd/E/idmQ0F8oCdwFUVVbCIVB/7snK59sUUZq/J4M7zujLupx1DXVJYs8AglsqXmJjoycnJIXltEQl/hReleGREL4b3axvqksKCmaW4e2Jx+3QRSkTCzprtgUUpdh7I4fmrBnB6Z33mVhoKdBEJK/M37mLsxG+IqmFMHXcyvdo2DnVJVYYCXUTCxiep27jhv/Np0TCGyWOSaBdbL9QlVSkKdBEJC1PnbeTPbyyle+uGTLhqgMaYHwMFuoiElLvz2IcreXrmGk7v3Ix/je5HPY0xPyb6WxORkMnOy+e2aYuZsXALo5LiuH9YD6KjNMb8WCnQRSQkdmfmMG5yCvPW7+S2IV247vSOmFWvFYbKmwJdRCrdxoxMrpo4j7SdB/nHqL4M7d061CVFBAW6iFSq+Rt3cc2kZPLdmfLrgSS1bxLqkiKGAl1EKs27S7by+1cW0qJhDC9cPYCOzeqHuqSIokAXkQrn7oz/Yi0PvbeC/u1OYPzl/YnVsMRyp0AXkQqVl1/A3W8u479zN3J+r1b87ZLexERHhbqsiKRAF5EKsy8rl9/+dwGff7ud6wZ35I/ndKFGDY1kqSgKdBGpEJt3H2TsxG9Ylb6fh4b3ZFRSfKhLingKdBEpd4vTdjN2UjJZOflMvHoAP+mk2RIrgwJdRMrVB8u+46apC4itV5uXrh9I5xYNQl1StaFAF5Fy4e48OyswkqV328Y8e0UizRpoJEtlUqCLyHHLzS/g7hnLeHneRs7r2ZLHL+2jkSwhoEAXkeOyOzOH66bM5+u1GVw/uCO3aiRLyCjQReSYrd2+n7GTktm86yB/u6Q3F/fXup+hpEAXkWPy1eodXDclhZpRNXjpmoEMSNCcLKGmQBeRMntp7gbunrGMDk3r8fyVA4iPrRvqkgQFuoiUQW5+AQ+8vZzJX2/g9M7N+Ocv+9IwJjrUZUnQUQPdzGKAL4DaweOnufs9RY6pDUwG+gMZwEh3X1/u1YpIyOzOzOH6l+Yze00G1/ykPbefexJR+vAzrJTmDD0bOMPd95tZNPClmb3n7nMKHTMW2OXuJ5rZZcBfgZEVUK+IhMDq9H2MnZTM1t1ZPDqiF5ckxoW6JCnGURfv84D9wW+jg19e5LBhwKTg9jTgTNNaUiIRYeaKdC56ejYHsvN5edxAhXkYK9VqrGYWZWYLgXTgI3efW+SQNsAmAHfPA/YAscU8zzgzSzaz5O3btx9f5SJSodyd/3y+hjGTviE+ti5v/vZU+rfTSJZwVqpAd/d8d+8DtAWSzKxHkUOKOxsvehaPu49390R3T2zWTJP1iISrgzn53DR1IQ+/t4Lze7Zi2m9OoXXjOqEuS46iTKNc3H23mX0GDAGWFtqVBsQBaWZWE2gE7CyvIkWk8mzefZBxk5NZvnUvtw3pwnWnd0RXUKuGo56hm1kzM2sc3K4DnAWsKHLYm8CVwe0RwKfuftgZuoiEt3nrdjL0n1+yMSOT569M5PrBJyrMq5DSnKG3AiaZWRSB/wBedfe3zex+INnd3wSeB140s9UEzswvq7CKRaTcuTsTZ6/nwXdSiWtSl2ev6M+JzTXtbVVz1EB398VA32La7y60nQVcUr6liUhl2J+dx5/+t5h3Fm/lrJOa87dL+9Cojm4Wqop0p6hINfbttn38ZkoK63cc4E9DunLtTztopsQqTIEuUk29sWAzd0xfQr3aNXnp1yczqONhI42lilGgi1QzmTl53P/WcqZ+s4mk9k14alRfmjeMCXVZUg4U6CLVyJK0Pdw0dQHrMg5w/eCO3HJ2Z2pGlep2FKkCFOgi1UBBgTN+1lr+9uFKmtavzX91iSUiKdBFItx3e7K45dWFzF6Twbk9WvLQ8J40rlsr1GVJBVCgi0Swd5ds5Y7pS8jJK+CvF/fk0sQ43SgUwRToIhFoX1Yu9721nGkpafRu24i/j+xDh2b1Q12WVDAFukiESdmwk5tfWcjmXQe58YwT+d2ZnYjWB5/VggJdJELk5hfwz09X89Snq2jduA6vXjuIRC3cXK0o0EUiwLodB7j5lYUs2rSb4f3acN/Q7jTQWp/VjgJdpApzd6Z+s4n731pOrZo1ePqX/Ti/V6tQlyUhokAXqaJ27M/m9v8t4ePUbZx2YlMeu6Q3LRvpjs/qTIEuUgV9umIbt01bwt6sXP7vgm5cfUqCJtUSBbpIVXIgO4//904qL8/bSNeWDZjy6yS6tmwY6rIkTCjQRaqIlA27uOXVhWzcmcm1p3fglrM7U7tmVKjLkjCiQBcJc7n5Bfzjk1U8PXM1rRrVYeo1JzOwg+ZhkcMp0EXC2Kpt+/j9qwtZunkvF/dry71Du2k4opRIgS4ShgoKnAlfreORD1ZSr1YU/x7dj3N7ajiiHJkCXSTMpO3K5NbXFjFn7U7OOqk5fxnek+YNNBxRjk6BLhIm3J1pKWnc99Zy3J1HLu7FJYltNTuilJoCXSQM7DqQw52vL+G9pd+R1L4Jf7ukN3FN6oa6LKlijhroZhYHTAZaAgXAeHd/ssgxg4EZwLpg03R3v798SxWJTF98u51bX1vErswcbj+3K9f8pANRuklIjkFpztDzgD+4+3wzawCkmNlH7r68yHGz3P2C8i9RJDJl5ebz8HsrmDh7PZ2a12fCVQPo0aZRqMuSKuyoge7uW4Gtwe19ZpYKtAGKBrqIlNKStD3c8upCVqXv56pTErj93K7EROsmITk+ZbqGbmYJQF9gbjG7B5nZImALcKu7Lyvm8eOAcQDx8fFlrVWkysvNL+CpT1fz1MzVNK1fi0ljkji9c7NQlyURotSBbmb1gf8BN7v73iK75wPt3H2/mZ0HvAF0Kvoc7j4eGA+QmJjox1y1SBW08rt93PLqQpZt2ctFfdtw74XdaVRXNwlJ+SlVoJtZNIEwf8ndpxfdXzjg3f1dM/uXmTV19x3lV6pI1ZRf4Dw7ay2Pf/gtDWJq8p9f9WdIj5ahLksiUGlGuRjwPJDq7o+XcExLYJu7u5klATWAjHKtVKQKWrfjALe+toiUDbsY0r0lD17Ug9j6tUNdlkSo0pyhnwpcDiwxs4XBtjuBeAB3/w8wArjOzPKAg8Bl7q5LKlJtFRQ4k79ez8Pvr6BWVA2eGNmHYX1a6yYhqVClGeXyJXDEn0J3fwp4qryKEqnK0nZl8sfXFvP12gwGd2nGw8N7aSUhqRS6U1SknLg7L8/bxF/eTcXdeXh4T0YOiNNZuVQaBbpIOdiYkcnt0xcze00GgzrE8siIXrp1XyqdAl3kOBQUOJO+Xs8j768kqobxl4t6MipJZ+USGgp0kWO0Zvt+/jRtMckbdvGzLs148KKetG5cJ9RlSTWmQBcpo4ICZ+Ls9fz1/RXEREfx+KW9uahvG52VS8gp0EXKYNPOTP44LbD4xJldm/PQ8J40b6gRLBIeFOgipeDuvJq8iQfeTgXQ4hMSlhToIkeRvjeL26cv4dMV6QzqEMujl/Si7QkawSLhR4EucgRvL97CXW8s5WBOPvdc2I0rByVQQ4tPSJhSoIsUY3dmDv83YxlvLdpC77jGPH5pbzo2qx/qskSOSIEuUsTMlen8adpidh7I4dZzOvOb0ztSM6pGqMsSOSoFukjQnoO5PPjOcl5NTqNLiwZaEk6qHAW6CDBzRTp3TF9C+r4srhvckZvO7KQl4aTKUaBLtbYnM5f7317O/+an0blFfZ65/FR6xzUOdVkix0SBLtXWx8u3cefrS8g4kMONZ5zIb884kdo1dVYuVZcCXaqdXQdyuO+tZbyxcAtdW+pauUQOBbpUK+8v/Y673ljK7swcbj6rE9cPPpFaNTWCRSKDAl2qhZ0HcrjnzcC48u6tGzJ5TBLdWjcMdVki5UqBLhHN3Xlr8VbufXMZ+7Jy+cPZnfnN4I5Ea1y5RCAFukSs7/ZkcdcbS/g4NZ3ecY155OJedGnZINRliVQYBbpEHHdn6jeb+Ms7qeQWFHDX+Sdx9antidIcLBLhFOgSUdbtOMCd05fw9drA2p4PX9yTdrH1Ql2WSKU4aqCbWRwwGWgJFADj3f3JIscY8CRwHpAJXOXu88u/XJHi5eQV8Mzna/jnzNXUrlmDh4b35LIBWttTqpfSnKHnAX9w9/lm1gBIMbOP3H15oWPOBToFvwYC/w7+KVLhktfv5I7pS1iVvp/ze7Xingu6aRUhqZaOGujuvhXYGtzeZ2apQBugcKAPAya7uwNzzKyxmbUKPlakQuzJzOWRD1bw0tyNtGlchwlXJXJG1xahLkskZMp0Dd3MEoC+wNwiu9oAmwp9nxZs+1Ggm9k4YBxAfHx82SoVCXJ3Xl+wmb+8m8rOAzmMPa09t5zdmXq19ZGQVG+l/hdgZvWB/wE3u/veoruLeYgf1uA+HhgPkJiYeNh+kaP5dts+7npjKfPW7aRvfGMmjUmie2vdti8CpQx0M4smEOYvufv0Yg5JA+IKfd8W2HL85YkEHMjO4x+frOL5L9dRP6YmDw/vyaWJcVoOTqSQ0oxyMeB5INXdHy/hsDeB35rZVAIfhu7R9XMpD4fu9PzLO6l8tzeLkYlx/OncrjSpVyvUpYmEndKcoZ8KXA4sMbOFwbY7gXgAd/8P8C6BIYurCQxbvLr8S5XqZuV3+7h7xlLmrttJjzYNeXp0P/q3OyHUZYmErdKMcvmS4q+RFz7GgRvKqyip3vYczOWJj79l8tcbaBBTkwcv6sFlA+J1p6fIUWhYgISN/ALnteRNPPrBSnZm5jB6YDx/OLsLJ+jyikipKNAlLCSv38m9by1j6ea9DEg4gUkXJmnRCZEyUqBLSG3dc5CH3l3Bm4u20KpRDP8Y1ZcLe7XSLfsix0CBLiGRnZfPc7PW8dSnqylw53dnnMhvBnekbi39SIocK/3rkUr3Seo27n97ORsyMhnSvSV/Pv8k4prUDXVZIlWeAl0qzbodB7j/rWXMXLmdjs3q8eLYJH7SqVmoyxKJGAp0qXAHsvN4euZqnpu1jlo1a/Dn807iylMStDizSDlToEuFKXqX5/C+bbj93K6a2lakgijQpUKkbt3LvW8uY+66nXRv3ZCnR/elf7smoS5LJKIp0KVc7TyQw98/+pb/zttIQ93lKVKpFOhSLnLyCpj89Xqe/GQVmTn5jB4Yzy1nd6ZxXd3lKVJZFOhyXNydD5dv46F3U1mfkcngLs3483kn0alFg1CXJlLtKNDlmC1J28OD7y5nztqddGpen4lXD2Bwl+ahLkuk2lKgS5ml7crksQ9W8sbCLcTWq8UDw7ozKimemlEahigSSgp0KbU9B3P512ereeGr9Rhww8868pvTO9IgJjrUpYkICnQphf3ZeUz8ah3PzlrH3qxcLurbhlvP6ULrxnVCXZqIFKJAlxLtz85j0uz1PDtrLbszczmza3N+f3ZnTWsrEqYU6HKYvVm5vDRnI+O/WMOuzFx+1qUZN5/Vmd5xjUNdmogcgQJdvrdl90Fe+GodL8/bxP7sPAZ3acZNZ3aib7zW8RSpChTowvIte3l21lreWrQFB87v2YpxP+2gSysiVYwCvZoqKHA+X7Wd52et48vVO6hXK4orT0ng6lMTaHuC5iYXqYoU6NVMVm4+byzYzHNfrmN1+n5aNKzNbUO6MDqpHY3qavihSFV21EA3swnABUC6u/coZv9gYAawLtg03d3vL88i5fht35fNlDkbmDJnAxkHcujWqiF/H9mb83u21rzkIhGiNGfoE4GngMlHOGaWu19QLhVJuUrdupcJX65jxsIt5OQXcEbX5vz6J+0Z1CFWCzGLRJijBrq7f2FmCRVfipQXd+ezldt5dtZaZq/JoE50FJcOaMvVp7anY7P6oS5PRCpIeV1DH2Rmi4AtwK3uvqy4g8xsHDAOID4+vpxeWg7JzS/grUVbeObztazcto+WDWP405CujEqK0zS2ItVAeQT6fKCdu+83s/OAN4BOxR3o7uOB8QCJiYleDq8tBNbsnPrNJp6ftZYte7Lo0qIBj1/amwt7tyZaE2aJVBvHHejuvrfQ9rtm9i8za+ruO473ueXItu45yOSvN/DfuRvZczCXpIQm/L+LevCzLs11fVykGjruQDezlsA2d3czSwJqABnHXZmUaMHGXUz4aj3vLtmKu3NOt5Zc89MO9G+nOzpFqrPSDFt8GRgMNDWzNOAeIBrA3f8DjACuM7M84CBwmbvrcko5y8kr4L2lW5k4ez0LNu6mQe2ajDk1gSsGJRDXRDcCiUjpRrmMOsr+pwgMa5QKkL4vi5fnbmLK3A1s35dNQmxd7hvanYv7t6V+bd0XJiI/UCKEIXdn4abdTJq9nneWbCU33zm9czOuGpHA6Z2aUaOGro+LyOEU6GEkKzeftxZtYfLXG1iyeQ/1a9dk9MB2XDGoHR00flxEjkKBHgY27cxkytwNvPLNJnZn5tKpeX0eGNadi/rpsoqIlJ7SIkTy8guYuXI7L83dwOffbqeGGed0a8EVgxI4uUMTDTsUkTJToFeybXuzeOWbTbw8byNb92TRvEFtbjyjE6OS4mjVSGt0isixU6BXgoICZ/aaDKbM2cBHqdvIL3B+0qkp91zYnTNPaq67OUWkXCjQK9CuAzlMS0njpbkbWJ+RyQl1oxl7Wnt+mRRPQtN6oS5PRCKMAr2cHTobfzV5E+8v+46cvAIGJJzAzWd1ZkiPlsRER4W6RBGJUAr0crJpZybTUtKYlpLG5t0HaVQnmlED4hg1MJ6uLRuGujwRqQYU6MfhYE4+7y/bymvJacxek4EZnHZiU24/tytnd2uhs3ERqVQK9DJydxZs2s1ryWm8vWgL+7LziGtSh9+f1ZkRiW1p01gjVUQkNBTopbRtbxavL9jMtJQ0VqfvJya6Buf1aMUliXEMbN9Et+OLSMgp0I8gKzefj5ZvY1pKGrNWbafAoX+7E3hoeE8u6NWKBjHRoS5RROR7CvQi3J2UDbuYvmAzby/awt6sPFo1iuH6wScyvF8bzakiImFLgR60MSOT1xdsZvqCNDZkZFInOoqfd2/BiP5xDOoYS5QuqYhImKvWgZ5f4Mxckc6kr9cza9UOzGBQh+itIvEAAAk5SURBVFhuPKMTQ3q01MRYIlKlVMvE2nUgh1eTN/HinA2k7TpIy4Yx3HJ2Zy7ur1EqIlJ1VatAX5y2mylzNjBj4Ray8woY2L4Jd553Emd3a6H5VESkyov4QD+0aMSUORtYlLaHOtFRDO/XhisGJXBSK93BKSKRI2IDfcV3e3ktOXAr/p6DuZzYvD73XtiN4f3b0lDDDUUkAkVUoO/OzGHGwi28lrKJpZv3Eh1lnNOtJb86uZ0WjRCRiFflAz07L5/PVm5nxsLNfLw8nZz8Arq3bsg9F3ZjWJ82NKlXK9QliohUiqMGuplNAC4A0t29RzH7DXgSOA/IBK5y9/nlXWhh+QXOnLUZzFi4mfeWfse+rDxi69Vi9MnxXNI/jm6tdW1cRKqf0pyhTwSeAiaXsP9coFPwayDw7+CfFeKT1G3cPn0J2/dlU792Tc7p3oJhfdpwasdYamqkiohUY0cNdHf/wswSjnDIMGCyuzswx8wam1krd99aTjX+SNsT6tIvvjHD+rThjK7NNUWtiEhQeVxDbwNsKvR9WrDtsEA3s3HAOID4+PhjerEuLRvwzOWJx/RYEZFIVh7XKIobOuLFHeju49090d0TmzVrVg4vLSIih5RHoKcBcYW+bwtsKYfnFRGRMiiPQH8TuMICTgb2VNT1cxERKVlphi2+DAwGmppZGnAPEA3g7v8B3iUwZHE1gWGLV1dUsSIiUrLSjHIZdZT9DtxQbhWJiMgx0cBtEZEIoUAXEYkQCnQRkQhhgUvgIXhhs+3AhmN8eFNgRzmWEw4irU+R1h+IvD5FWn8g8vpUXH/auXuxN/KELNCPh5klu3tE3S4aaX2KtP5A5PUp0voDkdensvZHl1xERCKEAl1EJEJU1UAfH+oCKkCk9SnS+gOR16dI6w9EXp/K1J8qeQ1dREQOV1XP0EVEpAgFuohIhAibQDezCWaWbmZLC7X1NrOvzWyJmb1lZg0L7esV3LcsuD8m2N4/+P1qM/tHcM3TsO6PmY02s4WFvgrMrE849ecY+hRtZpOC7almdkehxwwxs5XBPt0eir4E6yhLf2qZ2QvB9kVmNrjQY8LpPYozs5nBv/NlZnZTsL2JmX1kZquCf54QbLdgzavNbLGZ9Sv0XFcGj19lZldWkf50Db5/2WZ2a5HnCoufuwrl7mHxBfwU6AcsLdT2DXB6cHsM8EBwuyawGOgd/D4WiApuzwMGEVh44z3g3HDvT5HH9QTWFvo+LPpzDO/RL4Gpwe26wHogAYgC1gAdgFrAIqBbFejPDcALwe3mQApQIwzfo1ZAv+B2A+BboBvwCHB7sP124K/B7fOCNRtwMjA32N4EWBv884Tg9glVoD/NgQHAg8CthZ4nbH7uKvIrbM7Q3f0LYGeR5i7AF8Htj4CLg9vnAIvdfVHwsRnunm9mrYCG7v61B97FycAvKr76w5WxP4WNAl4GCKf+QJn75EA9M6sJ1AFygL1AErDa3de6ew4wlcC6tJWujP3pBnwSfFw6sBtIDMP3aKu7zw9u7wNSCSwJOQyYFDxsEj/U+P2awO4+B2gc7NPPgY/cfae77yLwdzGkErsClL0/7p7u7t8AuUWeKmx+7ipS2AR6CZYCQ4Pbl/DDykidATezD8xsvpndFmxvQ2AFpUMOrW8aLkrqT2EjCQY64d8fKLlP04ADBNaW3Qg85u47KXkN2nBRUn8WAcPMrKaZtQf6B/eF7XtkgcXd+wJzgRYeXHgm+Gfz4GElvR9h9z6Vsj8lCbv+VIRwD/QxwA1mlkLg162cYHtN4DRgdPDPi8zsTMqwvmmIlNQfAMxsIJDp7oeu6YZ7f6DkPiUB+UBroD3wBzPrQPj3qaT+TCAQAsnAE8BsII8w7Y+Z1Qf+B9zs7nuPdGgxbX6E9pAoQ39KfIpi2kL+PpW3oy5wEUruvoLA5RXMrDNwfnBXGvC5u+8I7nuXwLXQKQTWND0krNY3PUJ/DrmMH87OIdDPsO0PHLFPvwTed/dcIN3MvgISCZwlhe0atCX1x93zgN8fOs7MZgOrgF2E2XtkZtEEwu8ld58ebN5mZq3cfWvwkkp6sL2kNYHTCKxUVrj9s4qsuyRl7E9JqsXax2F9hm5mzYN/1gDuAv4T3PUB0MvM6gav0Z4OLA/+6rXPzE4OjjS4ApgRgtKLdYT+HGq7hMC1PeD7XyXDtj9wxD5tBM4IjqKoR+ADtxUEPnTsZGbtzawWgf/E3qz8yotXUn+CP2v1gttnA3nuHnY/c8EangdS3f3xQrveBA6NVLmSH2osaU3gD4BzzOyE4AiSc4JtleoY+lOSsP65Kzeh/lT20BeBM9OtBD7MSAPGAjcR+FT7W+Bhgne2Bo//FbCMwDXPRwq1Jwbb1gBPFX5MmPdnMDCnmOcJi/6UtU9AfeC14Hu0HPhjoec5L3j8GuDPVaQ/CcBKAh/KfUxgCtNwfI9OI3ApYTGwMPh1HoGRYJ8Q+K3iE6BJ8HgDng7WvgRILPRcYwisFbwauLqK9Kdl8L3cS+CD6zQCH1qHzc9dRX7p1n8RkQgR1pdcRESk9BToIiIRQoEuIhIhFOgiIhFCgS5SBmZ27aGJoETCjQJdIpqZXWRmbmZdy+G57gYOzW0iEnY0bFEimpm9SmDGvk/c/d4QlyNSoXSGLhErOP/HqQRuGLos2DbYzD4zs2lmtsLMXgrejYiZrTez+4ITvi05dFZvZvUsMHf6N2a2wMyGBdujzOzRYPtiM7s2RF0VARToEtl+QWA+mW+BnfbD4g19gZsJTInbgUDoH7LD3fsB/wYOLZDwZ+BTdx8A/Ax4NDgNwFgCt8oPIDAH9zXBmRhFQkKBLpFsFD/MjTM1+D3APHdPc/cCAreSJxR6zKHJn1IKtZ8D3G5mCwlMUBUDxAfbrwi2zyVwO3qniuiISGmE9WyLIsfKzGKBM4AeZuYEVqxx4F0gu9Ch+fz430F2Me0GXOzuK4u8hgE3unulT1olUhydoUukGkFgJZ527p7g7nHAOgKTPZXVB8CNha619y3Ufl1welfMrPOhGRlFQkGBLpFqFPB6kbb/EZinvaweAKKBxRZYUPqBYPtzBGaSnB9sfwb91ishpGGLIiIRQmfoIiIRQoEuIhIhFOgiIhFCgS4iEiEU6CIiEUKBLiISIRToIiIR4v8DxmVODdRA/QgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# importer la bibliothèque de graphiques dont nous avons besoin et la mettre en ligne avec le carnet (similaire a import mais spécifique a jupyter notebook pour que les graphiques soient affichés dans le notebook)\n",
    "%matplotlib inline\n",
    "# D'abord, triez par année pour vous assurer que tout est en ordre\n",
    "# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sort_values.html\n",
    "# inplace = True signifie qu'il va trier et enregistrer dans la variable df\n",
    "df.sort_values(by=\"Année\", inplace=True)\n",
    "# Ici, nous filtrons la colonne df[\"Country Name\"] uniquement par des valeurs == \"World\"\n",
    "# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.line.html\n",
    "df[df[\"Country Name\"] == \"World\"].plot(x=\"Année\", y=\"Valeur\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.5 Tutoriel - nettoyage de données complexes\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "    <p>Choisissez une feuille de calcul dans <a href=\"data/lesson-spreadsheet/\">données de formation</a> et restructurez-la selon les techniques et les exigences présentées dans cette leçon.</p>\n",
    "    <p>Les données de ce dossier proviennent également de la Banque mondiale, mais elles datent d'une époque bien antérieure, avant que la Banque mondiale ne sache ce qu'étaient des données ouvertes ou exploitables par une machine. Il contient certains des pires exemples de manipulation de données que vous n'aurez jamais vus.</p>\n",
    "</div>\n",
    "\n",
    "![Feuille de calcul du tutoriel](images/tutorial-1-2.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 1</th>\n",
       "      <th>Unnamed: 2</th>\n",
       "      <th>Monitoring human development: enlarging people's choices …</th>\n",
       "      <th>Unnamed: 4</th>\n",
       "      <th>Unnamed: 5</th>\n",
       "      <th>Unnamed: 6</th>\n",
       "      <th>Unnamed: 7</th>\n",
       "      <th>Unnamed: 8</th>\n",
       "      <th>Unnamed: 9</th>\n",
       "      <th>...</th>\n",
       "      <th>Unnamed: 21</th>\n",
       "      <th>Unnamed: 22</th>\n",
       "      <th>Unnamed: 23</th>\n",
       "      <th>Unnamed: 24</th>\n",
       "      <th>Unnamed: 25</th>\n",
       "      <th>Unnamed: 26</th>\n",
       "      <th>Unnamed: 27</th>\n",
       "      <th>Unnamed: 28</th>\n",
       "      <th>Unnamed: 29</th>\n",
       "      <th>Unnamed: 30</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3 Human and income poverty Developing countries</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Probability at birth of not surviving to age 4...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adult illiteracy rateb,†\\n(% aged 15 and older...</td>\n",
       "      <td>...</td>\n",
       "      <td>HPI-1 rank minus income poverty rankc</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Human poverty index                         (H...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>HDI rank</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Rank</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Value \\n (%)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          Unnamed: 0 Unnamed: 1  Unnamed: 2  \\\n",
       "0   3 Human and income poverty Developing countries         NaN         NaN   \n",
       "1                                                NaN        NaN         NaN   \n",
       "2                                                NaN        NaN         NaN   \n",
       "3                                                NaN        NaN         NaN   \n",
       "4                                                NaN        NaN         NaN   \n",
       "5                                                NaN        NaN         NaN   \n",
       "6                                                NaN        NaN         NaN   \n",
       "7                                                NaN        NaN         NaN   \n",
       "8                                                NaN        NaN         NaN   \n",
       "9                                                NaN        NaN         NaN   \n",
       "10                                          HDI rank        NaN         NaN   \n",
       "11                                               NaN        NaN         NaN   \n",
       "\n",
       "   Monitoring human development: enlarging people's choices … Unnamed: 4  \\\n",
       "0                                                 NaN                NaN   \n",
       "1                                                 NaN                NaN   \n",
       "2                                                 NaN                NaN   \n",
       "3                                                 NaN                NaN   \n",
       "4                                                 NaN                NaN   \n",
       "5                                                 NaN                NaN   \n",
       "6                                                 NaN                NaN   \n",
       "7                                                 NaN                NaN   \n",
       "8                                                 NaN                NaN   \n",
       "9   Human poverty index                         (H...                NaN   \n",
       "10                                                NaN                NaN   \n",
       "11                                               Rank                NaN   \n",
       "\n",
       "      Unnamed: 5  Unnamed: 6  \\\n",
       "0            NaN         NaN   \n",
       "1            NaN         NaN   \n",
       "2            NaN         NaN   \n",
       "3            NaN         NaN   \n",
       "4            NaN         NaN   \n",
       "5            NaN         NaN   \n",
       "6            NaN         NaN   \n",
       "7            NaN         NaN   \n",
       "8            NaN         NaN   \n",
       "9            NaN         NaN   \n",
       "10           NaN         NaN   \n",
       "11  Value \\n (%)         NaN   \n",
       "\n",
       "                                           Unnamed: 7 Unnamed: 8  \\\n",
       "0                                                 NaN        NaN   \n",
       "1                                                 NaN        NaN   \n",
       "2                                                 NaN        NaN   \n",
       "3                                                 NaN        NaN   \n",
       "4                                                 NaN        NaN   \n",
       "5                                                 NaN        NaN   \n",
       "6                                                 NaN        NaN   \n",
       "7   Probability at birth of not surviving to age 4...        NaN   \n",
       "8                                                 NaN        NaN   \n",
       "9                                                 NaN        NaN   \n",
       "10                                                NaN        NaN   \n",
       "11                                                NaN        NaN   \n",
       "\n",
       "                                           Unnamed: 9  ...  \\\n",
       "0                                                 NaN  ...   \n",
       "1                                                 NaN  ...   \n",
       "2                                                 NaN  ...   \n",
       "3                                                 NaN  ...   \n",
       "4                                                 NaN  ...   \n",
       "5                                                 NaN  ...   \n",
       "6                                                 NaN  ...   \n",
       "7                                                 NaN  ...   \n",
       "8   Adult illiteracy rateb,†\\n(% aged 15 and older...  ...   \n",
       "9                                                 NaN  ...   \n",
       "10                                                NaN  ...   \n",
       "11                                                NaN  ...   \n",
       "\n",
       "                              Unnamed: 21 Unnamed: 22 Unnamed: 23 Unnamed: 24  \\\n",
       "0                                     NaN         NaN         NaN         NaN   \n",
       "1                                     NaN         NaN         NaN         NaN   \n",
       "2                                     NaN         NaN         NaN         NaN   \n",
       "3                                     NaN         NaN         NaN         NaN   \n",
       "4                                     NaN         NaN         NaN         NaN   \n",
       "5                                     NaN         NaN         NaN         NaN   \n",
       "6                                     NaN         NaN         NaN         NaN   \n",
       "7                                     NaN         NaN         NaN         NaN   \n",
       "8   HPI-1 rank minus income poverty rankc         NaN         NaN         NaN   \n",
       "9                                     NaN         NaN         NaN         NaN   \n",
       "10                                    NaN         NaN         NaN         NaN   \n",
       "11                                    NaN         NaN         NaN         NaN   \n",
       "\n",
       "   Unnamed: 25 Unnamed: 26 Unnamed: 27 Unnamed: 28 Unnamed: 29 Unnamed: 30  \n",
       "0          NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "1          NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "2          NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "3          NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "4          NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "5          NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "6          NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "7          NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "8          NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "9          NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "10         NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "11         NaN         NaN         NaN         NaN         NaN         NaN  \n",
       "\n",
       "[12 rows x 31 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "source = \"data/lesson-spreadsheet/HDR 2007-2008 Table 03.xlsx\"\n",
    "\n",
    "df = pd.read_excel(source)\n",
    "df.head(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comme dans l'exemple précédent, vous pouvez voir que la \"vraie\" ligne d'en-tête ne commence qu'aux alentours de la ligne 7, et même alors, elle apparaît étalée sur plusieurs lignes. De plus, il existe toute une série de problèmes différents. Voyez si vous pouvez tous les repérer :\n",
    "\n",
    "- Colonnes et lignes apparemment vides (remplies de rien d'autre que des valeurs nulles `NaNs`)\n",
    "- Caractères d'impression spéciaux (comme `\\n`)\n",
    "- Références qui n'ont probablement pas leur place dans ce tableur\n",
    "\n",
    "Il y en aura d'autres, mais essayons de résoudre les problèmes immédiats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 1</th>\n",
       "      <th>Unnamed: 3</th>\n",
       "      <th>Unnamed: 4</th>\n",
       "      <th>Unnamed: 5</th>\n",
       "      <th>Unnamed: 7</th>\n",
       "      <th>Unnamed: 8</th>\n",
       "      <th>Unnamed: 9</th>\n",
       "      <th>Unnamed: 10</th>\n",
       "      <th>Unnamed: 11</th>\n",
       "      <th>Unnamed: 12</th>\n",
       "      <th>Unnamed: 13</th>\n",
       "      <th>Unnamed: 14</th>\n",
       "      <th>Unnamed: 15</th>\n",
       "      <th>Unnamed: 16</th>\n",
       "      <th>Unnamed: 17</th>\n",
       "      <th>Unnamed: 18</th>\n",
       "      <th>Unnamed: 19</th>\n",
       "      <th>Unnamed: 20</th>\n",
       "      <th>Unnamed: 21</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Probability at birth of not surviving to age 4...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MDG</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MDG</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adult illiteracy rateb,†\\n(% aged 15 and older...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Population not using an improved water source†...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Children under weight for age†\\n(% under age 5...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Population below\\n income poverty line\\n(%)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HPI-1 rank minus income poverty rankc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Human poverty index                         (H...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>HDI rank</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>$1 a day \\n1990-2005d</td>\n",
       "      <td>NaN</td>\n",
       "      <td>$2 a day \\n1990-2005d</td>\n",
       "      <td>NaN</td>\n",
       "      <td>National poverty line           1990-2004d</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Rank</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Value \\n (%)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>HIGH HUMAN DEVELOPMENT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Unnamed: 0 Unnamed: 1  \\\n",
       "0                     NaN        NaN   \n",
       "1                     NaN        NaN   \n",
       "2                     NaN        NaN   \n",
       "3                HDI rank        NaN   \n",
       "4                     NaN        NaN   \n",
       "5  HIGH HUMAN DEVELOPMENT        NaN   \n",
       "\n",
       "                                          Unnamed: 3 Unnamed: 4    Unnamed: 5  \\\n",
       "0                                                NaN        NaN           NaN   \n",
       "1                                                NaN        NaN           NaN   \n",
       "2  Human poverty index                         (H...        NaN           NaN   \n",
       "3                                                NaN        NaN           NaN   \n",
       "4                                               Rank        NaN  Value \\n (%)   \n",
       "5                                                NaN        NaN           NaN   \n",
       "\n",
       "                                          Unnamed: 7 Unnamed: 8  \\\n",
       "0  Probability at birth of not surviving to age 4...        NaN   \n",
       "1                                                NaN        NaN   \n",
       "2                                                NaN        NaN   \n",
       "3                                                NaN        NaN   \n",
       "4                                                NaN        NaN   \n",
       "5                                                NaN        NaN   \n",
       "\n",
       "                                          Unnamed: 9 Unnamed: 10  \\\n",
       "0                                                NaN         NaN   \n",
       "1  Adult illiteracy rateb,†\\n(% aged 15 and older...         NaN   \n",
       "2                                                NaN         NaN   \n",
       "3                                                NaN         NaN   \n",
       "4                                                NaN         NaN   \n",
       "5                                                NaN         NaN   \n",
       "\n",
       "                                         Unnamed: 11 Unnamed: 12  \\\n",
       "0                                                NaN         NaN   \n",
       "1  Population not using an improved water source†...         NaN   \n",
       "2                                                NaN         NaN   \n",
       "3                                                NaN         NaN   \n",
       "4                                                NaN         NaN   \n",
       "5                                                NaN         NaN   \n",
       "\n",
       "                                         Unnamed: 13 Unnamed: 14  \\\n",
       "0                                                MDG         NaN   \n",
       "1  Children under weight for age†\\n(% under age 5...         NaN   \n",
       "2                                                NaN         NaN   \n",
       "3                                                NaN         NaN   \n",
       "4                                                NaN         NaN   \n",
       "5                                                NaN         NaN   \n",
       "\n",
       "                                   Unnamed: 15 Unnamed: 16  \\\n",
       "0                                          MDG         NaN   \n",
       "1  Population below\\n income poverty line\\n(%)         NaN   \n",
       "2                                          NaN         NaN   \n",
       "3                        $1 a day \\n1990-2005d         NaN   \n",
       "4                                          NaN         NaN   \n",
       "5                                          NaN         NaN   \n",
       "\n",
       "              Unnamed: 17 Unnamed: 18  \\\n",
       "0                     NaN         NaN   \n",
       "1                     NaN         NaN   \n",
       "2                     NaN         NaN   \n",
       "3  $2 a day \\n1990-2005d          NaN   \n",
       "4                     NaN         NaN   \n",
       "5                     NaN         NaN   \n",
       "\n",
       "                                  Unnamed: 19 Unnamed: 20  \\\n",
       "0                                         NaN         NaN   \n",
       "1                                         NaN         NaN   \n",
       "2                                         NaN         NaN   \n",
       "3  National poverty line           1990-2004d         NaN   \n",
       "4                                         NaN         NaN   \n",
       "5                                         NaN         NaN   \n",
       "\n",
       "                             Unnamed: 21  \n",
       "0                                    NaN  \n",
       "1  HPI-1 rank minus income poverty rankc  \n",
       "2                                    NaN  \n",
       "3                                    NaN  \n",
       "4                                    NaN  \n",
       "5                                    NaN  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Repositionner la ligne d'en-tête\n",
    "df = pd.read_excel(source, header=7)\n",
    "# https://stackoverflow.com/a/51794989\n",
    "# Supprimez d'abord toutes les colonnes (axe=1) sans valeurs\n",
    "df = df.dropna(how=\"all\", axis=1)\n",
    "# Supprimer toutes les lignes (axe=0) sans valeurs\n",
    "df = df.dropna(how=\"all\", axis=0).reset_index(drop=True)\n",
    "df.head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous pourrions simplement fixer les colonnes, mais obtenons une liste des informations qui sont déjà là et utilisons cela pour guider les nouveaux noms que nous devons inclure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['HDI rank'],\n",
       " [],\n",
       " ['Human poverty index                         (HPI-1)', 'Rank'],\n",
       " [],\n",
       " ['Value \\n (%)'],\n",
       " ['Probability at birth of not surviving to age 40a,†\\n(% of cohort)\\n2000-05'],\n",
       " [],\n",
       " ['Adult illiteracy rateb,†\\n(% aged 15 and older)\\n1995-2005 '],\n",
       " [],\n",
       " ['Population not using an improved water source†\\n(%)\\n2004'],\n",
       " [],\n",
       " ['MDG', 'Children under weight for age†\\n(% under age 5)\\n1996-2005d '],\n",
       " [],\n",
       " ['MDG',\n",
       "  'Population below\\n income poverty line\\n(%)',\n",
       "  '$1 a day \\n1990-2005d'],\n",
       " [],\n",
       " ['$2 a day \\n1990-2005d '],\n",
       " [],\n",
       " ['National poverty line           1990-2004d'],\n",
       " [],\n",
       " ['HPI-1 rank minus income poverty rankc']]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = []\n",
    "for c in df.columns:\n",
    "    # Pour chaque colonne, obtenir les valeurs des 5 premières lignes\n",
    "    # puis conservez toutes les chaînes (c'est-à-dire excluez-les si elles sont nulles) et ajoutez-les en colonne\n",
    "    columns.append([str(v) for v in df[c][:5] if pd.notnull(v)])\n",
    "columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Il y a quelques colonnes qui semblent ne pas avoir de noms. Mais ces colonnes n'ont pas été supprimées lorsque nous avons supprimé les colonnes et les lignes vides, il y a donc des informations dedans. En cas de doute, ouvrez le fichier source et jetez un coup d'œil. Vous constaterez alors que la première colonne \"vide\" contient en fait les noms des pays, et que les autres colonnes vides contiennent les références individuelles. Cela nous permet d'obtenir une liste de nouveaux noms de colonnes :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>HDI rank</th>\n",
       "      <th>Country</th>\n",
       "      <th>Human poverty index (HPI-1) - Rank</th>\n",
       "      <th>Reference 1</th>\n",
       "      <th>Human poverty index (HPI-1) - Value (%)</th>\n",
       "      <th>Probability at birth of not surviving to age 40 (% of cohort) 2000-05</th>\n",
       "      <th>Reference 2</th>\n",
       "      <th>Adult illiteracy rate (% aged 15 and older) 1995-2005</th>\n",
       "      <th>Reference 3</th>\n",
       "      <th>Population not using an improved water source (%) 2004</th>\n",
       "      <th>Reference 4</th>\n",
       "      <th>Children under weight for age (% under age 5) 1996-2005</th>\n",
       "      <th>Reference 5</th>\n",
       "      <th>Population below income poverty line (%) - $1 a day 1990-2005</th>\n",
       "      <th>Reference 6</th>\n",
       "      <th>Population below income poverty line (%) - $2 a day 1990-2005</th>\n",
       "      <th>Reference 7</th>\n",
       "      <th>Population below income poverty line (%) - National poverty line 1990-2004</th>\n",
       "      <th>Reference 8</th>\n",
       "      <th>HPI-1 rank minus income poverty rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>HIGH HUMAN DEVELOPMENT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21</td>\n",
       "      <td>Hong Kong, China (SAR)</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>1.5</td>\n",
       "      <td>e</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>25</td>\n",
       "      <td>Singapore</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.2</td>\n",
       "      <td>1.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>26</td>\n",
       "      <td>Korea (Republic of)</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>2.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Cyprus</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>2.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 HDI rank                 Country  \\\n",
       "0  HIGH HUMAN DEVELOPMENT                     NaN   \n",
       "1                      21  Hong Kong, China (SAR)   \n",
       "2                      25               Singapore   \n",
       "3                      26     Korea (Republic of)   \n",
       "4                      28                  Cyprus   \n",
       "\n",
       "  Human poverty index (HPI-1) - Rank Reference 1  \\\n",
       "0                                NaN         NaN   \n",
       "1                                 ..         NaN   \n",
       "2                                  7         NaN   \n",
       "3                                 ..         NaN   \n",
       "4                                 ..         NaN   \n",
       "\n",
       "  Human poverty index (HPI-1) - Value (%)  \\\n",
       "0                                     NaN   \n",
       "1                                      ..   \n",
       "2                                     5.2   \n",
       "3                                      ..   \n",
       "4                                      ..   \n",
       "\n",
       "  Probability at birth of not surviving to age 40 (% of cohort) 2000-05  \\\n",
       "0                                                NaN                      \n",
       "1                                                1.5                      \n",
       "2                                                1.8                      \n",
       "3                                                2.5                      \n",
       "4                                                2.4                      \n",
       "\n",
       "  Reference 2 Adult illiteracy rate (% aged 15 and older) 1995-2005  \\\n",
       "0         NaN                                                NaN      \n",
       "1           e                                                 ..      \n",
       "2         NaN                                                7.5      \n",
       "3         NaN                                                1.0      \n",
       "4         NaN                                                3.2      \n",
       "\n",
       "  Reference 3 Population not using an improved water source (%) 2004  \\\n",
       "0         NaN                                                NaN       \n",
       "1         NaN                                                 ..       \n",
       "2         NaN                                                  0       \n",
       "3         NaN                                                  8       \n",
       "4         NaN                                                  0       \n",
       "\n",
       "  Reference 4 Children under weight for age (% under age 5) 1996-2005  \\\n",
       "0         NaN                                                NaN        \n",
       "1         NaN                                                 ..        \n",
       "2         NaN                                                  3        \n",
       "3         NaN                                                 ..        \n",
       "4         NaN                                                 ..        \n",
       "\n",
       "  Reference 5 Population below income poverty line (%) - $1 a day 1990-2005  \\\n",
       "0         NaN                                                NaN              \n",
       "1         NaN                                                 ..              \n",
       "2         NaN                                                 ..              \n",
       "3         NaN                                                 <2              \n",
       "4         NaN                                                 ..              \n",
       "\n",
       "  Reference 6 Population below income poverty line (%) - $2 a day 1990-2005  \\\n",
       "0         NaN                                                NaN              \n",
       "1         NaN                                                 ..              \n",
       "2         NaN                                                 ..              \n",
       "3         NaN                                                 <2              \n",
       "4         NaN                                                 ..              \n",
       "\n",
       "  Reference 7  \\\n",
       "0         NaN   \n",
       "1         NaN   \n",
       "2         NaN   \n",
       "3         NaN   \n",
       "4         NaN   \n",
       "\n",
       "  Population below income poverty line (%) - National poverty line 1990-2004  \\\n",
       "0                                                NaN                           \n",
       "1                                                 ..                           \n",
       "2                                                 ..                           \n",
       "3                                                 ..                           \n",
       "4                                                 ..                           \n",
       "\n",
       "  Reference 8 HPI-1 rank minus income poverty rank  \n",
       "0         NaN                                  NaN  \n",
       "1         NaN                                   ..  \n",
       "2         NaN                                   ..  \n",
       "3         NaN                                   ..  \n",
       "4         NaN                                   ..  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = [\n",
    "    \"HDI rank\",\n",
    "    \"Country\",\n",
    "    \"Human poverty index (HPI-1) - Rank\",\n",
    "    \"Reference 1\",\n",
    "    \"Human poverty index (HPI-1) - Value (%)\",\n",
    "    \"Probability at birth of not surviving to age 40 (% of cohort) 2000-05\",\n",
    "    \"Reference 2\",\n",
    "    \"Adult illiteracy rate (% aged 15 and older) 1995-2005\",\n",
    "    \"Reference 3\",\n",
    "    \"Population not using an improved water source (%) 2004\",\n",
    "    \"Reference 4\",\n",
    "    \"Children under weight for age (% under age 5) 1996-2005\",\n",
    "    \"Reference 5\",\n",
    "    \"Population below income poverty line (%) - $1 a day 1990-2005\",\n",
    "    \"Reference 6\",\n",
    "    \"Population below income poverty line (%) - $2 a day 1990-2005\",   \n",
    "    \"Reference 7\",\n",
    "    \"Population below income poverty line (%) - National poverty line 1990-2004\",   \n",
    "    \"Reference 8\",\n",
    "    \"HPI-1 rank minus income poverty rank\"\n",
    "]\n",
    "# Réinitialiser les colonnes des cadres de données\n",
    "df.columns = columns\n",
    "# Supprimer les lignes qui contenaient les informations dont nous n'avons plus besoin\n",
    "# et réinitialiser l'index\n",
    "# np.arange(5) crée un tableau = [0, 1, 2, 3, 4, 5]\n",
    "df = df.drop(np.arange(5), axis=0).reset_index(drop=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ce fichier n'est pas accompagné d'un fichier de métadonnées distinct. Toutes les références et notes de bas de page se trouvent littéralement au bas du fichier. Jetez un coup d'œil :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>HDI rank</th>\n",
       "      <th>Country</th>\n",
       "      <th>Human poverty index (HPI-1) - Rank</th>\n",
       "      <th>Reference 1</th>\n",
       "      <th>Human poverty index (HPI-1) - Value (%)</th>\n",
       "      <th>Probability at birth of not surviving to age 40 (% of cohort) 2000-05</th>\n",
       "      <th>Reference 2</th>\n",
       "      <th>Adult illiteracy rate (% aged 15 and older) 1995-2005</th>\n",
       "      <th>Reference 3</th>\n",
       "      <th>Population not using an improved water source (%) 2004</th>\n",
       "      <th>Reference 4</th>\n",
       "      <th>Children under weight for age (% under age 5) 1996-2005</th>\n",
       "      <th>Reference 5</th>\n",
       "      <th>Population below income poverty line (%) - $1 a day 1990-2005</th>\n",
       "      <th>Reference 6</th>\n",
       "      <th>Population below income poverty line (%) - $2 a day 1990-2005</th>\n",
       "      <th>Reference 7</th>\n",
       "      <th>Population below income poverty line (%) - National poverty line 1990-2004</th>\n",
       "      <th>Reference 8</th>\n",
       "      <th>HPI-1 rank minus income poverty rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>176</td>\n",
       "      <td>Burkina Faso</td>\n",
       "      <td>106</td>\n",
       "      <td>NaN</td>\n",
       "      <td>55.8</td>\n",
       "      <td>26.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>76.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>39</td>\n",
       "      <td>NaN</td>\n",
       "      <td>38</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>71.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>46.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>177</td>\n",
       "      <td>Sierra Leone</td>\n",
       "      <td>102</td>\n",
       "      <td>NaN</td>\n",
       "      <td>51.7</td>\n",
       "      <td>45.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>65.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>43</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27</td>\n",
       "      <td>NaN</td>\n",
       "      <td>57.0</td>\n",
       "      <td>e</td>\n",
       "      <td>74.5</td>\n",
       "      <td>e</td>\n",
       "      <td>70.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>NOTES</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>† Denotes indicators used to calculate the hum...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>a. Data refer to the probability at birth of n...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>SOURCES\\nColumn 1: Determined on the basis of ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133</th>\n",
       "      <td>HPI-1 ranks for 108 developing countries and a...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>1</td>\n",
       "      <td>Barbados</td>\n",
       "      <td>22</td>\n",
       "      <td>Turkey</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>45</td>\n",
       "      <td>Tunisia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>68</td>\n",
       "      <td>Morocco</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>89</td>\n",
       "      <td>Angola</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>2</td>\n",
       "      <td>Uruguay</td>\n",
       "      <td>23</td>\n",
       "      <td>Brazil</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>46</td>\n",
       "      <td>Nicaragua</td>\n",
       "      <td>NaN</td>\n",
       "      <td>69</td>\n",
       "      <td>Sudan</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>90</td>\n",
       "      <td>Papua New Guinea</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>3</td>\n",
       "      <td>Chile</td>\n",
       "      <td>24</td>\n",
       "      <td>Thailand</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>47</td>\n",
       "      <td>Indonesia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>70</td>\n",
       "      <td>Lao People's Democratic Republic</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>91</td>\n",
       "      <td>Zimbabwe</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>4</td>\n",
       "      <td>Argentina</td>\n",
       "      <td>25</td>\n",
       "      <td>Suriname</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48</td>\n",
       "      <td>Egypt</td>\n",
       "      <td>NaN</td>\n",
       "      <td>71</td>\n",
       "      <td>Lesotho</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>Côte d'Ivoire</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>5</td>\n",
       "      <td>Costa Rica</td>\n",
       "      <td>26</td>\n",
       "      <td>Dominican Republic</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49</td>\n",
       "      <td>Gabon</td>\n",
       "      <td>NaN</td>\n",
       "      <td>72</td>\n",
       "      <td>Uganda</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>93</td>\n",
       "      <td>Bangladesh</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>6</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>27</td>\n",
       "      <td>Mauritius</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>50</td>\n",
       "      <td>Fiji</td>\n",
       "      <td>NaN</td>\n",
       "      <td>73</td>\n",
       "      <td>Swaziland</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>94</td>\n",
       "      <td>Gambia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>7</td>\n",
       "      <td>Singapore</td>\n",
       "      <td>28</td>\n",
       "      <td>Peru</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>51</td>\n",
       "      <td>Algeria</td>\n",
       "      <td>NaN</td>\n",
       "      <td>74</td>\n",
       "      <td>Haiti</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>95</td>\n",
       "      <td>Timor-Leste</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>8</td>\n",
       "      <td>Saint Lucia</td>\n",
       "      <td>29</td>\n",
       "      <td>China</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>52</td>\n",
       "      <td>Myanmar</td>\n",
       "      <td>NaN</td>\n",
       "      <td>75</td>\n",
       "      <td>Madagascar</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>96</td>\n",
       "      <td>Zambia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>9</td>\n",
       "      <td>Occupied Palestinian Territories</td>\n",
       "      <td>30</td>\n",
       "      <td>Iran (Islamic Republic of)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>53</td>\n",
       "      <td>Solomon Islands</td>\n",
       "      <td>NaN</td>\n",
       "      <td>76</td>\n",
       "      <td>Eritrea</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>97</td>\n",
       "      <td>Senegal</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>10</td>\n",
       "      <td>Mexico</td>\n",
       "      <td>31</td>\n",
       "      <td>Syrian Arab Republic</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>54</td>\n",
       "      <td>Guatemala</td>\n",
       "      <td>NaN</td>\n",
       "      <td>77</td>\n",
       "      <td>Pakistan</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>98</td>\n",
       "      <td>Central African Republic</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>11</td>\n",
       "      <td>Jordan</td>\n",
       "      <td>32</td>\n",
       "      <td>Bolivia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>55</td>\n",
       "      <td>South Africa</td>\n",
       "      <td>NaN</td>\n",
       "      <td>78</td>\n",
       "      <td>Rwanda</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>99</td>\n",
       "      <td>Guinea-Bissau</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>12</td>\n",
       "      <td>Trinidad and Tobago</td>\n",
       "      <td>33</td>\n",
       "      <td>Guyana</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>56</td>\n",
       "      <td>Vanuatu</td>\n",
       "      <td>NaN</td>\n",
       "      <td>79</td>\n",
       "      <td>Malawi</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100</td>\n",
       "      <td>Benin</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>13</td>\n",
       "      <td>Qatar</td>\n",
       "      <td>34</td>\n",
       "      <td>Jamaica</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>57</td>\n",
       "      <td>Congo</td>\n",
       "      <td>NaN</td>\n",
       "      <td>80</td>\n",
       "      <td>Nigeria</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>101</td>\n",
       "      <td>Mozambique</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>14</td>\n",
       "      <td>Colombia</td>\n",
       "      <td>35</td>\n",
       "      <td>El Salvador</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>58</td>\n",
       "      <td>Namibia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>81</td>\n",
       "      <td>Burundi</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>102</td>\n",
       "      <td>Sierra Leone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>15</td>\n",
       "      <td>Panama</td>\n",
       "      <td>36</td>\n",
       "      <td>Viet Nam</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>59</td>\n",
       "      <td>Djibouti</td>\n",
       "      <td>NaN</td>\n",
       "      <td>82</td>\n",
       "      <td>Yemen</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>103</td>\n",
       "      <td>Guinea</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>16</td>\n",
       "      <td>Malaysia</td>\n",
       "      <td>37</td>\n",
       "      <td>Philippines</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60</td>\n",
       "      <td>Kenya</td>\n",
       "      <td>NaN</td>\n",
       "      <td>83</td>\n",
       "      <td>Togo</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>104</td>\n",
       "      <td>Niger</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>17</td>\n",
       "      <td>United Arab Emirates</td>\n",
       "      <td>38</td>\n",
       "      <td>Cape Verde</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>61</td>\n",
       "      <td>Comoros</td>\n",
       "      <td>NaN</td>\n",
       "      <td>84</td>\n",
       "      <td>Nepal</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>105</td>\n",
       "      <td>Ethiopia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>151</th>\n",
       "      <td>18</td>\n",
       "      <td>Lebanon</td>\n",
       "      <td>39</td>\n",
       "      <td>Sao Tome and Principe</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>62</td>\n",
       "      <td>India</td>\n",
       "      <td>NaN</td>\n",
       "      <td>85</td>\n",
       "      <td>Cambodia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>106</td>\n",
       "      <td>Burkina Faso</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>19</td>\n",
       "      <td>Ecuador</td>\n",
       "      <td>40</td>\n",
       "      <td>Mongolia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>63</td>\n",
       "      <td>Botswana</td>\n",
       "      <td>NaN</td>\n",
       "      <td>86</td>\n",
       "      <td>Bhutan</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>107</td>\n",
       "      <td>Mali</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153</th>\n",
       "      <td>20</td>\n",
       "      <td>Paraguay</td>\n",
       "      <td>41</td>\n",
       "      <td>Honduras</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>64</td>\n",
       "      <td>Cameroon</td>\n",
       "      <td>NaN</td>\n",
       "      <td>87</td>\n",
       "      <td>Mauritania</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>108</td>\n",
       "      <td>Chad</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>154</th>\n",
       "      <td>21</td>\n",
       "      <td>Venezuela (Bolivarian Republic of)</td>\n",
       "      <td>42</td>\n",
       "      <td>Maldives</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>65</td>\n",
       "      <td>Ghana</td>\n",
       "      <td>NaN</td>\n",
       "      <td>88</td>\n",
       "      <td>Congo (Democratic Republic of the)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>155</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>43</td>\n",
       "      <td>Belize</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>66</td>\n",
       "      <td>Equatorial Guinea</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>44</td>\n",
       "      <td>Sri Lanka</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>67</td>\n",
       "      <td>Tanzania (United Republic of)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              HDI rank  \\\n",
       "127                                                176   \n",
       "128                                                177   \n",
       "129                                              NOTES   \n",
       "130  † Denotes indicators used to calculate the hum...   \n",
       "131  a. Data refer to the probability at birth of n...   \n",
       "132  SOURCES\\nColumn 1: Determined on the basis of ...   \n",
       "133  HPI-1 ranks for 108 developing countries and a...   \n",
       "134                                                  1   \n",
       "135                                                  2   \n",
       "136                                                  3   \n",
       "137                                                  4   \n",
       "138                                                  5   \n",
       "139                                                  6   \n",
       "140                                                  7   \n",
       "141                                                  8   \n",
       "142                                                  9   \n",
       "143                                                 10   \n",
       "144                                                 11   \n",
       "145                                                 12   \n",
       "146                                                 13   \n",
       "147                                                 14   \n",
       "148                                                 15   \n",
       "149                                                 16   \n",
       "150                                                 17   \n",
       "151                                                 18   \n",
       "152                                                 19   \n",
       "153                                                 20   \n",
       "154                                                 21   \n",
       "155                                                NaN   \n",
       "156                                                NaN   \n",
       "\n",
       "                                Country Human poverty index (HPI-1) - Rank  \\\n",
       "127                        Burkina Faso                                106   \n",
       "128                        Sierra Leone                                102   \n",
       "129                                 NaN                                NaN   \n",
       "130                                 NaN                                NaN   \n",
       "131                                 NaN                                NaN   \n",
       "132                                 NaN                                NaN   \n",
       "133                                 NaN                                NaN   \n",
       "134                            Barbados                                 22   \n",
       "135                             Uruguay                                 23   \n",
       "136                               Chile                                 24   \n",
       "137                           Argentina                                 25   \n",
       "138                          Costa Rica                                 26   \n",
       "139                                Cuba                                 27   \n",
       "140                           Singapore                                 28   \n",
       "141                         Saint Lucia                                 29   \n",
       "142    Occupied Palestinian Territories                                 30   \n",
       "143                              Mexico                                 31   \n",
       "144                              Jordan                                 32   \n",
       "145                 Trinidad and Tobago                                 33   \n",
       "146                               Qatar                                 34   \n",
       "147                            Colombia                                 35   \n",
       "148                              Panama                                 36   \n",
       "149                            Malaysia                                 37   \n",
       "150                United Arab Emirates                                 38   \n",
       "151                             Lebanon                                 39   \n",
       "152                             Ecuador                                 40   \n",
       "153                            Paraguay                                 41   \n",
       "154  Venezuela (Bolivarian Republic of)                                 42   \n",
       "155                                 NaN                                 43   \n",
       "156                                 NaN                                 44   \n",
       "\n",
       "                    Reference 1 Human poverty index (HPI-1) - Value (%)  \\\n",
       "127                         NaN                                    55.8   \n",
       "128                         NaN                                    51.7   \n",
       "129                         NaN                                     NaN   \n",
       "130                         NaN                                     NaN   \n",
       "131                         NaN                                     NaN   \n",
       "132                         NaN                                     NaN   \n",
       "133                         NaN                                     NaN   \n",
       "134                      Turkey                                     NaN   \n",
       "135                      Brazil                                     NaN   \n",
       "136                    Thailand                                     NaN   \n",
       "137                    Suriname                                     NaN   \n",
       "138          Dominican Republic                                     NaN   \n",
       "139                   Mauritius                                     NaN   \n",
       "140                        Peru                                     NaN   \n",
       "141                       China                                     NaN   \n",
       "142  Iran (Islamic Republic of)                                     NaN   \n",
       "143        Syrian Arab Republic                                     NaN   \n",
       "144                     Bolivia                                     NaN   \n",
       "145                      Guyana                                     NaN   \n",
       "146                     Jamaica                                     NaN   \n",
       "147                 El Salvador                                     NaN   \n",
       "148                    Viet Nam                                     NaN   \n",
       "149                 Philippines                                     NaN   \n",
       "150                  Cape Verde                                     NaN   \n",
       "151       Sao Tome and Principe                                     NaN   \n",
       "152                    Mongolia                                     NaN   \n",
       "153                    Honduras                                     NaN   \n",
       "154                    Maldives                                     NaN   \n",
       "155                      Belize                                     NaN   \n",
       "156                   Sri Lanka                                     NaN   \n",
       "\n",
       "    Probability at birth of not surviving to age 40 (% of cohort) 2000-05  \\\n",
       "127                                               26.5                      \n",
       "128                                               45.6                      \n",
       "129                                                NaN                      \n",
       "130                                                NaN                      \n",
       "131                                                NaN                      \n",
       "132                                                NaN                      \n",
       "133                                                NaN                      \n",
       "134                                                NaN                      \n",
       "135                                                NaN                      \n",
       "136                                                NaN                      \n",
       "137                                                NaN                      \n",
       "138                                                NaN                      \n",
       "139                                                NaN                      \n",
       "140                                                NaN                      \n",
       "141                                                NaN                      \n",
       "142                                                NaN                      \n",
       "143                                                NaN                      \n",
       "144                                                NaN                      \n",
       "145                                                NaN                      \n",
       "146                                                NaN                      \n",
       "147                                                NaN                      \n",
       "148                                                NaN                      \n",
       "149                                                NaN                      \n",
       "150                                                NaN                      \n",
       "151                                                NaN                      \n",
       "152                                                NaN                      \n",
       "153                                                NaN                      \n",
       "154                                                NaN                      \n",
       "155                                                NaN                      \n",
       "156                                                NaN                      \n",
       "\n",
       "    Reference 2 Adult illiteracy rate (% aged 15 and older) 1995-2005  \\\n",
       "127         NaN                                               76.4      \n",
       "128         NaN                                               65.2      \n",
       "129         NaN                                                NaN      \n",
       "130         NaN                                                NaN      \n",
       "131         NaN                                                NaN      \n",
       "132         NaN                                                NaN      \n",
       "133         NaN                                                NaN      \n",
       "134          45                                            Tunisia      \n",
       "135          46                                          Nicaragua      \n",
       "136          47                                          Indonesia      \n",
       "137          48                                              Egypt      \n",
       "138          49                                              Gabon      \n",
       "139          50                                               Fiji      \n",
       "140          51                                            Algeria      \n",
       "141          52                                            Myanmar      \n",
       "142          53                                    Solomon Islands      \n",
       "143          54                                          Guatemala      \n",
       "144          55                                       South Africa      \n",
       "145          56                                            Vanuatu      \n",
       "146          57                                              Congo      \n",
       "147          58                                            Namibia      \n",
       "148          59                                           Djibouti      \n",
       "149          60                                              Kenya      \n",
       "150          61                                            Comoros      \n",
       "151          62                                              India      \n",
       "152          63                                           Botswana      \n",
       "153          64                                           Cameroon      \n",
       "154          65                                              Ghana      \n",
       "155          66                                  Equatorial Guinea      \n",
       "156          67                      Tanzania (United Republic of)      \n",
       "\n",
       "    Reference 3 Population not using an improved water source (%) 2004  \\\n",
       "127         NaN                                                 39       \n",
       "128         NaN                                                 43       \n",
       "129         NaN                                                NaN       \n",
       "130         NaN                                                NaN       \n",
       "131         NaN                                                NaN       \n",
       "132         NaN                                                NaN       \n",
       "133         NaN                                                NaN       \n",
       "134         NaN                                                 68       \n",
       "135         NaN                                                 69       \n",
       "136         NaN                                                 70       \n",
       "137         NaN                                                 71       \n",
       "138         NaN                                                 72       \n",
       "139         NaN                                                 73       \n",
       "140         NaN                                                 74       \n",
       "141         NaN                                                 75       \n",
       "142         NaN                                                 76       \n",
       "143         NaN                                                 77       \n",
       "144         NaN                                                 78       \n",
       "145         NaN                                                 79       \n",
       "146         NaN                                                 80       \n",
       "147         NaN                                                 81       \n",
       "148         NaN                                                 82       \n",
       "149         NaN                                                 83       \n",
       "150         NaN                                                 84       \n",
       "151         NaN                                                 85       \n",
       "152         NaN                                                 86       \n",
       "153         NaN                                                 87       \n",
       "154         NaN                                                 88       \n",
       "155         NaN                                                NaN       \n",
       "156         NaN                                                NaN       \n",
       "\n",
       "                            Reference 4  \\\n",
       "127                                 NaN   \n",
       "128                                 NaN   \n",
       "129                                 NaN   \n",
       "130                                 NaN   \n",
       "131                                 NaN   \n",
       "132                                 NaN   \n",
       "133                                 NaN   \n",
       "134                             Morocco   \n",
       "135                               Sudan   \n",
       "136    Lao People's Democratic Republic   \n",
       "137                             Lesotho   \n",
       "138                              Uganda   \n",
       "139                           Swaziland   \n",
       "140                               Haiti   \n",
       "141                          Madagascar   \n",
       "142                             Eritrea   \n",
       "143                            Pakistan   \n",
       "144                              Rwanda   \n",
       "145                              Malawi   \n",
       "146                             Nigeria   \n",
       "147                             Burundi   \n",
       "148                               Yemen   \n",
       "149                                Togo   \n",
       "150                               Nepal   \n",
       "151                            Cambodia   \n",
       "152                              Bhutan   \n",
       "153                          Mauritania   \n",
       "154  Congo (Democratic Republic of the)   \n",
       "155                                 NaN   \n",
       "156                                 NaN   \n",
       "\n",
       "    Children under weight for age (% under age 5) 1996-2005 Reference 5  \\\n",
       "127                                                 38              NaN   \n",
       "128                                                 27              NaN   \n",
       "129                                                NaN              NaN   \n",
       "130                                                NaN              NaN   \n",
       "131                                                NaN              NaN   \n",
       "132                                                NaN              NaN   \n",
       "133                                                NaN              NaN   \n",
       "134                                                NaN              NaN   \n",
       "135                                                NaN              NaN   \n",
       "136                                                NaN              NaN   \n",
       "137                                                NaN              NaN   \n",
       "138                                                NaN              NaN   \n",
       "139                                                NaN              NaN   \n",
       "140                                                NaN              NaN   \n",
       "141                                                NaN              NaN   \n",
       "142                                                NaN              NaN   \n",
       "143                                                NaN              NaN   \n",
       "144                                                NaN              NaN   \n",
       "145                                                NaN              NaN   \n",
       "146                                                NaN              NaN   \n",
       "147                                                NaN              NaN   \n",
       "148                                                NaN              NaN   \n",
       "149                                                NaN              NaN   \n",
       "150                                                NaN              NaN   \n",
       "151                                                NaN              NaN   \n",
       "152                                                NaN              NaN   \n",
       "153                                                NaN              NaN   \n",
       "154                                                NaN              NaN   \n",
       "155                                                NaN              NaN   \n",
       "156                                                NaN              NaN   \n",
       "\n",
       "    Population below income poverty line (%) - $1 a day 1990-2005  \\\n",
       "127                                               27.2              \n",
       "128                                               57.0              \n",
       "129                                                NaN              \n",
       "130                                                NaN              \n",
       "131                                                NaN              \n",
       "132                                                NaN              \n",
       "133                                                NaN              \n",
       "134                                                 89              \n",
       "135                                                 90              \n",
       "136                                                 91              \n",
       "137                                                 92              \n",
       "138                                                 93              \n",
       "139                                                 94              \n",
       "140                                                 95              \n",
       "141                                                 96              \n",
       "142                                                 97              \n",
       "143                                                 98              \n",
       "144                                                 99              \n",
       "145                                                100              \n",
       "146                                                101              \n",
       "147                                                102              \n",
       "148                                                103              \n",
       "149                                                104              \n",
       "150                                                105              \n",
       "151                                                106              \n",
       "152                                                107              \n",
       "153                                                108              \n",
       "154                                                NaN              \n",
       "155                                                NaN              \n",
       "156                                                NaN              \n",
       "\n",
       "                  Reference 6  \\\n",
       "127                       NaN   \n",
       "128                         e   \n",
       "129                       NaN   \n",
       "130                       NaN   \n",
       "131                       NaN   \n",
       "132                       NaN   \n",
       "133                       NaN   \n",
       "134                    Angola   \n",
       "135          Papua New Guinea   \n",
       "136                  Zimbabwe   \n",
       "137             Côte d'Ivoire   \n",
       "138                Bangladesh   \n",
       "139                    Gambia   \n",
       "140               Timor-Leste   \n",
       "141                    Zambia   \n",
       "142                   Senegal   \n",
       "143  Central African Republic   \n",
       "144             Guinea-Bissau   \n",
       "145                     Benin   \n",
       "146                Mozambique   \n",
       "147              Sierra Leone   \n",
       "148                    Guinea   \n",
       "149                     Niger   \n",
       "150                  Ethiopia   \n",
       "151              Burkina Faso   \n",
       "152                      Mali   \n",
       "153                      Chad   \n",
       "154                       NaN   \n",
       "155                       NaN   \n",
       "156                       NaN   \n",
       "\n",
       "    Population below income poverty line (%) - $2 a day 1990-2005 Reference 7  \\\n",
       "127                                               71.8                    NaN   \n",
       "128                                               74.5                      e   \n",
       "129                                                NaN                    NaN   \n",
       "130                                                NaN                    NaN   \n",
       "131                                                NaN                    NaN   \n",
       "132                                                NaN                    NaN   \n",
       "133                                                NaN                    NaN   \n",
       "134                                                NaN                    NaN   \n",
       "135                                                NaN                    NaN   \n",
       "136                                                NaN                    NaN   \n",
       "137                                                NaN                    NaN   \n",
       "138                                                NaN                    NaN   \n",
       "139                                                NaN                    NaN   \n",
       "140                                                NaN                    NaN   \n",
       "141                                                NaN                    NaN   \n",
       "142                                                NaN                    NaN   \n",
       "143                                                NaN                    NaN   \n",
       "144                                                NaN                    NaN   \n",
       "145                                                NaN                    NaN   \n",
       "146                                                NaN                    NaN   \n",
       "147                                                NaN                    NaN   \n",
       "148                                                NaN                    NaN   \n",
       "149                                                NaN                    NaN   \n",
       "150                                                NaN                    NaN   \n",
       "151                                                NaN                    NaN   \n",
       "152                                                NaN                    NaN   \n",
       "153                                                NaN                    NaN   \n",
       "154                                                NaN                    NaN   \n",
       "155                                                NaN                    NaN   \n",
       "156                                                NaN                    NaN   \n",
       "\n",
       "    Population below income poverty line (%) - National poverty line 1990-2004  \\\n",
       "127                                               46.4                           \n",
       "128                                               70.2                           \n",
       "129                                                NaN                           \n",
       "130                                                NaN                           \n",
       "131                                                NaN                           \n",
       "132                                                NaN                           \n",
       "133                                                NaN                           \n",
       "134                                                NaN                           \n",
       "135                                                NaN                           \n",
       "136                                                NaN                           \n",
       "137                                                NaN                           \n",
       "138                                                NaN                           \n",
       "139                                                NaN                           \n",
       "140                                                NaN                           \n",
       "141                                                NaN                           \n",
       "142                                                NaN                           \n",
       "143                                                NaN                           \n",
       "144                                                NaN                           \n",
       "145                                                NaN                           \n",
       "146                                                NaN                           \n",
       "147                                                NaN                           \n",
       "148                                                NaN                           \n",
       "149                                                NaN                           \n",
       "150                                                NaN                           \n",
       "151                                                NaN                           \n",
       "152                                                NaN                           \n",
       "153                                                NaN                           \n",
       "154                                                NaN                           \n",
       "155                                                NaN                           \n",
       "156                                                NaN                           \n",
       "\n",
       "    Reference 8 HPI-1 rank minus income poverty rank  \n",
       "127         NaN                                   23  \n",
       "128         NaN                                    4  \n",
       "129         NaN                                  NaN  \n",
       "130         NaN                                  NaN  \n",
       "131         NaN                                  NaN  \n",
       "132         NaN                                  NaN  \n",
       "133         NaN                                  NaN  \n",
       "134         NaN                                  NaN  \n",
       "135         NaN                                  NaN  \n",
       "136         NaN                                  NaN  \n",
       "137         NaN                                  NaN  \n",
       "138         NaN                                  NaN  \n",
       "139         NaN                                  NaN  \n",
       "140         NaN                                  NaN  \n",
       "141         NaN                                  NaN  \n",
       "142         NaN                                  NaN  \n",
       "143         NaN                                  NaN  \n",
       "144         NaN                                  NaN  \n",
       "145         NaN                                  NaN  \n",
       "146         NaN                                  NaN  \n",
       "147         NaN                                  NaN  \n",
       "148         NaN                                  NaN  \n",
       "149         NaN                                  NaN  \n",
       "150         NaN                                  NaN  \n",
       "151         NaN                                  NaN  \n",
       "152         NaN                                  NaN  \n",
       "153         NaN                                  NaN  \n",
       "154         NaN                                  NaN  \n",
       "155         NaN                                  NaN  \n",
       "156         NaN                                  NaN  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous devons supprimer cette information du cadre de données ___mais___ nous ne pouvons pas la perdre. Vous devez également créer un fichier de métadonnées contenant ces informations à conserver avec votre fichier de données restructuré, de cette façon vous avez toutes les indications dont un utilisateur peut avoir besoin, ainsi que des données lisibles par machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Déposer les lignes du bas des données que nous voulons conserver, jusqu'à la fin (défini comme len(df))\n",
    "df = df.drop(np.arange(129,len(df)), axis=0).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous avons également un autre problème. Il existe trois catégories d'indice de développement humain, définies arbitrairement comme suit:\n",
    "\n",
    "- `LOW HUMAN DEVELOPMENT` pour `HDI ranks > 155`\n",
    "- `MEDIUM HUMAN DEVELOPMENT` pour `HDI ranks > 70`\n",
    "- `HIGH HUMAN DEVELOPMENT` pour tout le reste\n",
    "\n",
    "Nous devons supprimer ces lignes (car elles gênent les données de base) et les réaffecter à des colonnes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>HDI rank</th>\n",
       "      <th>Country</th>\n",
       "      <th>Human poverty index (HPI-1) - Rank</th>\n",
       "      <th>Reference 1</th>\n",
       "      <th>Human poverty index (HPI-1) - Value (%)</th>\n",
       "      <th>Probability at birth of not surviving to age 40 (% of cohort) 2000-05</th>\n",
       "      <th>Reference 2</th>\n",
       "      <th>Adult illiteracy rate (% aged 15 and older) 1995-2005</th>\n",
       "      <th>Reference 3</th>\n",
       "      <th>Population not using an improved water source (%) 2004</th>\n",
       "      <th>...</th>\n",
       "      <th>Children under weight for age (% under age 5) 1996-2005</th>\n",
       "      <th>Reference 5</th>\n",
       "      <th>Population below income poverty line (%) - $1 a day 1990-2005</th>\n",
       "      <th>Reference 6</th>\n",
       "      <th>Population below income poverty line (%) - $2 a day 1990-2005</th>\n",
       "      <th>Reference 7</th>\n",
       "      <th>Population below income poverty line (%) - National poverty line 1990-2004</th>\n",
       "      <th>Reference 8</th>\n",
       "      <th>HPI-1 rank minus income poverty rank</th>\n",
       "      <th>HDI category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21</td>\n",
       "      <td>Hong Kong, China (SAR)</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>1.5</td>\n",
       "      <td>e</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>...</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>LOW HUMAN DEVELOPMENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25</td>\n",
       "      <td>Singapore</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.2</td>\n",
       "      <td>1.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>LOW HUMAN DEVELOPMENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>26</td>\n",
       "      <td>Korea (Republic of)</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>2.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>LOW HUMAN DEVELOPMENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>28</td>\n",
       "      <td>Cyprus</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>2.4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>LOW HUMAN DEVELOPMENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30</td>\n",
       "      <td>Brunei Darussalam</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>...</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>NaN</td>\n",
       "      <td>..</td>\n",
       "      <td>LOW HUMAN DEVELOPMENT</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  HDI rank                 Country Human poverty index (HPI-1) - Rank  \\\n",
       "0       21  Hong Kong, China (SAR)                                 ..   \n",
       "1       25               Singapore                                  7   \n",
       "2       26     Korea (Republic of)                                 ..   \n",
       "3       28                  Cyprus                                 ..   \n",
       "4       30       Brunei Darussalam                                 ..   \n",
       "\n",
       "  Reference 1 Human poverty index (HPI-1) - Value (%)  \\\n",
       "0         NaN                                      ..   \n",
       "1         NaN                                     5.2   \n",
       "2         NaN                                      ..   \n",
       "3         NaN                                      ..   \n",
       "4         NaN                                      ..   \n",
       "\n",
       "  Probability at birth of not surviving to age 40 (% of cohort) 2000-05  \\\n",
       "0                                                1.5                      \n",
       "1                                                1.8                      \n",
       "2                                                2.5                      \n",
       "3                                                2.4                      \n",
       "4                                                3.0                      \n",
       "\n",
       "  Reference 2 Adult illiteracy rate (% aged 15 and older) 1995-2005  \\\n",
       "0           e                                                 ..      \n",
       "1         NaN                                                7.5      \n",
       "2         NaN                                                1.0      \n",
       "3         NaN                                                3.2      \n",
       "4         NaN                                                7.3      \n",
       "\n",
       "  Reference 3 Population not using an improved water source (%) 2004  ...  \\\n",
       "0         NaN                                                 ..      ...   \n",
       "1         NaN                                                  0      ...   \n",
       "2         NaN                                                  8      ...   \n",
       "3         NaN                                                  0      ...   \n",
       "4         NaN                                                 ..      ...   \n",
       "\n",
       "  Children under weight for age (% under age 5) 1996-2005 Reference 5  \\\n",
       "0                                                 ..              NaN   \n",
       "1                                                  3              NaN   \n",
       "2                                                 ..              NaN   \n",
       "3                                                 ..              NaN   \n",
       "4                                                 ..              NaN   \n",
       "\n",
       "  Population below income poverty line (%) - $1 a day 1990-2005 Reference 6  \\\n",
       "0                                                 ..                    NaN   \n",
       "1                                                 ..                    NaN   \n",
       "2                                                 <2                    NaN   \n",
       "3                                                 ..                    NaN   \n",
       "4                                                 ..                    NaN   \n",
       "\n",
       "  Population below income poverty line (%) - $2 a day 1990-2005 Reference 7  \\\n",
       "0                                                 ..                    NaN   \n",
       "1                                                 ..                    NaN   \n",
       "2                                                 <2                    NaN   \n",
       "3                                                 ..                    NaN   \n",
       "4                                                 ..                    NaN   \n",
       "\n",
       "  Population below income poverty line (%) - National poverty line 1990-2004  \\\n",
       "0                                                 ..                           \n",
       "1                                                 ..                           \n",
       "2                                                 ..                           \n",
       "3                                                 ..                           \n",
       "4                                                 ..                           \n",
       "\n",
       "  Reference 8 HPI-1 rank minus income poverty rank           HDI category  \n",
       "0         NaN                                   ..  LOW HUMAN DEVELOPMENT  \n",
       "1         NaN                                   ..  LOW HUMAN DEVELOPMENT  \n",
       "2         NaN                                   ..  LOW HUMAN DEVELOPMENT  \n",
       "3         NaN                                   ..  LOW HUMAN DEVELOPMENT  \n",
       "4         NaN                                   ..  LOW HUMAN DEVELOPMENT  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set \"HDI rank\" to integers\n",
    "df[\"HDI rank\"] = pd.to_numeric(df[\"HDI rank\"])\n",
    "# Categories\n",
    "hdi_categories = [\"HIGH HUMAN DEVELOPMENT\", \"MEDIUM HUMAN DEVELOPMENT\", \"LOW HUMAN DEVELOPMENT\"]\n",
    "# Se débarrasser des rangées avec les catégories HDI\n",
    "# https://stackoverflow.com/a/27360130 & https://stackoverflow.com/a/12098586\n",
    "df = df.drop(df[df[\"HDI rank\"].isin(hdi_categories)].index, axis=0).reset_index(drop=True)\n",
    "# Référence pour np.where https://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html\n",
    "# Pensez que cela ressemble à une déclaration du type \"if\" : where(test est vrai, vrai-quelque chose, faux-quelque chose)\n",
    "# ceci peut aussi être imbriqué : where(test is true, true-thing, where(new-test is true, true-thing, false-thing))\n",
    "df.loc[:, \"HDI category\"] = np.where(df[\"HDI rank\"] > 155,\n",
    "                                     \"LOW HUMAN DEVELOPMENT\",\n",
    "                                     np.where(df[\"HDI rank\"] > 70,\n",
    "                                              \"MEDIUM HUMAN DEVELOPMENT\",\n",
    "                                              \"HIGH HUMAN DEVELOPMENT\"\n",
    "                                             )\n",
    "                                    )\n",
    "df.sort_values(by=\"HDI rank\", inplace=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous pouvons faire beaucoup plus pour nettoyer cette feuille de calcul, mais c'est un bon début et cela vous apprend beaucoup sur ce que vous pouvez faire avec  pandas. Lorsque vous aurez terminé, n'oubliez pas de sauvegarder votre feuille de données en tant que `CSV` en utilisant la commande `.to_csv`, comme vous l'avez fait lors de l'exercice précédent.\n",
    "\n",
    "---\n",
    "\n",
    "## 2.6 Tutoriel de la leçon\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "    <p><b>Tutoriel:</b></p>\n",
    "    <p>Choisissez une feuille de calcul dans <a href=\"data/lesson-spreadsheet/\">données de formation</a> et restructurez-la selon les techniques et les exigences présentées dans cette leçon.</p>\n",
    "    <p>N'oubliez pas de créer un fichier de métadonnées en même temps que votre fichier de données.</p>\n",
    "</div>\n",
    "\n",
    "Veuillez terminer ce tutoriel avant de commencer la leçon suivante."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}