lakefs-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook\n",
    "\n",
    "This is quite a simple example of using `lakefs-spec`, hopefully doing the things  we want it to do. Link at the bottom for a more in depth example, but I've taken the relevant bits from that.\n",
    "\n",
    "Import the relevant libraries that we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lakefs_spec import LakeFSFileSystem\n",
    "import pandas as pd\n",
    "import lakefs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating the repo can be done through lakectl (command line), or through Python with the below cell.\n",
    "\n",
    "If you have errors with the buckets/minio not existing, just create one called `example-data` and put some parquet files inside. It should hopefully still work with this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "fs = LakeFSFileSystem()\n",
    "\n",
    "REPO_NAME = \"testing\"\n",
    "repo = lakefs.Repository(REPO_NAME, fs.client).create(storage_namespace=\"s3://example-data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you check the LakeFS UI, you'll see that a repo has been created.\n",
    "\n",
    "We now want to ingest our data from s3:\n",
    "\n",
    "## Import data from bucket to LakeFS\n",
    "\n",
    "The easiest way to get the data into LakeFS is through the UI. Click the green `Import` button, and point it to your bucket `s3://example-data`. Give it a nice commit.\n",
    "\n",
    "You can also do this through the command line tool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a new branch, where we will start modifying the data we can pull from LakeFS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Branch(repository=\"testing\", id=\"transform-raw-data\")"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NEW_BRANCH = lakefs.Branch(REPO_NAME, \"transform-raw-data\", client=fs.client)\n",
    "NEW_BRANCH.create(\"main\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now pull one of the files from the LakeFS repo and do some things to it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simply read the parquet file\n",
    "my_data = pd.read_parquet(f\"lakefs://testing/main/30390.parquet\")\n",
    "\n",
    "# Make some random changes\n",
    "my_data['new_column'] = 'new_values'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, let's put that onto our branch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "with fs.transaction(REPO_NAME, NEW_BRANCH) as tx:\n",
    "    my_data.to_parquet(f\"lakefs://{REPO_NAME}/{tx.branch.id}/30390.parquet\")\n",
    "    tx.commit(message=\"Added some data to 30390\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we check the UI, we will see that in `transform-raw-data` branch, the parquet file has been updated and committed. \n",
    "\n",
    "Now we decide we want to access the older version of that file, not the updated one. We can get the commit ID through here, or find the commit we want through the UI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a2cf1363af2548e54954fd2f4735cc2f9fb84a43d96843e9e1d6ea91014c068a\n"
     ]
    }
   ],
   "source": [
    "repo = lakefs.Repository(REPO_NAME, fs.client)\n",
    "\n",
    "# access the data of the previous commit with a lakefs ref expression, in this case the same as in git.\n",
    "previous_commit = repo.ref(f\"{NEW_BRANCH.id}~\").get_commit()\n",
    "fixed_commit_id = previous_commit.id\n",
    "print(fixed_commit_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, let's use that to get the original version of the 30390 file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dataset_item_uuid</th>\n",
       "      <th>description</th>\n",
       "      <th>file_name</th>\n",
       "      <th>format</th>\n",
       "      <th>label</th>\n",
       "      <th>mds_name</th>\n",
       "      <th>name</th>\n",
       "      <th>pass_</th>\n",
       "      <th>rank</th>\n",
       "      <th>shape</th>\n",
       "      <th>...</th>\n",
       "      <th>right</th>\n",
       "      <th>strobe</th>\n",
       "      <th>taps</th>\n",
       "      <th>top</th>\n",
       "      <th>trigger</th>\n",
       "      <th>vbin</th>\n",
       "      <th>view</th>\n",
       "      <th>width</th>\n",
       "      <th>_ARRAY_DIMENSIONS</th>\n",
       "      <th>INTERLACE_MODE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>fe655d78-9c51-5974-b75c-f0038f64bcd5</td>\n",
       "      <td>Shot used for calibration (obsolete)</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Calibration Shot</td>\n",
       "      <td>\\TOP.ANALYSED.ABM:CALIB_SHOT</td>\n",
       "      <td>ABM_CALIB_SHOT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[1]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d8f2807e-a3c8-5d90-8f81-25318efb4bfc</td>\n",
       "      <td>Failed = 0, OK = 1</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>channel_status</td>\n",
       "      <td>\\TOP.ANALYSED.ABM.CHANNEL:STATUS</td>\n",
       "      <td>ABM_CHANNEL_STATUS</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[1, 32]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>87598b5c-3805-5796-b918-33f65925c6e1</td>\n",
       "      <td>Channel type (0 = poloidal, 1 = co-tangential,...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>channel_type</td>\n",
       "      <td>\\TOP.ANALYSED.ABM:CHANNEL_TYPE</td>\n",
       "      <td>ABM_CHANNEL_TYPE</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[1, 32]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15cba832-1046-58f8-975e-0ca2f2c6236d</td>\n",
       "      <td>Gain of pre-amplifiers</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>GAIN</td>\n",
       "      <td>\\TOP.ANALYSED.ABM:GAIN</td>\n",
       "      <td>ABM_GAIN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[1, 32]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ee332e1b-7404-5bcb-a933-4fd77a27e02e</td>\n",
       "      <td>Incident powers (x - channel)</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>i-bol</td>\n",
       "      <td>\\TOP.ANALYSED.ABM:I_BOL</td>\n",
       "      <td>ABM_I-BOL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[7500, 32]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1042</th>\n",
       "      <td>d1a508cf-2c3b-5111-b22f-26e078afb051</td>\n",
       "      <td>Phantom colour camera</td>\n",
       "      <td>rco030390.ipx</td>\n",
       "      <td>IPX</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>RCO</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>512.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>149.0</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>HM11 - normal</td>\n",
       "      <td>512.0</td>\n",
       "      <td>None</td>\n",
       "      <td>INTERLACE_PIXEL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1043</th>\n",
       "      <td>75db70bc-a66c-5cf1-aff0-676ab4a0e5c8</td>\n",
       "      <td>RBG 2D multi-colour visible bremsstrahlung camera</td>\n",
       "      <td>rgb030390.ipx</td>\n",
       "      <td>IPX</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>RGB</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>640.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Sector9U</td>\n",
       "      <td>640.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1044</th>\n",
       "      <td>a6e4aa24-84a7-5754-95ae-601b5b9567bf</td>\n",
       "      <td>RBG 2D multi-colour visible bremsstrahlung camera</td>\n",
       "      <td>rgc030390.ipx</td>\n",
       "      <td>IPX</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>RGC</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>640.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Sector9U</td>\n",
       "      <td>640.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1045</th>\n",
       "      <td>f273f736-ca87-5a22-a2f0-fcbba9ca00fb</td>\n",
       "      <td>Medium wavelength infrared camera</td>\n",
       "      <td>rir030390.ipx</td>\n",
       "      <td>IPX</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>RIR</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>320.0</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>4.0</td>\n",
       "      <td>185.0</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Lower divertor view#6</td>\n",
       "      <td>320.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1046</th>\n",
       "      <td>76ea0e0b-35f1-5af2-afed-ea6674c0c303</td>\n",
       "      <td>Long wavelength infrared camera</td>\n",
       "      <td>rit030390.ipx</td>\n",
       "      <td>IPX</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>RIT</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.070524e+09</td>\n",
       "      <td>0.0</td>\n",
       "      <td>153.0</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>HL01 Upper divertor view#1</td>\n",
       "      <td>256.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1047 rows × 51 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                         dataset_item_uuid  \\\n",
       "0     fe655d78-9c51-5974-b75c-f0038f64bcd5   \n",
       "1     d8f2807e-a3c8-5d90-8f81-25318efb4bfc   \n",
       "2     87598b5c-3805-5796-b918-33f65925c6e1   \n",
       "3     15cba832-1046-58f8-975e-0ca2f2c6236d   \n",
       "4     ee332e1b-7404-5bcb-a933-4fd77a27e02e   \n",
       "...                                    ...   \n",
       "1042  d1a508cf-2c3b-5111-b22f-26e078afb051   \n",
       "1043  75db70bc-a66c-5cf1-aff0-676ab4a0e5c8   \n",
       "1044  a6e4aa24-84a7-5754-95ae-601b5b9567bf   \n",
       "1045  f273f736-ca87-5a22-a2f0-fcbba9ca00fb   \n",
       "1046  76ea0e0b-35f1-5af2-afed-ea6674c0c303   \n",
       "\n",
       "                                            description      file_name format  \\\n",
       "0                  Shot used for calibration (obsolete)           None   None   \n",
       "1                                    Failed = 0, OK = 1           None   None   \n",
       "2     Channel type (0 = poloidal, 1 = co-tangential,...           None   None   \n",
       "3                                Gain of pre-amplifiers           None   None   \n",
       "4                         Incident powers (x - channel)           None   None   \n",
       "...                                                 ...            ...    ...   \n",
       "1042                              Phantom colour camera  rco030390.ipx    IPX   \n",
       "1043  RBG 2D multi-colour visible bremsstrahlung camera  rgb030390.ipx    IPX   \n",
       "1044  RBG 2D multi-colour visible bremsstrahlung camera  rgc030390.ipx    IPX   \n",
       "1045                  Medium wavelength infrared camera  rir030390.ipx    IPX   \n",
       "1046                    Long wavelength infrared camera  rit030390.ipx    IPX   \n",
       "\n",
       "                 label                          mds_name                name  \\\n",
       "0     Calibration Shot      \\TOP.ANALYSED.ABM:CALIB_SHOT      ABM_CALIB_SHOT   \n",
       "1       channel_status  \\TOP.ANALYSED.ABM.CHANNEL:STATUS  ABM_CHANNEL_STATUS   \n",
       "2         channel_type    \\TOP.ANALYSED.ABM:CHANNEL_TYPE    ABM_CHANNEL_TYPE   \n",
       "3                 GAIN            \\TOP.ANALYSED.ABM:GAIN            ABM_GAIN   \n",
       "4                i-bol           \\TOP.ANALYSED.ABM:I_BOL           ABM_I-BOL   \n",
       "...                ...                               ...                 ...   \n",
       "1042              None                              None                 RCO   \n",
       "1043              None                              None                 RGB   \n",
       "1044              None                              None                 RGC   \n",
       "1045              None                              None                 RIR   \n",
       "1046              None                              None                 RIT   \n",
       "\n",
       "      pass_  rank       shape  ...  right        strobe taps    top  trigger  \\\n",
       "0       0.0   1.0         [1]  ...    NaN           NaN  NaN    NaN      NaN   \n",
       "1       0.0   2.0     [1, 32]  ...    NaN           NaN  NaN    NaN      NaN   \n",
       "2       0.0   2.0     [1, 32]  ...    NaN           NaN  NaN    NaN      NaN   \n",
       "3       0.0   2.0     [1, 32]  ...    NaN           NaN  NaN    NaN      NaN   \n",
       "4       0.0   2.0  [7500, 32]  ...    NaN           NaN  NaN    NaN      NaN   \n",
       "...     ...   ...         ...  ...    ...           ...  ...    ...      ...   \n",
       "1042   -1.0   NaN        None  ...  512.0  0.000000e+00  0.0  149.0     -0.1   \n",
       "1043   -1.0   NaN        None  ...  640.0  0.000000e+00  1.0    1.0     -0.1   \n",
       "1044   -1.0   NaN        None  ...  640.0  0.000000e+00  2.0    1.0     -0.1   \n",
       "1045   -1.0   NaN        None  ...  320.0  0.000000e+00  4.0  185.0     -0.1   \n",
       "1046   -1.0   NaN        None  ...    0.0  1.070524e+09  0.0  153.0     -0.5   \n",
       "\n",
       "     vbin                        view  width _ARRAY_DIMENSIONS  \\\n",
       "0     NaN                        None    NaN              None   \n",
       "1     NaN                        None    NaN              None   \n",
       "2     NaN                        None    NaN              None   \n",
       "3     NaN                        None    NaN              None   \n",
       "4     NaN                        None    NaN              None   \n",
       "...   ...                         ...    ...               ...   \n",
       "1042  0.0              HM11 - normal   512.0              None   \n",
       "1043  0.0                    Sector9U  640.0              None   \n",
       "1044  0.0                    Sector9U  640.0              None   \n",
       "1045  0.0       Lower divertor view#6  320.0              None   \n",
       "1046  0.0  HL01 Upper divertor view#1  256.0              None   \n",
       "\n",
       "       INTERLACE_MODE  \n",
       "0                None  \n",
       "1                None  \n",
       "2                None  \n",
       "3                None  \n",
       "4                None  \n",
       "...               ...  \n",
       "1042  INTERLACE_PIXEL  \n",
       "1043             None  \n",
       "1044             None  \n",
       "1045             None  \n",
       "1046             None  \n",
       "\n",
       "[1047 rows x 51 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orig_file = pd.read_parquet(f\"lakefs://{REPO_NAME}/{fixed_commit_id}/30390.parquet\")\n",
    "orig_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No `new_column`!\n",
    "\n",
    "We haven't gone through merging branches here, but we can do that through the UI or through the command line tool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aaaaand then we can also get the new version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dataset_item_uuid</th>\n",
       "      <th>description</th>\n",
       "      <th>file_name</th>\n",
       "      <th>format</th>\n",
       "      <th>label</th>\n",
       "      <th>mds_name</th>\n",
       "      <th>name</th>\n",
       "      <th>pass_</th>\n",
       "      <th>rank</th>\n",
       "      <th>shape</th>\n",
       "      <th>...</th>\n",
       "      <th>strobe</th>\n",
       "      <th>taps</th>\n",
       "      <th>top</th>\n",
       "      <th>trigger</th>\n",
       "      <th>vbin</th>\n",
       "      <th>view</th>\n",
       "      <th>width</th>\n",
       "      <th>_ARRAY_DIMENSIONS</th>\n",
       "      <th>INTERLACE_MODE</th>\n",
       "      <th>new_column</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>fe655d78-9c51-5974-b75c-f0038f64bcd5</td>\n",
       "      <td>Shot used for calibration (obsolete)</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Calibration Shot</td>\n",
       "      <td>\\TOP.ANALYSED.ABM:CALIB_SHOT</td>\n",
       "      <td>ABM_CALIB_SHOT</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[1]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>new_values</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d8f2807e-a3c8-5d90-8f81-25318efb4bfc</td>\n",
       "      <td>Failed = 0, OK = 1</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>channel_status</td>\n",
       "      <td>\\TOP.ANALYSED.ABM.CHANNEL:STATUS</td>\n",
       "      <td>ABM_CHANNEL_STATUS</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[1, 32]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>new_values</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>87598b5c-3805-5796-b918-33f65925c6e1</td>\n",
       "      <td>Channel type (0 = poloidal, 1 = co-tangential,...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>channel_type</td>\n",
       "      <td>\\TOP.ANALYSED.ABM:CHANNEL_TYPE</td>\n",
       "      <td>ABM_CHANNEL_TYPE</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[1, 32]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>new_values</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15cba832-1046-58f8-975e-0ca2f2c6236d</td>\n",
       "      <td>Gain of pre-amplifiers</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>GAIN</td>\n",
       "      <td>\\TOP.ANALYSED.ABM:GAIN</td>\n",
       "      <td>ABM_GAIN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[1, 32]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>new_values</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ee332e1b-7404-5bcb-a933-4fd77a27e02e</td>\n",
       "      <td>Incident powers (x - channel)</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>i-bol</td>\n",
       "      <td>\\TOP.ANALYSED.ABM:I_BOL</td>\n",
       "      <td>ABM_I-BOL</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>[7500, 32]</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>new_values</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1042</th>\n",
       "      <td>d1a508cf-2c3b-5111-b22f-26e078afb051</td>\n",
       "      <td>Phantom colour camera</td>\n",
       "      <td>rco030390.ipx</td>\n",
       "      <td>IPX</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>RCO</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>149.0</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>HM11 - normal</td>\n",
       "      <td>512.0</td>\n",
       "      <td>None</td>\n",
       "      <td>INTERLACE_PIXEL</td>\n",
       "      <td>new_values</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1043</th>\n",
       "      <td>75db70bc-a66c-5cf1-aff0-676ab4a0e5c8</td>\n",
       "      <td>RBG 2D multi-colour visible bremsstrahlung camera</td>\n",
       "      <td>rgb030390.ipx</td>\n",
       "      <td>IPX</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>RGB</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Sector9U</td>\n",
       "      <td>640.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>new_values</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1044</th>\n",
       "      <td>a6e4aa24-84a7-5754-95ae-601b5b9567bf</td>\n",
       "      <td>RBG 2D multi-colour visible bremsstrahlung camera</td>\n",
       "      <td>rgc030390.ipx</td>\n",
       "      <td>IPX</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>RGC</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Sector9U</td>\n",
       "      <td>640.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>new_values</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1045</th>\n",
       "      <td>f273f736-ca87-5a22-a2f0-fcbba9ca00fb</td>\n",
       "      <td>Medium wavelength infrared camera</td>\n",
       "      <td>rir030390.ipx</td>\n",
       "      <td>IPX</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>RIR</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>4.0</td>\n",
       "      <td>185.0</td>\n",
       "      <td>-0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Lower divertor view#6</td>\n",
       "      <td>320.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>new_values</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1046</th>\n",
       "      <td>76ea0e0b-35f1-5af2-afed-ea6674c0c303</td>\n",
       "      <td>Long wavelength infrared camera</td>\n",
       "      <td>rit030390.ipx</td>\n",
       "      <td>IPX</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>RIT</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>1.070524e+09</td>\n",
       "      <td>0.0</td>\n",
       "      <td>153.0</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>HL01 Upper divertor view#1</td>\n",
       "      <td>256.0</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>new_values</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1047 rows × 52 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                         dataset_item_uuid  \\\n",
       "0     fe655d78-9c51-5974-b75c-f0038f64bcd5   \n",
       "1     d8f2807e-a3c8-5d90-8f81-25318efb4bfc   \n",
       "2     87598b5c-3805-5796-b918-33f65925c6e1   \n",
       "3     15cba832-1046-58f8-975e-0ca2f2c6236d   \n",
       "4     ee332e1b-7404-5bcb-a933-4fd77a27e02e   \n",
       "...                                    ...   \n",
       "1042  d1a508cf-2c3b-5111-b22f-26e078afb051   \n",
       "1043  75db70bc-a66c-5cf1-aff0-676ab4a0e5c8   \n",
       "1044  a6e4aa24-84a7-5754-95ae-601b5b9567bf   \n",
       "1045  f273f736-ca87-5a22-a2f0-fcbba9ca00fb   \n",
       "1046  76ea0e0b-35f1-5af2-afed-ea6674c0c303   \n",
       "\n",
       "                                            description      file_name format  \\\n",
       "0                  Shot used for calibration (obsolete)           None   None   \n",
       "1                                    Failed = 0, OK = 1           None   None   \n",
       "2     Channel type (0 = poloidal, 1 = co-tangential,...           None   None   \n",
       "3                                Gain of pre-amplifiers           None   None   \n",
       "4                         Incident powers (x - channel)           None   None   \n",
       "...                                                 ...            ...    ...   \n",
       "1042                              Phantom colour camera  rco030390.ipx    IPX   \n",
       "1043  RBG 2D multi-colour visible bremsstrahlung camera  rgb030390.ipx    IPX   \n",
       "1044  RBG 2D multi-colour visible bremsstrahlung camera  rgc030390.ipx    IPX   \n",
       "1045                  Medium wavelength infrared camera  rir030390.ipx    IPX   \n",
       "1046                    Long wavelength infrared camera  rit030390.ipx    IPX   \n",
       "\n",
       "                 label                          mds_name                name  \\\n",
       "0     Calibration Shot      \\TOP.ANALYSED.ABM:CALIB_SHOT      ABM_CALIB_SHOT   \n",
       "1       channel_status  \\TOP.ANALYSED.ABM.CHANNEL:STATUS  ABM_CHANNEL_STATUS   \n",
       "2         channel_type    \\TOP.ANALYSED.ABM:CHANNEL_TYPE    ABM_CHANNEL_TYPE   \n",
       "3                 GAIN            \\TOP.ANALYSED.ABM:GAIN            ABM_GAIN   \n",
       "4                i-bol           \\TOP.ANALYSED.ABM:I_BOL           ABM_I-BOL   \n",
       "...                ...                               ...                 ...   \n",
       "1042              None                              None                 RCO   \n",
       "1043              None                              None                 RGB   \n",
       "1044              None                              None                 RGC   \n",
       "1045              None                              None                 RIR   \n",
       "1046              None                              None                 RIT   \n",
       "\n",
       "      pass_  rank       shape  ...        strobe  taps    top  trigger  vbin  \\\n",
       "0       0.0   1.0         [1]  ...           NaN   NaN    NaN      NaN   NaN   \n",
       "1       0.0   2.0     [1, 32]  ...           NaN   NaN    NaN      NaN   NaN   \n",
       "2       0.0   2.0     [1, 32]  ...           NaN   NaN    NaN      NaN   NaN   \n",
       "3       0.0   2.0     [1, 32]  ...           NaN   NaN    NaN      NaN   NaN   \n",
       "4       0.0   2.0  [7500, 32]  ...           NaN   NaN    NaN      NaN   NaN   \n",
       "...     ...   ...         ...  ...           ...   ...    ...      ...   ...   \n",
       "1042   -1.0   NaN        None  ...  0.000000e+00   0.0  149.0     -0.1   0.0   \n",
       "1043   -1.0   NaN        None  ...  0.000000e+00   1.0    1.0     -0.1   0.0   \n",
       "1044   -1.0   NaN        None  ...  0.000000e+00   2.0    1.0     -0.1   0.0   \n",
       "1045   -1.0   NaN        None  ...  0.000000e+00   4.0  185.0     -0.1   0.0   \n",
       "1046   -1.0   NaN        None  ...  1.070524e+09   0.0  153.0     -0.5   0.0   \n",
       "\n",
       "                            view  width _ARRAY_DIMENSIONS   INTERLACE_MODE  \\\n",
       "0                           None    NaN              None             None   \n",
       "1                           None    NaN              None             None   \n",
       "2                           None    NaN              None             None   \n",
       "3                           None    NaN              None             None   \n",
       "4                           None    NaN              None             None   \n",
       "...                          ...    ...               ...              ...   \n",
       "1042              HM11 - normal   512.0              None  INTERLACE_PIXEL   \n",
       "1043                    Sector9U  640.0              None             None   \n",
       "1044                    Sector9U  640.0              None             None   \n",
       "1045       Lower divertor view#6  320.0              None             None   \n",
       "1046  HL01 Upper divertor view#1  256.0              None             None   \n",
       "\n",
       "      new_column  \n",
       "0     new_values  \n",
       "1     new_values  \n",
       "2     new_values  \n",
       "3     new_values  \n",
       "4     new_values  \n",
       "...          ...  \n",
       "1042  new_values  \n",
       "1043  new_values  \n",
       "1044  new_values  \n",
       "1045  new_values  \n",
       "1046  new_values  \n",
       "\n",
       "[1047 rows x 52 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_file = pd.read_parquet(f\"lakefs://{REPO_NAME}/transform-raw-data/30390.parquet\")\n",
    "new_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following [this](https://lakefs-spec.org/latest/tutorials/demo_data_science_project/), you can also TAG commits with your own string, so we could tag the above, updated, file with something like `first-file-change` and then use that tag to pull that version of the file."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}