01-introduction-julia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/jolin-io/workshop-accelerate-Python-with-Julia/main?filepath=01-introduction-julia.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.jolin.io\" target=\"_blank\" rel=\"noreferrer noopener\">\n",
    "<img src=\"https://www.jolin.io/assets/Jolin/Jolin-Banner-Website-v1.3-darkmode.webp\">\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Tutorial PyCon 2023: Accelerate Python with Julia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"display: flex\">\n",
    "<div style=\"width: 20%\">\n",
    "<a href=\"https://www.jolin.io/en/about-us\" target=\"_blank\" rel=\"noreferrer noopener\">\n",
    "<img src=\"https://www.jolin.io/assets/stephansahm-extreme-small.webp\" style=\"height: 300px\">\n",
    "</a>\n",
    "</div>\n",
    "    \n",
    "<div style=\"width: 50%\">\n",
    "    \n",
    "### Stephan Sahm\n",
    "    \n",
    "- Founder of Jolin.io julia consultancy\n",
    "- Organizer of Julia User Group Munich\n",
    "- Full stack data science consultant\n",
    "- Applied stochastics, uncertainty handling\n",
    "- Big Data, High Performance Computing and Real-time processing\n",
    "- Making things production ready\n",
    "\n",
    "</div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------\n",
    "\n",
    "### Outline for today\n",
    "\n",
    "1. **Introduction to Julia I:** calling Julia from Python\n",
    "2. **Introduction to Julia II:** Pluto, pure Julia\n",
    "3. **Simulation example:** Python vs Cython vs C++ vs Julia\n",
    "\n",
    "<br>\n",
    "\n",
    "-------\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Introduction to Julia I:** Calling Julia from Python\n",
    "\n",
    "For further study, https://julialang.org/learning/ is the perfect place to start."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While there are multiple options to use Julia from Python I recommend using python package `juliacall`.\n",
    "- 🙂 it does not copy data, but passes mutable references between the languages\n",
    "- 🙂 good defaults\n",
    "- 🙂 nice printing in Jupyter\n",
    "\n",
    "In depth documentation about `juliacall` (and the corresponding `PythonCall.jl` julia package) can be found at [`PythonCall.jl`](https://cjdoris.github.io/PythonCall.jl/stable/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from juliacall import Main as jl\n",
    "%load_ext juliacall.ipython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us access to cell magic `%%julia` and line magic `%julia`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# JuliaCall comes with its own Julia dependency file juliapkg.json\n",
    "# however for binder it is much simpler to just reuse binder's installation mechanism\n",
    "%julia Pkg.activate(Base.current_project())\n",
    "%julia using PythonCall\n",
    "%julia set_var(k, v) = @eval $(Symbol(k)) = $v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we defined a little helper called `set_var` which we can use to copy Python objects to Julia.\n",
    "\n",
    "Any julia function or variable defined in the global julia namespace can be accessed directly on `jl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 1000\n",
    "jl.set_var(\"n\", n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arrays\n",
    "\n",
    "Julia has excellent support for arrays. Unlike NumPy, Julia supports excellent performance also for custom data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "nparray = np.arange(n)\n",
    "jl.set_var(\"nparray\", nparray)\n",
    "nparray[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another goodie: Any custom julia function can be broadcasted over arrays (in high speed)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%julia\n",
    "@show typeof(nparray)\n",
    "\n",
    "double(x) = 2x\n",
    "result = double.(nparray)\n",
    "\n",
    "@show typeof(result)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jlarray = _\n",
    "print(type(jlarray))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "back_to_python = np.array(jlarray) / 2\n",
    "back_to_python[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Minibenchmarks\n",
    "\n",
    "In order to call our double function from python we define an extra helper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%julia\n",
    "array_double(a) = double.(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When comparing to python's standard range, going to julia and back to python is still faster for larger `n`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit [x*2 for x in range(n)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit jl.array_double(range(n))\n",
    "# for n=1000, julia is faster, while for n=100 python is still faster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When comparing with `numpy`, numpy is faster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit nparray * 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit jl.array_double(nparray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to inspect what the pure julia time would be, we can benchmark directly within julia using `BenchmarkTools.jl`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%julia using BenchmarkTools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%julia\n",
    "@btime double.(nparray)\n",
    "\n",
    "# when converting the Python wrapper to julia's standard type Vector\n",
    "# we get a little extra boost (not too much actually)\n",
    "jlarray = pyconvert(Vector, nparray)\n",
    "@btime double.(jlarray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another tiny example: Matrix multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%timeit np.random.rand(1, n) @ np.random.rand(n, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using Julias multiplication on python objects\n",
    "%julia multiply(a, b) = a * b\n",
    "%timeit jl.multiply(np.random.rand(1, n), np.random.rand(n, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using plain julia\n",
    "%julia @btime rand(1, n) * rand(n, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------\n",
    "#### 💻 your space\n",
    "- 👉 try a different `n`\n",
    "- 👉 try mapping some further numpy operations to julia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your space ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What to do, if you neither have Jupyter nor `%%julia`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if you are not in a jupyter notebook, you can simply write a julia file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile example.jl\n",
    "\n",
    "a = 2\n",
    "myfunc(args...; kwargs...) = (args, kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`jl.seval(...)` executes arbitrary julia code.\n",
    "\n",
    "In addition, julia comes with a special function `include`, which loads the given filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jl.seval('include(\"example.jl\")')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or use `jl.seval` directly\n",
    "\n",
    "⚠️ when using multiple statements they need to be wrapped into `begin` - `end`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jl.seval(\"\"\"begin\n",
    "    a = 2\n",
    "    myfunc(args...; kwargs...) = (args, kwargs)\n",
    "end\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jl.myfunc(1,4, [1,2,3], range(10), mykey=list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## DataFrames\n",
    "\n",
    "Julia has excellent support for DataFrames, too. Again with excellent support for custom data types.\n",
    "\n",
    "See https://dataframes.juliadata.org/stable/man/comparisons/ for a detailed mapping from pandas to DataFrames.jl."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.DataFrame({'grp': [1, 2] * 3,\n",
    "                   'x': range(6, 0, -1),\n",
    "                   'y': range(4, 10),\n",
    "                   'z': [3, 4, 5, 6, 7, None]},\n",
    "                   index = list('abcdef'))\n",
    "df2 = pd.DataFrame({'grp': [1, 3], 'w': [10, 11]})\n",
    "\n",
    "df.groupby('grp')['x'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%julia\n",
    "using DataFrames\n",
    "using Statistics\n",
    "\n",
    "df = DataFrame(grp=repeat(1:2, 3), x=6:-1:1, y=4:9, z=[3:7; missing], id='a':'f')\n",
    "df2 = DataFrame(grp=[1, 3], w=[10, 11])\n",
    "\n",
    "combine(groupby(df, :grp), :x => mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "mini benchmarks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3 = pd.DataFrame({'grp': [1, 2] * n, 'x': range(2*n, 0, -1)})\n",
    "%timeit df3.groupby('grp')['x'].mean()\n",
    "df3.groupby('grp')['x'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%julia\n",
    "df3 = DataFrame(grp = repeat(1:2, n), x = 2n:-1:1)\n",
    "@btime combine(groupby(df3, :grp), :x => mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------\n",
    "#### 💻 your space\n",
    "- 👉 look at the documentation and try mapping some further pandas operations\n",
    "- 👉 try a different `n`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Further resources\n",
    "\n",
    "- [learning Julia](https://julialang.org/learning/)\n",
    "- [juliacall / PythonCall.jl](https://cjdoris.github.io/PythonCall.jl/stable/)\n",
    "- using `juliacall` you can specify your julia dependencies using a json config file which will automatically be initialized by juliacall (see [these docs](https://cjdoris.github.io/PythonCall.jl/stable/juliacall/#julia-deps))\n",
    "\n",
    "ways to add Julia itself as a dependency:\n",
    "- `juliacall` uses [`juliapkg`](https://github.com/cjdoris/pyjuliapkg/) internally which also installs julia if it is not available\n",
    "- alternatively, the python package [`jill`](https://pypi.org/project/jill/) is widely used to install julia from python\n",
    "- the conda-forge [`julia`](https://github.com/conda-forge/julia-feedstock) package is well maintained\n",
    "\n",
    "# Next\n",
    "\n",
    "Next you will...\n",
    "- learn why Julia is both easy and fast\n",
    "- get to know Pluto, a reactive alternative to Jupyter\n",
    "\n",
    "[Next Notebook](https://mybinder.org/v2/gh/jolin-io/workshop-accelerate-Python-with-Julia/main?urlpath=pluto/open?path=/home/jovyan/02-introduction-pluto.jl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.jolin.io\" target=\"_blank\" rel=\"noreferrer noopener\">\n",
    "<img src=\"https://www.jolin.io/assets/Jolin/Jolin-Banner-Website-v1.3-darkmode.webp\">\n",
    "</a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "workshop-accelerate-Python-with-Julia",
   "language": "python",
   "name": "workshop-accelerate-python-with-julia"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "543b09ee78a375d6a601bebe4e67ef1844e93448f09439d9c76edcacec30d7c5"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}