{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 6B - Tools for high-performance computing applications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "J.R. Johansson (jrjohansson at gmail.com)\n",
    "\n",
    "The latest version of this [IPython notebook](http://ipython.org/notebook.html) lecture is available at [http://github.com/jrjohansson/scientific-python-lectures](http://github.com/jrjohansson/scientific-python-lectures).\n",
    "\n",
    "The other notebooks in this lecture series are indexed at [http://jrjohansson.github.io](http://jrjohansson.github.io)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## multiprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python has a built-in process-based library for concurrent computing, called `multiprocessing`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import multiprocessing\n",
    "import os\n",
    "import time\n",
    "import numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def task(args):\n",
    "    print(\"PID =\", os.getpid(), \", args =\", args)\n",
    "    \n",
    "    return os.getpid(), args"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PID = 28995 , args = test\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(28995, 'test')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task(\"test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pool = multiprocessing.Pool(processes=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PID = 29006 , args = 1\n",
      "PID = 29009 , args = 4\n",
      "PID = 29007 , args = 2\n",
      "PID = 29008 , args = 3\n",
      "PID = 29006 , args = 6\n",
      "PID = 29009 , args = 5\n",
      "PID = 29007 , args = 8\n",
      "PID = 29008 , args = 7\n"
     ]
    }
   ],
   "source": [
    "result = pool.map(task, [1,2,3,4,5,6,7,8])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(29006, 1),\n",
       " (29007, 2),\n",
       " (29008, 3),\n",
       " (29009, 4),\n",
       " (29009, 5),\n",
       " (29006, 6),\n",
       " (29008, 7),\n",
       " (29007, 8)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The multiprocessing package is very useful for highly parallel tasks that do not need to communicate with each other, other than when sending the initial data to the pool of processes and when and collecting the results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IPython parallel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IPython includes a very interesting and versatile parallel computing environment, which is very easy to use. It builds on the concept of ipython engines and controllers, that one can connect to and submit tasks to. To get started using this framework for parallel computing, one first have to start up an IPython cluster of engines. The easiest way to do this is to use the `ipcluster` command,\n",
    "\n",
    "    $ ipcluster start -n 4\n",
    "\n",
    "Or, alternatively, from the \"Clusters\" tab on the IPython notebook dashboard page. This will start 4 IPython engines on the current host, which is useful for multicore systems. It is also possible to setup IPython clusters that spans over many nodes in a computing cluster. For more information about possible use cases, see the official documentation [Using IPython for parallel computing](http://ipython.org/ipython-doc/dev/parallel/).\n",
    "\n",
    "To use the IPython cluster in our Python programs or notebooks, we start by creating an instance of `IPython.parallel.Client`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython.parallel import Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cli = Client()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the 'ids' attribute we can retrieve a list of ids for the IPython engines in the cluster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cli.ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of these engines are ready to execute tasks. We can selectively run code on individual engines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def getpid():\n",
    "    \"\"\" return the unique ID of the current process \"\"\"\n",
    "    import os\n",
    "    return os.getpid()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28995"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first try it on the notebook process\n",
    "getpid()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30181"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run it on one of the engines\n",
    "cli[0].apply_sync(getpid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[30181, 30182, 30183, 30185]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run it on ALL of the engines at the same time\n",
    "cli[:].apply_sync(getpid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this cluster of IPython engines to execute tasks in parallel. The easiest way to dispatch a function to different engines is to define the function with the decorator:\n",
    "\n",
    "    @view.parallel(block=True)\n",
    "\n",
    "Here, `view` is supposed to be the engine pool which we want to dispatch the function (task). Once our function is defined this way we can dispatch it to the engine using the `map` method in the resulting class (in Python, a decorator is a language construct which automatically wraps the function into another function or a class).\n",
    "\n",
    "To see how all this works, lets look at an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dview = cli[:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "@dview.parallel(block=True)\n",
    "def dummy_task(delay):\n",
    "    \"\"\" a dummy task that takes 'delay' seconds to finish \"\"\"\n",
    "    import os, time\n",
    "\n",
    "    t0 = time.time()\n",
    "    pid = os.getpid()\n",
    "    time.sleep(delay)\n",
    "    t1 = time.time()\n",
    "    \n",
    "    return [pid, t0, t1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# generate random delay times for dummy tasks\n",
    "delay_times = numpy.random.rand(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to map the function `dummy_task` to the random delay time data, we use the `map` method in `dummy_task`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[30181, 1395044753.2096598, 1395044753.9150908],\n",
       " [30182, 1395044753.2084103, 1395044753.4959202],\n",
       " [30183, 1395044753.2113762, 1395044753.6453338],\n",
       " [30185, 1395044753.2130392, 1395044754.1905618]]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy_task.map(delay_times)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do the same thing again with many more tasks and visualize how these tasks are executed on different IPython engines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def visualize_tasks(results):\n",
    "    res = numpy.array(results)\n",
    "    fig, ax = plt.subplots(figsize=(10, res.shape[1]))\n",
    "    \n",
    "    yticks = []\n",
    "    yticklabels = []\n",
    "    tmin = min(res[:,1])\n",
    "    for n, pid in enumerate(numpy.unique(res[:,0])):\n",
    "        yticks.append(n)\n",
    "        yticklabels.append(\"%d\" % pid)\n",
    "        for m in numpy.where(res[:,0] == pid)[0]:\n",
    "            ax.add_patch(plt.Rectangle((res[m,1] - tmin, n-0.25),\n",
    "                         res[m,2] - res[m,1], 0.5, color=\"green\", alpha=0.5))\n",
    "        \n",
    "    ax.set_ylim(-.5, n+.5)\n",
    "    ax.set_xlim(0, max(res[:,2]) - tmin + 0.)\n",
    "    ax.set_yticks(yticks)\n",
    "    ax.set_yticklabels(yticklabels)\n",
    "    ax.set_ylabel(\"PID\")\n",
    "    ax.set_xlabel(\"seconds\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "delay_times = numpy.random.rand(64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 10356 characters]",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7ae440f210>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "result = dummy_task.map(delay_times)\n",
    "visualize_tasks(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's a nice and easy parallelization! We can see that we utilize all four engines quite well.\n",
    "\n",
    "But one short coming so far is that the tasks are not load balanced, so one engine might be idle while others still have more tasks to work on.\n",
    "\n",
    "However, the IPython parallel environment provides a number of alternative \"views\" of the engine cluster, and there is a view that provides load balancing as well (above we have used the \"direct view\", which is why we called it \"dview\").\n",
    "\n",
    "To obtain a load balanced view we simply use the `load_balanced_view` method in the engine cluster client instance `cli`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lbview = cli.load_balanced_view()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "@lbview.parallel(block=True)\n",
    "def dummy_task_load_balanced(delay):\n",
    "    \"\"\" a dummy task that takes 'delay' seconds to finish \"\"\"\n",
    "    import os, time\n",
    "\n",
    "    t0 = time.time()\n",
    "    pid = os.getpid()\n",
    "    time.sleep(delay)\n",
    "    t1 = time.time()\n",
    "    \n",
    "    return [pid, t0, t1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 10212 characters]",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7ae43ec990>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "result = dummy_task_load_balanced.map(delay_times)\n",
    "visualize_tasks(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above we can see that the engine cluster is a bit more efficiently used, and the time to completion is shorter than in the previous example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many other ways to use the IPython parallel environment. The official documentation has a nice guide:\n",
    "\n",
    "* http://ipython.org/ipython-doc/dev/parallel/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MPI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When more communication between processes is required, sophisticated solutions such as MPI and OpenMP are often needed. MPI is process based parallel processing library/protocol, and can be used in Python programs through the `mpi4py` package:\n",
    "\n",
    "http://mpi4py.scipy.org/\n",
    "\n",
    "To use the `mpi4py` package we include `MPI` from `mpi4py`:\n",
    "\n",
    "    from mpi4py import MPI\n",
    "\n",
    "A MPI python program must be started using the `mpirun -n N` command, where `N` is the number of processes that should be included in the process group.\n",
    "\n",
    "Note that the IPython parallel environment also has support for MPI, but to begin with we will use `mpi4py` and the `mpirun` in the follow examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mpitest.py\n"
     ]
    }
   ],
   "source": [
    "%%file mpitest.py\n",
    "\n",
    "from mpi4py import MPI\n",
    "\n",
    "comm = MPI.COMM_WORLD\n",
    "rank = comm.Get_rank()\n",
    "\n",
    "if rank == 0:\n",
    "   data = [1.0, 2.0, 3.0, 4.0]\n",
    "   comm.send(data, dest=1, tag=11)\n",
    "elif rank == 1:\n",
    "   data = comm.recv(source=0, tag=11)\n",
    "    \n",
    "print \"rank =\", rank, \", data =\", data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rank = 0 , data = [1.0, 2.0, 3.0, 4.0]\r\n",
      "rank = 1 , data = [1.0, 2.0, 3.0, 4.0]\r\n"
     ]
    }
   ],
   "source": [
    "!mpirun -n 2 python mpitest.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send a numpy array from one process to another:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mpi-numpy-array.py\n"
     ]
    }
   ],
   "source": [
    "%%file mpi-numpy-array.py\n",
    "\n",
    "from mpi4py import MPI\n",
    "import numpy\n",
    "\n",
    "comm = MPI.COMM_WORLD\n",
    "rank = comm.Get_rank()\n",
    "\n",
    "if rank == 0:\n",
    "   data = numpy.random.rand(10)\n",
    "   comm.Send(data, dest=1, tag=13)\n",
    "elif rank == 1:\n",
    "   data = numpy.empty(10, dtype=numpy.float64)\n",
    "   comm.Recv(data, source=0, tag=13)\n",
    "    \n",
    "print \"rank =\", rank, \", data =\", data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rank = 0 , data = [ 0.71397658  0.37182268  0.25863587  0.08007216  0.50832534  0.80038331\r\n",
      "  0.90613024  0.99535428  0.11717776  0.48353805]\r\n",
      "rank = 1 , data = [ 0.71397658  0.37182268  0.25863587  0.08007216  0.50832534  0.80038331\r\n",
      "  0.90613024  0.99535428  0.11717776  0.48353805]\r\n"
     ]
    }
   ],
   "source": [
    "!mpirun -n 2 python mpi-numpy-array.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3: Matrix-vector multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# prepare some random data\n",
    "N = 16\n",
    "A = numpy.random.rand(N, N)\n",
    "numpy.save(\"random-matrix.npy\", A)\n",
    "x = numpy.random.rand(N)\n",
    "numpy.save(\"random-vector.npy\", x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mpi-matrix-vector.py\n"
     ]
    }
   ],
   "source": [
    "%%file mpi-matrix-vector.py\n",
    "\n",
    "from mpi4py import MPI\n",
    "import numpy\n",
    "\n",
    "comm = MPI.COMM_WORLD\n",
    "rank = comm.Get_rank()\n",
    "p = comm.Get_size()\n",
    "\n",
    "def matvec(comm, A, x):\n",
    "    m = A.shape[0] / p\n",
    "    y_part = numpy.dot(A[rank * m:(rank+1)*m], x)\n",
    "    y = numpy.zeros_like(x)\n",
    "    comm.Allgather([y_part,  MPI.DOUBLE], [y, MPI.DOUBLE])\n",
    "    return y\n",
    "\n",
    "A = numpy.load(\"random-matrix.npy\")\n",
    "x = numpy.load(\"random-vector.npy\")\n",
    "y_mpi = matvec(comm, A, x)\n",
    "\n",
    "if rank == 0:\n",
    "    y = numpy.dot(A, x)\n",
    "    print(y_mpi)\n",
    "    print \"sum(y - y_mpi) =\", (y - y_mpi).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 6.40342716  3.62421625  3.42334637  3.99854639  4.95852419  6.13378754\r\n",
      "  5.33319708  5.42803442  5.12403754  4.87891654  2.38660728  6.72030412\r\n",
      "  4.05218475  3.37415974  3.90903001  5.82330226]\r\n",
      "sum(y - y_mpi) = 0.0\r\n"
     ]
    }
   ],
   "source": [
    "!mpirun -n 4 python mpi-matrix-vector.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 4: Sum of the elements in a vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# prepare some random data\n",
    "N = 128\n",
    "a = numpy.random.rand(N)\n",
    "numpy.save(\"random-vector.npy\", a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mpi-psum.py\n"
     ]
    }
   ],
   "source": [
    "%%file mpi-psum.py\n",
    "\n",
    "from mpi4py import MPI\n",
    "import numpy as np\n",
    "\n",
    "def psum(a):\n",
    "    r = MPI.COMM_WORLD.Get_rank()\n",
    "    size = MPI.COMM_WORLD.Get_size()\n",
    "    m = len(a) / size\n",
    "    locsum = np.sum(a[r*m:(r+1)*m])\n",
    "    rcvBuf = np.array(0.0, 'd')\n",
    "    MPI.COMM_WORLD.Allreduce([locsum, MPI.DOUBLE], [rcvBuf, MPI.DOUBLE], op=MPI.SUM)\n",
    "    return rcvBuf\n",
    "\n",
    "a = np.load(\"random-vector.npy\")\n",
    "s = psum(a)\n",
    "\n",
    "if MPI.COMM_WORLD.Get_rank() == 0:\n",
    "    print \"sum =\", s, \", numpy sum =\", a.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sum = 64.948311241 , numpy sum = 64.948311241\r\n"
     ]
    }
   ],
   "source": [
    "!mpirun -n 4 python mpi-psum.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* http://mpi4py.scipy.org\n",
    "\n",
    "* http://mpi4py.scipy.org/docs/usrman/tutorial.html\n",
    "\n",
    "* https://computing.llnl.gov/tutorials/mpi/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OpenMP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about OpenMP? OpenMP is a standard and widely used thread-based parallel API that unfortunately is **not** useful directly in Python. The reason is that the CPython implementation use a global interpreter lock, making it impossible to simultaneously run several Python threads. Threads are therefore not useful for parallel computing in Python, unless it is only used to wrap compiled code that do the OpenMP parallelization (Numpy can do something like that). \n",
    "\n",
    "This is clearly a limitation in the Python interpreter, and as a consequence all parallelization in Python must use processes (not threads).\n",
    "\n",
    "However, there is a way around this that is not that painful. When calling out to compiled code the GIL is released, and it is possible to write Python-like code in Cython where we can selectively release the GIL and do OpenMP computations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This system has 12 cores\n"
     ]
    }
   ],
   "source": [
    "N_core = multiprocessing.cpu_count()\n",
    "\n",
    "print(\"This system has %d cores\" % N_core)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a simple example that shows how OpenMP can be used via cython:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext cythonmagic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%cython -f -c-fopenmp --link-args=-fopenmp -c-g\n",
    "\n",
    "cimport cython\n",
    "cimport numpy\n",
    "from cython.parallel import prange, parallel\n",
    "cimport openmp\n",
    "\n",
    "def cy_openmp_test():\n",
    "\n",
    "    cdef int n, N\n",
    "\n",
    "    # release GIL so that we can use OpenMP\n",
    "    with nogil, parallel():\n",
    "        N = openmp.omp_get_num_threads()\n",
    "        n = openmp.omp_get_thread_num()\n",
    "        with gil:\n",
    "            print(\"Number of threads %d: thread number %d\" % (N, n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of threads 12: thread number 0\n",
      "Number of threads 12: thread number 10\n",
      "Number of threads 12: thread number 8\n",
      "Number of threads 12: thread number 4\n",
      "Number of threads 12: thread number 7\n",
      "Number of threads 12: thread number 3\n",
      "Number of threads 12: thread number 2\n",
      "Number of threads 12: thread number 1\n",
      "Number of threads 12: thread number 11\n",
      "Number of threads 12: thread number 9\n",
      "Number of threads 12: thread number 5\n",
      "Number of threads 12: thread number 6\n"
     ]
    }
   ],
   "source": [
    "cy_openmp_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: matrix vector multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# prepare some random data\n",
    "N = 4 * N_core\n",
    "\n",
    "M = numpy.random.rand(N, N)\n",
    "x = numpy.random.rand(N)\n",
    "y = numpy.zeros_like(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first look at a simple implementation of matrix-vector multiplication in Cython:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%cython\n",
    "\n",
    "cimport cython\n",
    "cimport numpy\n",
    "import numpy\n",
    "\n",
    "@cython.boundscheck(False)\n",
    "@cython.wraparound(False)\n",
    "def cy_matvec(numpy.ndarray[numpy.float64_t, ndim=2] M, \n",
    "              numpy.ndarray[numpy.float64_t, ndim=1] x, \n",
    "              numpy.ndarray[numpy.float64_t, ndim=1] y):\n",
    "\n",
    "    cdef int i, j, n = len(x)\n",
    "\n",
    "    for i from 0 <= i < n:\n",
    "        for j from 0 <= j < n:\n",
    "            y[i] += M[i, j] * x[j]\n",
    "            \n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check that we get the same results\n",
    "y = numpy.zeros_like(x)\n",
    "cy_matvec(M, x, y)\n",
    "numpy.dot(M, x) - y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100000 loops, best of 3: 2.93 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit numpy.dot(M, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100000 loops, best of 3: 5.4 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit cy_matvec(M, x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Cython implementation here is a bit slower than numpy.dot, but not by much, so if we can use multiple cores with OpenMP it should be possible to beat the performance of numpy.dot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%cython -f -c-fopenmp --link-args=-fopenmp -c-g\n",
    "\n",
    "cimport cython\n",
    "cimport numpy\n",
    "from cython.parallel import parallel\n",
    "cimport openmp\n",
    "\n",
    "@cython.boundscheck(False)\n",
    "@cython.wraparound(False)\n",
    "def cy_matvec_omp(numpy.ndarray[numpy.float64_t, ndim=2] M, \n",
    "                  numpy.ndarray[numpy.float64_t, ndim=1] x, \n",
    "                  numpy.ndarray[numpy.float64_t, ndim=1] y):\n",
    "\n",
    "    cdef int i, j, n = len(x), N, r, m\n",
    "\n",
    "    # release GIL, so that we can use OpenMP\n",
    "    with nogil, parallel():\n",
    "        N = openmp.omp_get_num_threads()\n",
    "        r = openmp.omp_get_thread_num()\n",
    "        m = n / N\n",
    "        \n",
    "        for i from 0 <= i < m:\n",
    "            for j from 0 <= j < n:\n",
    "                y[r * m + i] += M[r * m + i, j] * x[j]\n",
    "\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check that we get the same results\n",
    "y = numpy.zeros_like(x)\n",
    "cy_matvec_omp(M, x, y)\n",
    "numpy.dot(M, x) - y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100000 loops, best of 3: 2.95 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit numpy.dot(M, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 loops, best of 3: 209 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit cy_matvec_omp(M, x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, this implementation is much slower than numpy.dot for this problem size, because of overhead associated with OpenMP and threading, etc. But let's look at the how the different implementations compare with larger matrix sizes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "N_vec  = numpy.arange(25, 2000, 25) * N_core"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "duration_ref = numpy.zeros(len(N_vec))\n",
    "duration_cy = numpy.zeros(len(N_vec))\n",
    "duration_cy_omp = numpy.zeros(len(N_vec))\n",
    "\n",
    "for idx, N in enumerate(N_vec):\n",
    "    \n",
    "    M = numpy.random.rand(N, N)\n",
    "    x = numpy.random.rand(N)\n",
    "    y = numpy.zeros_like(x)\n",
    "    \n",
    "    t0 = time.time()\n",
    "    numpy.dot(M, x)\n",
    "    duration_ref[idx] = time.time() - t0\n",
    "    \n",
    "    t0 = time.time()\n",
    "    cy_matvec(M, x, y)\n",
    "    duration_cy[idx] = time.time() - t0\n",
    "    \n",
    "    t0 = time.time()\n",
    "    cy_matvec_omp(M, x, y)\n",
    "    duration_cy_omp[idx] = time.time() - t0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 39400 characters]",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7ad90badd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(12, 6))\n",
    "\n",
    "ax.loglog(N_vec, duration_ref, label='numpy')\n",
    "ax.loglog(N_vec, duration_cy, label='cython')\n",
    "ax.loglog(N_vec, duration_cy_omp, label='cython+openmp')\n",
    "\n",
    "ax.legend(loc=2)\n",
    "ax.set_yscale(\"log\")\n",
    "ax.set_ylabel(\"matrix-vector multiplication duration\")\n",
    "ax.set_xlabel(\"matrix size\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For large problem sizes the the cython+OpenMP implementation is faster than numpy.dot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this simple implementation, the speedup for large problem sizes is about:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0072232987815148"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "((duration_ref / duration_cy_omp)[-10:]).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Obviously one could do a better job with more effort, since the theoretical limit of the speed-up is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N_core"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " * http://openmp.org\n",
    " * http://docs.cython.org/src/userguide/parallelism.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OpenCL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OpenCL is an API for heterogenous computing, for example using GPUs for numerical computations. There is a python package called `pyopencl` that allows OpenCL code to be compiled, loaded and executed on the compute units completely from within Python. This is a nice way to work with OpenCL, because the time-consuming computations should be done on the compute units in compiled code, and in this Python only server as a control language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting opencl-dense-mv.py\n"
     ]
    }
   ],
   "source": [
    "%%file opencl-dense-mv.py\n",
    "\n",
    "import pyopencl as cl\n",
    "import numpy\n",
    "import time\n",
    "\n",
    "# problem size\n",
    "n = 10000\n",
    "\n",
    "# platform\n",
    "platform_list = cl.get_platforms()\n",
    "platform = platform_list[0]\n",
    "\n",
    "# device\n",
    "device_list = platform.get_devices()\n",
    "device = device_list[0]\n",
    "\n",
    "if False:\n",
    "    print(\"Platform name:\" + platform.name)\n",
    "    print(\"Platform version:\" + platform.version)\n",
    "    print(\"Device name:\" + device.name)\n",
    "    print(\"Device type:\" + cl.device_type.to_string(device.type))\n",
    "    print(\"Device memory: \" + str(device.global_mem_size//1024//1024) + ' MB')\n",
    "    print(\"Device max clock speed:\" + str(device.max_clock_frequency) + ' MHz')\n",
    "    print(\"Device compute units:\" + str(device.max_compute_units))\n",
    "\n",
    "# context\n",
    "ctx = cl.Context([device]) # or we can use cl.create_some_context()\n",
    "\n",
    "# command queue\n",
    "queue = cl.CommandQueue(ctx)\n",
    "\n",
    "# kernel\n",
    "KERNEL_CODE = \"\"\"\n",
    "//\n",
    "// Matrix-vector multiplication: r = m * v\n",
    "//\n",
    "#define N %(mat_size)d\n",
    "__kernel\n",
    "void dmv_cl(__global float *m, __global float *v, __global float *r)\n",
    "{\n",
    "    int i, gid = get_global_id(0);\n",
    "    \n",
    "    r[gid] = 0;\n",
    "    for (i = 0; i < N; i++)\n",
    "    {\n",
    "        r[gid] += m[gid * N + i] * v[i];\n",
    "    }\n",
    "}\n",
    "\"\"\"\n",
    "\n",
    "kernel_params = {\"mat_size\": n}\n",
    "program = cl.Program(ctx, KERNEL_CODE % kernel_params).build()\n",
    "\n",
    "# data\n",
    "A = numpy.random.rand(n, n)\n",
    "x = numpy.random.rand(n, 1)\n",
    "\n",
    "# host buffers\n",
    "h_y = numpy.empty(numpy.shape(x)).astype(numpy.float32)\n",
    "h_A = numpy.real(A).astype(numpy.float32)\n",
    "h_x = numpy.real(x).astype(numpy.float32)\n",
    "\n",
    "# device buffers\n",
    "mf = cl.mem_flags\n",
    "d_A_buf = cl.Buffer(ctx, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=h_A)\n",
    "d_x_buf = cl.Buffer(ctx, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=h_x)\n",
    "d_y_buf = cl.Buffer(ctx, mf.WRITE_ONLY, size=h_y.nbytes)\n",
    "\n",
    "# execute OpenCL code\n",
    "t0 = time.time()\n",
    "event = program.dmv_cl(queue, h_y.shape, None, d_A_buf, d_x_buf, d_y_buf)\n",
    "event.wait()\n",
    "cl.enqueue_copy(queue, h_y, d_y_buf)\n",
    "t1 = time.time()\n",
    "\n",
    "print \"opencl elapsed time =\", (t1-t0)\n",
    "\n",
    "# Same calculation with numpy\n",
    "t0 = time.time()\n",
    "y = numpy.dot(h_A, h_x)\n",
    "t1 = time.time()\n",
    "\n",
    "print \"numpy elapsed time =\", (t1-t0)\n",
    "\n",
    "# see if the results are the same\n",
    "print \"max deviation =\", numpy.abs(y-h_y).max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/pyopencl-2012.1-py2.7-linux-x86_64.egg/pyopencl/__init__.py:36: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.\n",
      "  \"to see more.\", CompilerWarning)\n",
      "opencl elapsed time = 0.0188570022583\n",
      "numpy elapsed time = 0.0755031108856\n",
      "max deviation = 0.0136719\n"
     ]
    }
   ],
   "source": [
    "!python opencl-dense-mv.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* http://mathema.tician.de/software/pyopencl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/json": {
       "Software versions": [
        {
         "module": "Python",
         "version": "3.3.2+ (default, Oct  9 2013, 14:50:09) [GCC 4.8.1]"
        },
        {
         "module": "IPython",
         "version": "2.0.0-b1"
        },
        {
         "module": "OS",
         "version": "posix [linux]"
        },
        {
         "module": "numpy",
         "version": "1.9.0.dev-d4c7c3a"
        },
        {
         "module": "mpi4py",
         "version": "1.3.1"
        },
        {
         "module": "Cython",
         "version": "0.20.post0"
        }
       ]
      },
      "text/html": [
       "<table><tr><th>Software</th><th>Version</th></tr><tr><td>Python</td><td>3.3.2+ (default, Oct  9 2013, 14:50:09) [GCC 4.8.1]</td></tr><tr><td>IPython</td><td>2.0.0-b1</td></tr><tr><td>OS</td><td>posix [linux]</td></tr><tr><td>numpy</td><td>1.9.0.dev-d4c7c3a</td></tr><tr><td>mpi4py</td><td>1.3.1</td></tr><tr><td>Cython</td><td>0.20.post0</td></tr><tr><td colspan='2'>Mon Mar 17 17:32:10 2014 JST</td></tr></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{|l|l|}\\hline\n",
       "{\\bf Software} & {\\bf Version} \\\\ \\hline\\hline\n",
       "Python & 3.3.2+ (default, Oct  9 2013, 14:50:09) [GCC 4.8.1] \\\\ \\hline\n",
       "IPython & 2.0.0-b1 \\\\ \\hline\n",
       "OS & posix [linux] \\\\ \\hline\n",
       "numpy & 1.9.0.dev-d4c7c3a \\\\ \\hline\n",
       "mpi4py & 1.3.1 \\\\ \\hline\n",
       "Cython & 0.20.post0 \\\\ \\hline\n",
       "\\hline \\multicolumn{2}{|l|}{Mon Mar 17 17:32:10 2014 JST} \\\\ \\hline\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "Software versions\n",
       "Python 3.3.2+ (default, Oct  9 2013, 14:50:09) [GCC 4.8.1]\n",
       "IPython 2.0.0-b1\n",
       "OS posix [linux]\n",
       "numpy 1.9.0.dev-d4c7c3a\n",
       "mpi4py 1.3.1\n",
       "Cython 0.20.post0\n",
       "\n",
       "Mon Mar 17 17:32:10 2014 JST"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%load_ext version_information\n",
    "\n",
    "%version_information numpy, mpi4py, Cython"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
