Add a troubleshooting section

docs/index.rst

@@ -41,3 +41,14 @@ Guides provide in-depth explanations of specific topics.
    guides/requirements
    guides/install
    guides/admin
+
+Troubleshooting
+===============
+
+In time, all systems have issues that need to be debugged. Troubleshooting
+guides help you find what is broken & hopefully fix it.
+
+.. toctree::
+   :titlesonly:
+
+   troubleshooting/logs

docs/troubleshooting/logs.rst

@@ -0,0 +1,95 @@
+.. _troubleshoot_logs:
+
+Looking at Logs
+===============
+
+**Logs** are extremely useful in piecing together what went wrong when things go wrong.
+They contain a forensic record of what individual pieces of software were doing
+before things went bad, and can help us understand the problem so we can fix it.
+
+TLJH collects logs from JupyterHub, Configurable HTTP Proxy & from each individual
+user's notebook server. All the logs are accessible via `journalctl <https://www.freedesktop.org/software/systemd/man/journalctl.html>`_.
+
+JupyterHub Logs
+---------------
+
+JupyterHub is responsible for user authentication, & starting / stopping user
+notebook servers. When there is a general systemic issue with JupyterHub (rather
+than a specific issue with a particular user's notebook), looking at the JupyterHub
+logs is a great first step.
+
+.. warning::
+
+   If you are providing a snippet from the logs to someone else to help debug
+   a problem you might have, be careful to redact any private information (such
+   as usernames) from the snippet first!
+
+.. code-block:: bash
+
+   sudo journalctl -u jupyterhub
+
+This command displays logs from JupyterHub itself. See :ref:`journactl_tips`
+for tips on navigating the logs.
+
+Configurable HTTP Proxy Logs
+----------------------------
+
+Configurable HTTP Proxy redirects traffic to JupyterHub / user notebook servers
+as necessary & handles HTTPS. It usually is the least problematic of the components,
+but things do go wrong sometimes!
+
+.. code-block:: bash
+
+   sudo journalctl -u configurable-http-proxy
+
+This command displays logs from Configurable HTTP Proxy. See :ref:`journactl_tips`
+for tips on navigating the logs.
+
+User Server Logs
+----------------
+
+Each user gets their own notebook server, and this server also produces logs.
+Looking at these can be useful when a user can launch their server but run into
+problems after that.
+
+.. code-block:: bash
+
+   sudo journalctl -u jupyter-<name-of-user>
+
+This command displays logs from the given user's notebook server. See :ref:`journactl_tips`
+for tips on navigating the logs.
+
+.. _journalctl_tips:
+journalctl tips
+---------------
+
+``journalctl`` has a lot of options to make your life as an administrator
+easier. Here are some very basic tips on effective ``journalctl`` usage.
+
+1. When looking at full logs (via ``sudo journalctl -u <some-name>``), the output
+   usually does not fit into one screen. Hence, it is *paginated* with
+   `less <https://en.wikipedia.org/wiki/Less_(Unix)>`_. This allows you to
+   scroll up / down, search for specific words, etc. Some common keyboard shortcuts
+   are:
+
+   * Arrow keys to move up / down / left / right
+   * ``G`` to navigate to the end of the logs
+   * ``g`` to navigate to the start of the logs
+   * ``/`` followed by a string to search for & ``enter`` key to search the logs
+     from current position on screen to the end of the logs. If there are multiple
+     results, you can use ``n`` key to jump to the next search result. Use ``?``
+     instead of ``/`` to search backwards from current position
+   * ``q`` or ``Ctrl + C`` to exit
+
+   There are plenty of `other commands & options <https://linux.die.net/man/1/less>`_
+   to explore if you wish.
+
+2. Add ``-f`` to any ``journalctl`` command to view live logging output
+   that updates as new log lines are written. This is extremely useful when
+   actively debugging an issue.
+
+   For example, to watch live logs of JupyterHub, you can run:
+
+   .. code-block:: bash
+
+      sudo journalctl -u jupyterhub -f