adopt myst

run rst2myst, with minimal manual formatting fixes

docs/troubleshooting/logs.md

@@ -0,0 +1,105 @@
+(troubleshooting-logs)=
+
+# Looking at Logs
+
+**Logs** are extremely useful in piecing together what went wrong when things go wrong.
+They contain a forensic record of what individual pieces of software were doing
+before things went bad, and can help us understand the problem so we can fix it.
+
+TLJH collects logs from JupyterHub, Traefik Proxy, & from each individual
+user's notebook server. All the logs are accessible via [journalctl](https://www.freedesktop.org/software/systemd/man/journalctl.html).
+The installer also writes logs to disk, to help with cases where the
+installer did not succeed.
+
+:::{warning}
+If you are providing a snippet from the logs to someone else to help debug
+a problem you might have, be careful to redact any private information (such
+as usernames) from the snippet first!
+:::
+
+(troubleshooting-logs-installer)=
+
+## Installer Logs
+
+The JupyterHub installer writes log messages to `/opt/tljh/installer.log`.
+This is very useful if the installation fails for any reason.
+
+(troubleshoot-logs-jupyterhub)=
+
+## JupyterHub Logs
+
+JupyterHub is responsible for user authentication, & starting / stopping user
+notebook servers. When there is a general systemic issue with JupyterHub (rather
+than a specific issue with a particular user's notebook), looking at the JupyterHub
+logs is a great first step.
+
+```bash
+sudo journalctl -u jupyterhub
+```
+
+This command displays logs from JupyterHub itself. See {ref}`journalctl_tips`
+for tips on navigating the logs.
+
+(troubleshooting-logs-traefik)=
+
+## Traefik Proxy Logs
+
+[traefik](https://traefik.io/) redirects traffic to JupyterHub / user notebook servers
+as necessary & handles HTTPS. Look at this if all you can see in your browser
+is one line cryptic error messages, or if you are having trouble with HTTPS.
+
+```bash
+sudo journalctl -u traefik
+```
+
+This command displays logs from Traefik. See {ref}`journalctl_tips`
+for tips on navigating the logs.
+
+## User Server Logs
+
+Each user gets their own notebook server, and this server also produces logs.
+Looking at these can be useful when a user can launch their server but run into
+problems after that.
+
+```bash
+sudo journalctl -u jupyter-<name-of-user>
+```
+
+This command displays logs from the given user's notebook server. You can get a
+list of all users from the "users" button at the top-right of the Admin page.
+See {ref}`journalctl_tips` for tips on navigating the logs.
+
+(journalctl-tips)=
+
+## journalctl tips
+
+`journalctl` has a lot of options to make your life as an administrator
+easier. Here are some very basic tips on effective `journalctl` usage.
+
+1. When looking at full logs (via `sudo journalctl -u <some-name>`), the output
+   usually does not fit into one screen. Hence, it is _paginated_ with
+   [less](<https://en.wikipedia.org/wiki/Less_(Unix)>). This allows you to
+   scroll up / down, search for specific words, etc. Some common keyboard shortcuts
+   are:
+
+   - Arrow keys to move up / down / left / right
+   - `G` to navigate to the end of the logs
+   - `g` to navigate to the start of the logs
+   - `/` followed by a string to search for & `enter` key to search the logs
+     from current position on screen to the end of the logs. If there are multiple
+     results, you can use `n` key to jump to the next search result. Use `?`
+     instead of `/` to search backwards from current position
+   - `q` or `Ctrl + C` to exit
+
+   There are plenty of [other commands & options](https://linux.die.net/man/1/less)
+   to explore if you wish.
+
+2. Add `-f` to any `journalctl` command to view live logging output
+   that updates as new log lines are written. This is extremely useful when
+   actively debugging an issue.
+
+   For example, to watch live logs of JupyterHub, you can run:
+
+   ```bash
+   sudo journalctl -u jupyterhub -f
+   ```