mirror of
https://github.com/jupyterhub/the-littlest-jupyterhub.git
synced 2025-12-18 21:54:05 +08:00
adopt myst
run rst2myst, with minimal manual formatting fixes
This commit is contained in:
Min RK
@@ -1,25 +1,25 @@
|
||||
===============
|
||||
Troubleshooting
|
||||
===============
|
||||
# Troubleshooting
|
||||
|
||||
In time, all systems have issues that need to be debugged. Troubleshooting
|
||||
guides help you find what is broken & hopefully fix it.
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
:caption: Troubleshooting
|
||||
```{toctree}
|
||||
:caption: Troubleshooting
|
||||
:titlesonly: true
|
||||
|
||||
logs
|
||||
restart
|
||||
logs
|
||||
restart
|
||||
```
|
||||
|
||||
Often, your issues are not related to TLJH itself but to the cloud provider
|
||||
your server is running on. We have some documentation on common issues you
|
||||
might run into with various providers and how to fix them. We welcome contributions
|
||||
here to better support your favorite provider!
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
```{toctree}
|
||||
:titlesonly: true
|
||||
|
||||
providers/google
|
||||
providers/amazon
|
||||
providers/custom
|
||||
providers/google
|
||||
providers/amazon
|
||||
providers/custom
|
||||
```
|
||||
105
docs/troubleshooting/logs.md
Normal file
105
docs/troubleshooting/logs.md
Normal file
@@ -0,0 +1,105 @@
|
||||
(troubleshooting-logs)=
|
||||
|
||||
# Looking at Logs
|
||||
|
||||
**Logs** are extremely useful in piecing together what went wrong when things go wrong.
|
||||
They contain a forensic record of what individual pieces of software were doing
|
||||
before things went bad, and can help us understand the problem so we can fix it.
|
||||
|
||||
TLJH collects logs from JupyterHub, Traefik Proxy, & from each individual
|
||||
user's notebook server. All the logs are accessible via [journalctl](https://www.freedesktop.org/software/systemd/man/journalctl.html).
|
||||
The installer also writes logs to disk, to help with cases where the
|
||||
installer did not succeed.
|
||||
|
||||
:::{warning}
|
||||
If you are providing a snippet from the logs to someone else to help debug
|
||||
a problem you might have, be careful to redact any private information (such
|
||||
as usernames) from the snippet first!
|
||||
:::
|
||||
|
||||
(troubleshooting-logs-installer)=
|
||||
|
||||
## Installer Logs
|
||||
|
||||
The JupyterHub installer writes log messages to `/opt/tljh/installer.log`.
|
||||
This is very useful if the installation fails for any reason.
|
||||
|
||||
(troubleshoot-logs-jupyterhub)=
|
||||
|
||||
## JupyterHub Logs
|
||||
|
||||
JupyterHub is responsible for user authentication, & starting / stopping user
|
||||
notebook servers. When there is a general systemic issue with JupyterHub (rather
|
||||
than a specific issue with a particular user's notebook), looking at the JupyterHub
|
||||
logs is a great first step.
|
||||
|
||||
```bash
|
||||
sudo journalctl -u jupyterhub
|
||||
```
|
||||
|
||||
This command displays logs from JupyterHub itself. See {ref}`journalctl_tips`
|
||||
for tips on navigating the logs.
|
||||
|
||||
(troubleshooting-logs-traefik)=
|
||||
|
||||
## Traefik Proxy Logs
|
||||
|
||||
[traefik](https://traefik.io/) redirects traffic to JupyterHub / user notebook servers
|
||||
as necessary & handles HTTPS. Look at this if all you can see in your browser
|
||||
is one line cryptic error messages, or if you are having trouble with HTTPS.
|
||||
|
||||
```bash
|
||||
sudo journalctl -u traefik
|
||||
```
|
||||
|
||||
This command displays logs from Traefik. See {ref}`journalctl_tips`
|
||||
for tips on navigating the logs.
|
||||
|
||||
## User Server Logs
|
||||
|
||||
Each user gets their own notebook server, and this server also produces logs.
|
||||
Looking at these can be useful when a user can launch their server but run into
|
||||
problems after that.
|
||||
|
||||
```bash
|
||||
sudo journalctl -u jupyter-<name-of-user>
|
||||
```
|
||||
|
||||
This command displays logs from the given user's notebook server. You can get a
|
||||
list of all users from the "users" button at the top-right of the Admin page.
|
||||
See {ref}`journalctl_tips` for tips on navigating the logs.
|
||||
|
||||
(journalctl-tips)=
|
||||
|
||||
## journalctl tips
|
||||
|
||||
`journalctl` has a lot of options to make your life as an administrator
|
||||
easier. Here are some very basic tips on effective `journalctl` usage.
|
||||
|
||||
1. When looking at full logs (via `sudo journalctl -u <some-name>`), the output
|
||||
usually does not fit into one screen. Hence, it is _paginated_ with
|
||||
[less](<https://en.wikipedia.org/wiki/Less_(Unix)>). This allows you to
|
||||
scroll up / down, search for specific words, etc. Some common keyboard shortcuts
|
||||
are:
|
||||
|
||||
- Arrow keys to move up / down / left / right
|
||||
- `G` to navigate to the end of the logs
|
||||
- `g` to navigate to the start of the logs
|
||||
- `/` followed by a string to search for & `enter` key to search the logs
|
||||
from current position on screen to the end of the logs. If there are multiple
|
||||
results, you can use `n` key to jump to the next search result. Use `?`
|
||||
instead of `/` to search backwards from current position
|
||||
- `q` or `Ctrl + C` to exit
|
||||
|
||||
There are plenty of [other commands & options](https://linux.die.net/man/1/less)
|
||||
to explore if you wish.
|
||||
|
||||
2. Add `-f` to any `journalctl` command to view live logging output
|
||||
that updates as new log lines are written. This is extremely useful when
|
||||
actively debugging an issue.
|
||||
|
||||
For example, to watch live logs of JupyterHub, you can run:
|
||||
|
||||
```bash
|
||||
sudo journalctl -u jupyterhub -f
|
||||
```
|
||||
@@ -1,112 +0,0 @@
|
||||
.. _troubleshooting/logs:
|
||||
|
||||
===============
|
||||
Looking at Logs
|
||||
===============
|
||||
|
||||
**Logs** are extremely useful in piecing together what went wrong when things go wrong.
|
||||
They contain a forensic record of what individual pieces of software were doing
|
||||
before things went bad, and can help us understand the problem so we can fix it.
|
||||
|
||||
TLJH collects logs from JupyterHub, Traefik Proxy, & from each individual
|
||||
user's notebook server. All the logs are accessible via `journalctl <https://www.freedesktop.org/software/systemd/man/journalctl.html>`_.
|
||||
The installer also writes logs to disk, to help with cases where the
|
||||
installer did not succeed.
|
||||
|
||||
.. warning::
|
||||
|
||||
If you are providing a snippet from the logs to someone else to help debug
|
||||
a problem you might have, be careful to redact any private information (such
|
||||
as usernames) from the snippet first!
|
||||
|
||||
.. _troubleshooting/logs#installer:
|
||||
|
||||
Installer Logs
|
||||
==============
|
||||
|
||||
The JupyterHub installer writes log messages to ``/opt/tljh/installer.log``.
|
||||
This is very useful if the installation fails for any reason.
|
||||
|
||||
.. _troubleshoot_logs_jupyterhub:
|
||||
|
||||
JupyterHub Logs
|
||||
===============
|
||||
|
||||
JupyterHub is responsible for user authentication, & starting / stopping user
|
||||
notebook servers. When there is a general systemic issue with JupyterHub (rather
|
||||
than a specific issue with a particular user's notebook), looking at the JupyterHub
|
||||
logs is a great first step.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo journalctl -u jupyterhub
|
||||
|
||||
This command displays logs from JupyterHub itself. See :ref:`journalctl_tips`
|
||||
for tips on navigating the logs.
|
||||
|
||||
.. _troubleshooting/logs/traefik:
|
||||
|
||||
Traefik Proxy Logs
|
||||
==================
|
||||
|
||||
`traefik <https://traefik.io/>`_ redirects traffic to JupyterHub / user notebook servers
|
||||
as necessary & handles HTTPS. Look at this if all you can see in your browser
|
||||
is one line cryptic error messages, or if you are having trouble with HTTPS.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo journalctl -u traefik
|
||||
|
||||
This command displays logs from Traefik. See :ref:`journalctl_tips`
|
||||
for tips on navigating the logs.
|
||||
|
||||
User Server Logs
|
||||
================
|
||||
|
||||
Each user gets their own notebook server, and this server also produces logs.
|
||||
Looking at these can be useful when a user can launch their server but run into
|
||||
problems after that.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo journalctl -u jupyter-<name-of-user>
|
||||
|
||||
This command displays logs from the given user's notebook server. You can get a
|
||||
list of all users from the "users" button at the top-right of the Admin page.
|
||||
See :ref:`journalctl_tips` for tips on navigating the logs.
|
||||
|
||||
.. _journalctl_tips:
|
||||
|
||||
journalctl tips
|
||||
===============
|
||||
|
||||
``journalctl`` has a lot of options to make your life as an administrator
|
||||
easier. Here are some very basic tips on effective ``journalctl`` usage.
|
||||
|
||||
1. When looking at full logs (via ``sudo journalctl -u <some-name>``), the output
|
||||
usually does not fit into one screen. Hence, it is *paginated* with
|
||||
`less <https://en.wikipedia.org/wiki/Less_(Unix)>`_. This allows you to
|
||||
scroll up / down, search for specific words, etc. Some common keyboard shortcuts
|
||||
are:
|
||||
|
||||
* Arrow keys to move up / down / left / right
|
||||
* ``G`` to navigate to the end of the logs
|
||||
* ``g`` to navigate to the start of the logs
|
||||
* ``/`` followed by a string to search for & ``enter`` key to search the logs
|
||||
from current position on screen to the end of the logs. If there are multiple
|
||||
results, you can use ``n`` key to jump to the next search result. Use ``?``
|
||||
instead of ``/`` to search backwards from current position
|
||||
* ``q`` or ``Ctrl + C`` to exit
|
||||
|
||||
There are plenty of `other commands & options <https://linux.die.net/man/1/less>`_
|
||||
to explore if you wish.
|
||||
|
||||
2. Add ``-f`` to any ``journalctl`` command to view live logging output
|
||||
that updates as new log lines are written. This is extremely useful when
|
||||
actively debugging an issue.
|
||||
|
||||
For example, to watch live logs of JupyterHub, you can run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo journalctl -u jupyterhub -f
|
||||
26
docs/troubleshooting/providers/amazon.md
Normal file
26
docs/troubleshooting/providers/amazon.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# Troubleshooting issues on Amazon Web Services
|
||||
|
||||
This is an incomplete list of issues people have run into when running
|
||||
TLJH on Amazon Web Services (AWS), and how they have fixed them!
|
||||
|
||||
## 'Connection Refused' error after restarting server
|
||||
|
||||
If you restarted your server from the EC2 Management Console & then try to access
|
||||
your JupyterHub from a browser, you might get a **Connection Refused** error.
|
||||
This is most likely because the **External IP** of your server has changed.
|
||||
|
||||
Check the **IPv4 Public IP** dislayed in the bottom of the `EC2 Management Console`
|
||||
screen for that instance matches the IP you are trying to access. If you have a
|
||||
domain name pointing to the IP address, you might have to change it to point to
|
||||
the new correct IP.
|
||||
|
||||
You can prevent public IP changes by [associating a static IP](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html)
|
||||
with your server. In the Amazon Web Services ecosystem, the public static IP
|
||||
addresses are handled under `Elastic IP addresses` category of AWS; these
|
||||
addresses are tied to the overall AWS account. [This guide](https://dzone.com/articles/assign-fixed-ip-aws-ec2) might be helpful. Notice
|
||||
there can be a cost to this. Although [the guide](https://dzone.com/articles/assign-fixed-ip-aws-ec2) is outdated (generally
|
||||
half that [price now](https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Addresses)),
|
||||
Amazon describes [here](https://aws.amazon.com/premiumsupport/knowledge-center/elastic-ip-charges/)
|
||||
how the Elastic IP address feature is free when associated with a running
|
||||
instance, but that you'll be charged by the hour for maintaining that specific
|
||||
IP address when it isn't associated with a running instance.
|
||||
@@ -1,32 +0,0 @@
|
||||
=============================================
|
||||
Troubleshooting issues on Amazon Web Services
|
||||
=============================================
|
||||
|
||||
This is an incomplete list of issues people have run into when running
|
||||
TLJH on Amazon Web Services (AWS), and how they have fixed them!
|
||||
|
||||
'Connection Refused' error after restarting server
|
||||
==================================================
|
||||
|
||||
If you restarted your server from the EC2 Management Console & then try to access
|
||||
your JupyterHub from a browser, you might get a **Connection Refused** error.
|
||||
This is most likely because the **External IP** of your server has changed.
|
||||
|
||||
Check the **IPv4 Public IP** dislayed in the bottom of the `EC2 Management Console`
|
||||
screen for that instance matches the IP you are trying to access. If you have a
|
||||
domain name pointing to the IP address, you might have to change it to point to
|
||||
the new correct IP.
|
||||
|
||||
You can prevent public IP changes by `associating a static IP
|
||||
<https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html>`_
|
||||
with your server. In the Amazon Web Services ecosystem, the public static IP
|
||||
addresses are handled under `Elastic IP addresses` category of AWS; these
|
||||
addresses are tied to the overall AWS account. `This guide
|
||||
<https://dzone.com/articles/assign-fixed-ip-aws-ec2>`_ might be helpful. Notice
|
||||
there can be a cost to this. Although `the guide
|
||||
<https://dzone.com/articles/assign-fixed-ip-aws-ec2>`_ is outdated (generally
|
||||
half that `price now <https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Addresses>`_),
|
||||
Amazon describes `here <https://aws.amazon.com/premiumsupport/knowledge-center/elastic-ip-charges/>`_
|
||||
how the Elastic IP address feature is free when associated with a running
|
||||
instance, but that you'll be charged by the hour for maintaining that specific
|
||||
IP address when it isn't associated with a running instance.
|
||||
@@ -1,8 +1,6 @@
|
||||
.. _troubleshooting/providers/custom:
|
||||
(troubleshooting-providers-custom)=
|
||||
|
||||
=========================================
|
||||
Troubleshooting issues on your own server
|
||||
=========================================
|
||||
# Troubleshooting issues on your own server
|
||||
|
||||
This is an incomplete list of issues people have run into
|
||||
when installing TLJH on their own servers, and ways they
|
||||
@@ -11,23 +9,22 @@ Before trying any of them, also consider whether turning your machine on and off
|
||||
and/or deleting the VM and starting over could solve the problem;
|
||||
it has done so on a surprisingly high number of occasions!
|
||||
|
||||
Outgoing HTTP proxy required
|
||||
============================
|
||||
## Outgoing HTTP proxy required
|
||||
|
||||
If your server is behind a firewall that requires a HTTP proxy to reach
|
||||
the internet, run these commands before running the installer
|
||||
|
||||
.. code-block:: bash
|
||||
```bash
|
||||
export http_proxy=<your_proxy-server>
|
||||
```
|
||||
|
||||
export http_proxy=<your_proxy-server>
|
||||
|
||||
HTTPS certificate interception
|
||||
==============================
|
||||
## HTTPS certificate interception
|
||||
|
||||
If your server is behind a firewall that intercepts HTTPS requests
|
||||
and re-signs them, you might have to explicitly tell TLJH which
|
||||
certificates to use.
|
||||
|
||||
.. code::
|
||||
|
||||
export REQUESTS_CA_BUNDLE=</directory/with/your/ssl/certificates>
|
||||
sudo npm config set cafile=</directory/with/your/ssl/certificates>
|
||||
```
|
||||
export REQUESTS_CA_BUNDLE=</directory/with/your/ssl/certificates>
|
||||
sudo npm config set cafile=</directory/with/your/ssl/certificates>
|
||||
```
|
||||
17
docs/troubleshooting/providers/google.md
Normal file
17
docs/troubleshooting/providers/google.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Troubleshooting issues on Google Cloud
|
||||
|
||||
This is an incomplete list of issues people have run into when running
|
||||
TLJH on Google Cloud, and how they have fixed them!
|
||||
|
||||
## 'Connection Refused' error after restarting server
|
||||
|
||||
If you restarted your server from the Google Cloud console & then try to access
|
||||
your JupyterHub from a browser, you might get a **Connection Refused** error.
|
||||
This is most likely because the **External IP** of your server has changed.
|
||||
|
||||
Check the **External IP** in the [Google Cloud Console -> Compute Engine -> VM instances](https://console.cloud.google.com/compute/instances) screen
|
||||
matches the IP you are trying to access. If you have a domain name pointing to the
|
||||
IP address, you might have to change it to point to the new correct IP.
|
||||
|
||||
You can prevent External IP changes by [reserving the static IP](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip)
|
||||
your server is using.
|
||||
@@ -1,22 +0,0 @@
|
||||
======================================
|
||||
Troubleshooting issues on Google Cloud
|
||||
======================================
|
||||
|
||||
This is an incomplete list of issues people have run into when running
|
||||
TLJH on Google Cloud, and how they have fixed them!
|
||||
|
||||
'Connection Refused' error after restarting server
|
||||
==================================================
|
||||
|
||||
If you restarted your server from the Google Cloud console & then try to access
|
||||
your JupyterHub from a browser, you might get a **Connection Refused** error.
|
||||
This is most likely because the **External IP** of your server has changed.
|
||||
|
||||
Check the **External IP** in the `Google Cloud Console -> Compute Engine -> VM instances
|
||||
<https://console.cloud.google.com/compute/instances>`_ screen
|
||||
matches the IP you are trying to access. If you have a domain name pointing to the
|
||||
IP address, you might have to change it to point to the new correct IP.
|
||||
|
||||
You can prevent External IP changes by `reserving the static IP
|
||||
<https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip>`_
|
||||
your server is using.
|
||||
27
docs/troubleshooting/restart.md
Normal file
27
docs/troubleshooting/restart.md
Normal file
@@ -0,0 +1,27 @@
|
||||
# Stopping and Restarting the JupyterHub Server
|
||||
|
||||
The user can **stop** the JupyterHub server using:
|
||||
|
||||
```console
|
||||
$ systemctl stop jupyterhub.service
|
||||
```
|
||||
|
||||
:::{warning}
|
||||
Keep in mind that other services that may also require stopping:
|
||||
|
||||
- The user's Jupyter server: jupyter-username.service
|
||||
- traefik.service
|
||||
|
||||
:::
|
||||
|
||||
The user may **restart** JupyterHub and Traefik services by running:
|
||||
|
||||
```console
|
||||
$ sudo tljh-config reload proxy
|
||||
```
|
||||
|
||||
This calls systemctl and restarts Traefik. The user may call systemctl and restart only the JupyterHub using the command:
|
||||
|
||||
```console
|
||||
$ sudo tljh-config reload hub
|
||||
```
|
||||
@@ -1,29 +0,0 @@
|
||||
|
||||
=============================================
|
||||
Stopping and Restarting the JupyterHub Server
|
||||
=============================================
|
||||
|
||||
The user can **stop** the JupyterHub server using:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ systemctl stop jupyterhub.service
|
||||
|
||||
.. warning::
|
||||
|
||||
Keep in mind that other services that may also require stopping:
|
||||
|
||||
* The user's Jupyter server: jupyter-username.service
|
||||
* Traefik.service
|
||||
|
||||
The user may **restart** JupyterHub and Traefik services by running:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ sudo tljh-config reload proxy
|
||||
|
||||
This calls systemctl and restarts Traefik. The user may call systemctl and restart only the JupyterHub using the command:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ sudo tljh-config reload hub
|
||||
Reference in New Issue
Block a user