Merge pull request #77 from jupyterhub/config

Add tljh-config command

.circleci/config.yml

@@ -77,6 +77,13 @@ jobs:
                     python3 .circleci/integration-test.py copy . /srv/src
                     python3 .circleci/integration-test.py run 'python3 /srv/src/bootstrap/bootstrap.py'
 
+
+            - run:
+                name: switch to dummyauthenticator
+                command: |
+                    python3 .circleci/integration-test.py run '/opt/tljh/hub/bin/tljh-config set auth.type dummyauthenticator.DummyAuthenticator'
+                    python3 .circleci/integration-test.py run '/opt/tljh/hub/bin/tljh-config reload'
+
             - run:
                 name: print systemd status + logs
                 command: |

docs/guides/admin.rst

@@ -1,57 +0,0 @@
-.. _admin_access:
-
-=====================
-Administrative Access
-=====================
-
-In The Littlest JupyterHub, we try to allow users to do as many administrative
-tasks as possible within JupyterHub itself. Admin users can:
-
-1. Have full root access with passwordless ``sudo``
-2. Install system-wide packages with ``apt``
-3. Install ``conda`` / ``pip`` packages for all JupyterHub users
-4. Change the amount of RAM / CPU available to each user, and more!
-
-By default, there are no admin users. You should add some after installation.
-
-Adding admin users
-==================
-
-Admin users are specified in the `YAML <https://en.wikipedia.org/wiki/YAML>`_
-config file at ``/opt/tljh/config.yaml``. This file is created upon installing
-tljh.
-
-1. Open the ``config.yaml`` file for editing.
-
-   .. code-block:: bash
-
-      sudo nano /opt/tljh/config.yaml
-
-2. Add usernames that should have admin access.
-
-   .. code-block:: yaml
-
-      users:
-        admin:
-          - user1
-          - user2
-
-   Be careful around the syntax - indentation matters, and you should be using
-   spaces and not tabs.
-
-   When you are done, save the file and exit. In ``nano``, you can do this with
-   ``Ctrl+X`` key.
-
-3. When you are sure the format is ok, restart JupyterHub to let the config take
-   effect.
-
-   .. code-block:: bash
-
-      sudo systemctl restart jupyterhub
-
-This should give you admin access from JupyterHub! You can verify this by:
-
-1. Opening a Terminal in your JupyterHub and checking if ``sudo`` works
-2. Opening your JupyterHub ``Control Panel`` and checking for the **Admin** tab
-
-From now on, you can use the JupyterHub to do most configuration changes.

docs/howto/admin-users.rst

@@ -0,0 +1,55 @@
+.. _howto/admin-users:
+
+========================
+Add / Remove admin users
+========================
+
+Admin users in TLJH have the following powers:
+
+#. Full root access to the server with passwordless ``sudo``.
+   This lets them do literally whatever they want in the server
+#. Access servers / home directories of all other users
+#. Install new packages for everyone with ``conda``, ``pip`` or ``apt``
+#. Change configuration of TLJH
+
+This is a lot of power, so make sure you know who you are giving it
+to. Admin users should have decent passwords / secure logni mechanisms,
+so attackers can not easily gain control of the system.
+
+Make sure an admin user is present
+==================================
+
+You should make sure an admin user is present when you **install** TLJH
+the very first time. The ``:ref:`--admin <topic/customizing-installer/admin>```
+flag passed to the installer does this. If you had forgotten to do so, the
+easiest way to fix this is to run the installer again.
+
+Adding new admin users
+======================
+
+New admin users can be added by executing the following commands on an
+admin terminal:
+
+.. code-block:: bash
+
+   sudo -E tljh-config add-item users.admin <username>
+   sudo -E tljh-config reload
+
+If the user is already using the JupyterHub, they might have to stop and
+start their server from the control panel to gain new powers.
+
+Removing admin users
+====================
+
+You can remove an existing admin user by executing the following commands in
+an admin terminal:
+
+.. code-block:: bash
+
+   sudo -E tljh-config remove-item users.admin <username>
+   sudo -E tljh-config reload
+
+If the user is already using the JupyterHub, they will continue to have
+some of their admin powers until their server is stopped. Another admin
+can force their server to stop by clicking 'Stop Server' in the admin
+panel.

docs/howto/auth/dummy.rst

@@ -0,0 +1,43 @@
+.. _howto/auth/dummy:
+
+=====================================================
+Authenticate *any* user with a single shared password
+=====================================================
+
+The **Dummy Authenticator** lets *any* user log in with the given password.
+This authenticator is **extremely insecure**, so do not use it if you can
+avoid it.
+
+Enabling the authenticator
+==========================
+
+1. Always use DummyAuthenticator with a password. You can communicate this 
+   password to all your users via an out of band mechanism (like writing on
+   a whiteboard). Once you have selected a password, configure TLJH to use
+   the password by executing the following from an admin console.
+
+.. code-block:: bash
+   
+   sudo -E tljh-config set auth.DummyAuthenticator.password <password>
+
+   Remember to replace ``<password>`` with the password you choose.
+
+2. Enable the authenticator and reload config to apply configuration:
+
+   sudo -E tljh-config set auth.type dummyauthenticator.DummyAuthenticator
+   sudo -E tljh-config reload
+
+Users who are currently logged in will continue to be logged in. When they 
+log out and try to log back in, they will be asked to provide a username and
+password.
+
+Changing the password
+=====================
+
+The password used by DummyAuthenticator can be changed with the following
+commands:
+
+.. code-block:: bash
+
+   tljh-config set auth.DummyAuthenticator.password <new-password>
+   tljh-config reload

docs/index.rst

@@ -54,9 +54,18 @@ How-To guides answer the question 'How do I...?' for a lot of topics.
    :titlesonly:
 
    howto/user-environment
+   howto/admin-users
    howto/notebook-interfaces
    howto/resource-estimation
 
+We have a special set of How-To Guides on using various forms of authentication
+with your JupyterHub.
+
+.. toctree::
+   :titlesonly:
+
+   howto/auth/dummy
+
 Topic Guides
 ============
 
@@ -66,9 +75,9 @@ Topic guides provide in-depth explanations of specific topics.
    :titlesonly:
 
    topic/requirements
-   guides/admin
    topic/security
    topic/customizing-installer
+   topic/tljh-config
 
 
 Troubleshooting

docs/topic/customizing-installer.rst

@@ -13,6 +13,8 @@ is executed as:
 
 This page documents the various options you can pass as commandline parameters to the installer.
 
+.. _topic/customizing-installer/admin:
+
 Adding admin users
 ===================
 

docs/topic/tljh-config.rst

@@ -0,0 +1,81 @@
+.. _topic/tljh-config:
+
+=====================================
+Configuring TLJH with ``tljh-config``
+=====================================
+
+``tljh-config`` is the commandline program used to make configuration
+changes to TLJH. 
+
+Running ``tljh-config``
+======================`
+
+You can run ``tljh-config`` in two ways:
+
+#. From inside a terminal in JupyterHub while logged in as an admin user.
+   This method is **recommended**.
+
+#. By directly calling ``/opt/tljh/hub/bin/tljh-config`` as root when
+   logged in to the server via other means (such as SSH). This is an
+   advanced use case, and not covered much in this guide.
+
+Set a configuration property
+============================
+
+TLJH's configuration is organized in a nested tree structure. You can
+set a particular property with the following command:
+
+.. code-block:: bash
+
+   sudo -E tljh-config set <property-path> <value>
+
+
+where:
+
+#. ``<property-path>`` is a dot-separated path to the property you want
+   to set.
+#. ``<value>`` is the value you want to set the property to.
+
+For example, to set the password for the DummyAuthenticator, you
+need to set the ``auth.DummyAuthenticator.password`` property. You would
+do so with the following:
+
+.. code-block:: bash
+
+   sudo -E tljh-config set auth.DummyAuthenticator.password mypassword
+
+
+This can only set string and numerical properties, not lists.
+
+View current configuration
+==========================
+
+To see the current configuration, you can run the following command:
+
+.. code-block:: bash
+
+   sudo -E tljh-config show
+
+This will print the current configuration of your TLJH. This is very
+useful when asking for support!
+
+Reloading JupyterHub to apply configuration
+===========================================
+
+After modifying the configuration, you need to reload JupyterHub for
+it to take effect. You can do so with:
+
+.. code-block:: bash
+
+   sudo -E tljh-config reload
+
+This should not affect any running users. The JupyterHub will be
+restarted and loaded with the new configuration.
+
+Advanced: ``config.yaml``
+=========================
+
+``tljh-config`` is a simple program that modifies the contents of the 
+``config.yaml`` file located at ``/opt/tljh/config.yaml``. ``tljh-config`` 
+is the recommended method of editing / viewing configuration since editing
+YAML by hand in a terminal text editor is a large source of errors.

integration-tests/requirements.txt

@@ -1,2 +1,3 @@
 pytest
-requests
+pytest-asyncio
+git+https://github.com/yuvipanda/hubtraf.git

integration-tests/test_hub.py

@@ -1,5 +1,106 @@
 import requests
+from hubtraf.user import User
+from hubtraf.auth.dummy import login_dummy
+import secrets
+import pytest
+from functools import partial
+import asyncio
+import pwd
+import grp
+import sys
+
 
 def test_hub_up():
     r = requests.get('http://127.0.0.1')
     r.raise_for_status()
+
+
+@pytest.mark.asyncio
+async def test_user_code_execute():
+    """
+    User logs in, starts a server & executes code
+    """
+    # This *must* be localhost, not an IP
+    # aiohttp throws away cookies if we are connecting to an IP!
+    hub_url = 'http://localhost'
+    username = secrets.token_hex(8)
+
+    async with User(username, hub_url, partial(login_dummy, password='')) as u:
+            await u.login()
+            await u.ensure_server()
+            await u.start_kernel()
+            await u.assert_code_output("5 * 4", "20", 5, 5)
+
+            # Assert that the user exists
+            assert pwd.getpwnam(f'jupyter-{username}') is not None
+
+
+@pytest.mark.asyncio
+async def test_user_admin_add():
+    """
+    User is made an admin, logs in and we check if they are in admin group
+    """
+    # This *must* be localhost, not an IP
+    # aiohttp throws away cookies if we are connecting to an IP!
+    hub_url = 'http://localhost'
+    username = secrets.token_hex(8)
+
+    tljh_config_path = [sys.executable, '-m', 'tljh.config']
+
+    assert 0 == await (await asyncio.create_subprocess_exec(*tljh_config_path, 'add-item', 'users.admin', username)).wait()
+    assert 0 == await (await asyncio.create_subprocess_exec(*tljh_config_path, 'reload')).wait()
+
+    # FIXME: wait for reload to finish & hub to come up
+    # Should be part of tljh-config reload
+    await asyncio.sleep(1)
+    async with User(username, hub_url, partial(login_dummy, password='')) as u:
+            await u.login()
+            await u.ensure_server()
+
+            # Assert that the user exists
+            assert pwd.getpwnam(f'jupyter-{username}') is not None
+
+            # Assert that the user has admin rights
+            assert f'jupyter-{username}' in grp.getgrnam('jupyterhub-admins').gr_mem
+
+
+@pytest.mark.asyncio
+async def test_user_admin_remove():
+    """
+    User is made an admin, logs in and we check if they are in admin group.
+
+    Then we remove them from admin group, and check they *aren't* in admin group :D
+    """
+    # This *must* be localhost, not an IP
+    # aiohttp throws away cookies if we are connecting to an IP!
+    hub_url = 'http://localhost'
+    username = secrets.token_hex(8)
+
+    tljh_config_path = [sys.executable, '-m', 'tljh.config']
+
+    assert 0 == await (await asyncio.create_subprocess_exec(*tljh_config_path, 'add-item', 'users.admin', username)).wait()
+    assert 0 == await (await asyncio.create_subprocess_exec(*tljh_config_path, 'reload')).wait()
+
+    # FIXME: wait for reload to finish & hub to come up
+    # Should be part of tljh-config reload
+    await asyncio.sleep(1)
+    async with User(username, hub_url, partial(login_dummy, password='')) as u:
+            await u.login()
+            await u.ensure_server()
+
+            # Assert that the user exists
+            assert pwd.getpwnam(f'jupyter-{username}') is not None
+
+            # Assert that the user has admin rights
+            assert f'jupyter-{username}' in grp.getgrnam('jupyterhub-admins').gr_mem
+
+
+            assert 0 == await (await asyncio.create_subprocess_exec(*tljh_config_path, 'remove-item', 'users.admin', username)).wait()
+            assert 0 == await (await asyncio.create_subprocess_exec(*tljh_config_path, 'reload')).wait()
+            await asyncio.sleep(1)
+
+            await u.stop_server()
+            await u.ensure_server()
+
+            # Assert that the user does *not* have admin rights
+            assert f'jupyter-{username}' in grp.getgrnam('jupyterhub-admins').gr_mem

setup.py

@@ -13,5 +13,10 @@ setup(
     install_requires=[
         'pyyaml==3.*',
         'ruamel.yaml==0.15.*'
-    ]
+    ],
+    entry_points={
+        'console_scripts': [
+            'tljh-config = tljh.config:main'
+        ]
+    }
 )

tests/test_config.py

@@ -0,0 +1,135 @@
+"""
+Test configuration commandline tools
+"""
+from tljh import config
+from contextlib import redirect_stdout
+import io
+import pytest
+import tempfile
+
+
+def test_set_no_mutate():
+    conf = {}
+
+    new_conf = config.set_item_in_config(conf, 'a.b', 'c')
+    assert new_conf['a']['b'] == 'c'
+    assert conf == {}
+
+def test_set_one_level():
+    conf = {}
+
+    new_conf = config.set_item_in_config(conf, 'a', 'b')
+    assert new_conf['a'] == 'b'
+
+def test_set_multi_level():
+    conf = {}
+
+    new_conf = config.set_item_in_config(conf, 'a.b', 'c')
+    new_conf = config.set_item_in_config(new_conf, 'a.d', 'e')
+    new_conf = config.set_item_in_config(new_conf, 'f', 'g')
+    assert new_conf == {
+        'a': {'b': 'c', 'd': 'e'},
+        'f': 'g'
+    }
+
+def test_set_overwrite():
+    """
+    We can overwrite already existing config items.
+
+    This might be surprising destructive behavior to some :D
+    """
+    conf = {
+        'a': 'b'
+    }
+
+    new_conf = config.set_item_in_config(conf, 'a', 'c')
+    assert new_conf == {'a': 'c'}
+
+    new_conf = config.set_item_in_config(new_conf, 'a.b', 'd')
+    assert new_conf == {'a': {'b': 'd'}}
+
+    new_conf = config.set_item_in_config(new_conf, 'a', 'hi')
+    assert new_conf == {'a': 'hi'}
+
+
+def test_add_to_config_one_level():
+    conf = {}
+
+    new_conf = config.add_item_to_config(conf, 'a.b', 'c')
+    assert new_conf == {
+        'a': {'b': ['c']}
+    }
+
+
+def test_add_to_config_zero_level():
+    conf = {}
+
+    new_conf = config.add_item_to_config(conf, 'a', 'b')
+    assert new_conf == {
+        'a': ['b']
+    }
+
+def test_add_to_config_multiple():
+    conf = {}
+
+    new_conf = config.add_item_to_config(conf, 'a.b.c', 'd')
+    assert new_conf == {
+        'a': {'b': {'c': ['d']}}
+    }
+
+    new_conf = config.add_item_to_config(new_conf, 'a.b.c', 'e')
+    assert new_conf == {
+        'a': {'b': {'c': ['d', 'e']}}
+    }
+
+
+def test_remove_from_config():
+    conf = {}
+
+    new_conf = config.add_item_to_config(conf, 'a.b.c', 'd')
+    new_conf = config.add_item_to_config(new_conf, 'a.b.c', 'e')
+    assert new_conf == {
+        'a': {'b': {'c': ['d', 'e']}}
+    }
+
+    new_conf = config.remove_item_from_config(new_conf, 'a.b.c', 'e')
+    assert new_conf == {
+        'a': {'b': {'c': ['d']}}
+    }
+
+def test_remove_from_config_error():
+    with pytest.raises(ValueError):
+        config.remove_item_from_config({}, 'a.b.c', 'e')
+
+    with pytest.raises(ValueError):
+        config.remove_item_from_config({'a': 'b'}, 'a.b', 'e')
+
+    with pytest.raises(ValueError):
+        config.remove_item_from_config({'a': ['b']}, 'a', 'e')
+
+
+def test_show_config():
+    """
+    Test stdout output when showing config
+    """
+    conf = """
+# Just some test YAML
+a:
+  b:
+  - h
+  - 1
+    """.strip()
+
+
+    with tempfile.NamedTemporaryFile() as tmp:
+        tmp.write(conf.encode())
+        tmp.flush()
+
+        out = io.StringIO()
+        with redirect_stdout(out):
+            config.show_config(tmp.name)
+
+    assert out.getvalue().strip() == conf
+
+
+

tljh/config.py

@@ -0,0 +1,242 @@
+"""
+Commandline interface for setting config items in config.yaml.
+
+Used as:
+
+tljh-config set firstlevel.second_level something
+
+tljh-config show
+
+tljh-config show firstlevel
+
+tljh-config show firstlevel.second_level
+"""
+import sys
+import argparse
+from ruamel.yaml import YAML
+from copy import deepcopy
+from tljh import systemd
+
+
+yaml = YAML(typ='rt')
+
+
+def set_item_in_config(config, property_path, value):
+    """
+    Set key at property_path to value in config & return new config.
+
+    config is not mutated.
+
+    property_path is a series of dot separated values. Any part of the path
+    that does not exist is created. 
+    """
+    path_components = property_path.split('.')
+
+    # Mutate a copy of the config, not config itself
+    cur_part = config_copy = deepcopy(config)
+    for i, cur_path in enumerate(path_components):
+        cur_path = path_components[i]
+        if i == len(path_components) - 1:
+            # Final component
+            cur_part[cur_path] = value
+        else:
+            # If we are asked to create new non-leaf nodes, we will always make them dicts
+            # This means setting is *destructive* - will replace whatever is down there!
+            if cur_path not in cur_part or not isinstance(cur_part[cur_path], dict):
+                cur_part[cur_path] = {}
+            cur_part = cur_part[cur_path]
+
+    return config_copy
+
+
+def add_item_to_config(config, property_path, value):
+    """
+    Add an item to a list in config.
+    """
+    path_components = property_path.split('.')
+
+    # Mutate a copy of the config, not config itself
+    cur_part = config_copy = deepcopy(config)
+    for i, cur_path in enumerate(path_components):
+        if i == len(path_components) - 1:
+            # Final component, it must be a list and we append to it
+            if cur_path not in cur_part or not isinstance(cur_part[cur_path], list):
+                cur_part[cur_path] = []
+            cur_part = cur_part[cur_path]
+
+            cur_part.append(value)
+        else:
+            # If we are asked to create new non-leaf nodes, we will always make them dicts
+            # This means setting is *destructive* - will replace whatever is down there!
+            if cur_path not in cur_part or not isinstance(cur_part[cur_path], dict):
+                cur_part[cur_path] = {}
+            cur_part = cur_part[cur_path]
+
+    return config_copy
+
+def remove_item_from_config(config, property_path, value):
+    """
+    Add an item to a list in config.
+    """
+    path_components = property_path.split('.')
+
+    # Mutate a copy of the config, not config itself
+    cur_part = config_copy = deepcopy(config)
+    for i, cur_path in enumerate(path_components):
+        if i == len(path_components) - 1:
+            # Final component, it must be a list and we append to it
+            if cur_path not in cur_part or not isinstance(cur_part[cur_path], list):
+                raise ValueError(f'{property_path} is not a list')
+            cur_part = cur_part[cur_path]
+            cur_part.remove(value)
+        else:
+            if cur_path not in cur_part or not isinstance(cur_part[cur_path], dict):
+                raise ValueError(f'{property_path} does not exist in config!')
+            cur_part = cur_part[cur_path]
+
+    return config_copy
+
+
+def show_config(config_path):
+    """
+    Pretty print config from given config_path
+    """
+    try:
+        with open(config_path) as f:
+            config = yaml.load(f)
+    except FileNotFoundError:
+        config = {}
+    
+    yaml.dump(config, sys.stdout)
+
+
+def set_config_value(config_path, key_path, value):
+    """
+    Set key at key_path in config_path to value
+    """
+    # FIXME: Have a file lock here
+    # FIXME: Validate schema here
+    try:
+        with open(config_path) as f:
+            config = yaml.load(f)
+    except FileNotFoundError:
+        config = {}
+
+    config = set_item_in_config(config, key_path, value)
+
+    with open(config_path, 'w') as f:
+        yaml.dump(config, f)
+
+
+def add_config_value(config_path, key_path, value):
+    """
+    Add value to list at key_path
+    """
+    # FIXME: Have a file lock here
+    # FIXME: Validate schema here
+    try:
+        with open(config_path) as f:
+            config = yaml.load(f)
+    except FileNotFoundError:
+        config = {}
+
+    config = add_item_to_config(config, key_path, value)
+
+    with open(config_path, 'w') as f:
+        yaml.dump(config, f)
+
+def reload_component(component):
+    """
+    Reload a TLJH component.
+
+    component can be 'hub' or 'proxy'.
+    """
+    if component == 'hub':
+        systemd.restart_service('jupyterhub')
+        # FIXME: Verify hub is back up?
+        print('Hub reload with new configuration complete')
+    elif component == 'proxy':
+        systemd.restart_service('configurable-http-proxy')
+        print('Proxy reload with new configuration complete')
+
+
+def main():
+    argparser = argparse.ArgumentParser()
+    argparser.add_argument(
+        '--config-path',
+        default='/opt/tljh/config.yaml',
+        help='Path to TLJH config.yaml file'
+    )
+    subparsers = argparser.add_subparsers(dest='action')
+
+    show_parser = subparsers.add_parser(
+        'show',
+        help='Show current configuration'
+    )
+
+    set_parser = subparsers.add_parser(
+        'set',
+        help='Set a configuration property'
+    )
+    set_parser.add_argument(
+        'key_path',
+        help='Dot separated path to configuration key to set'
+    )
+    set_parser.add_argument(
+        'value',
+        help='Value ot set the configuration key to'
+    )
+
+    add_item_parser = subparsers.add_parser(
+        'add-item',
+        help='Add a value to a list for a configuration property'
+    )
+    add_item_parser.add_argument(
+        'key_path',
+        help='Dot separated path to configuration key to add value to'
+    )
+    add_item_parser.add_argument(
+        'value',
+        help='Value to add to the configuration key'
+    )
+
+    remove_item_parser = subparsers.add_parser(
+        'remove-item',
+        help='Remove a value from a list for a configuration property'
+    )
+    remove_item_parser.add_argument(
+        'key_path',
+        help='Dot separated path to configuration key to remove value from'
+    )
+    remove_item_parser.add_argument(
+        'value',
+        help='Value to remove from key_path'
+    )
+
+    reload_parser = subparsers.add_parser(
+        'reload',
+        help='Reload a component to apply configuration change'
+    )
+    reload_parser.add_argument(
+        'component',
+        choices=('hub', 'proxy'),
+        help='Which component to reload',
+        default='hub',
+        nargs='?'
+    )
+
+    args = argparser.parse_args()
+
+    if args.action == 'show':
+        show_config(args.config_path)
+    elif args.action == 'set':
+        set_config_value(args.config_path, args.key_path, args.value)
+    elif args.action == 'add-item':
+        add_config_value(args.config_path, args.key_path, args.value)
+    elif args.action == 'remove-item':
+        add_config_value(args.config_path, args.key_path, args.value)
+    elif args.action == 'reload':
+        reload_component(args.component)
+    
+if __name__ == '__main__':
+    main()