Adding support for spack monitor with containerize (#23777)

This should get us most of the way there to support using monitor during a spack container build, for both Singularity and Docker. Some quick notes:

### Docker
Docker works by way of BUILDKIT and being able to specify --secret. What this means is that you can prefix a line with a mount of type secret as follows:

```bash
# Install the software, remove unnecessary deps
RUN --mount=type=secret,id=su --mount=type=secret,id=st cd /opt/spack-environment && spack env activate . && export SPACKMON_USER=$(cat /run/secrets/su) && export SPACKMON_TOKEN=$(cat /run/secrets/st) && spack install --monitor --fail-fast && spack gc -y
```
Where the id for one or more secrets corresponds to the file mounted at `/run/secrets/<name>`. So, for example, to build this container with su (spackmon user) and sv (spackmon token) defined I would export them on my host and do:

```bash
$ DOCKER_BUILDKIT=1 docker build --network="host" --secret id=st,env=SPACKMON_TOKEN --secret id=su,env=SPACKMON_USER -t spack/container . 
```
And when we add `env` to the secret definition that tells the build to look for the secret with id "st" in the environment variable `SPACKMON_TOKEN` for example.

If the user is building locally with a local spack monitor, we also need to set the `--network` to be the host, otherwise you can't connect to it (a la isolation of course.)

## Singularity

Singularity doesn't have as nice an ability to clearly specify secrets, so (hoping this eventually gets implemented) what I'm doing now is providing the user instructions to write the credentials to a file, add it to the container to source, and remove when done.

## Tags

Note that the tags PR https://github.com/spack/spack/pull/23712 will need to be merged before `--monitor-tags` will actually work because I'm checking for the attribute (that doesn't exist yet):

```bash
"tags": getattr(args, "monitor_tags", None)
```
So when that PR is merged to update the argument group, it will work here, and I can either update the PR here to not check if the attribute is there (it will be) or open another one in the case this PR is already merged. 

Finally, I added a bunch of documetation for how to use monitor with containerize. I say "mostly working" because I can't do a full test run with this new version until the container base is built with the updated spack (the request to the monitor server for an env install was missing so I had to add it here).

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Co-authored-by: vsoch <vsoch@users.noreply.github.com>

lib/spack/docs/monitoring.rst

@@ -103,6 +103,140 @@ more tags to your build, you can do:
     $ spack install --monitor --monitor-tags pizza,pasta hdf5
 
 
+----------------------------
+Monitoring with Containerize
+----------------------------
+
+The same argument group is available to add to a containerize command. 
+
+^^^^^^
+Docker
+^^^^^^
+
+To add monitoring to a Docker container recipe generation using the defaults,
+and assuming a monitor server running on localhost, you would
+start with a spack.yaml in your present working directory:
+
+.. code-block:: yaml
+
+   spack:
+     specs:
+       - samtools
+
+And then do:
+
+.. code-block:: console
+
+    # preview first
+    spack containerize --monitor
+    
+    # and then write to a Dockerfile
+    spack containerize --monitor > Dockerfile
+    
+    
+The install command will be edited to include commands for enabling monitoring.
+However, getting secrets into the container for your monitor server is something
+that should be done carefully. Specifically you should:
+
+ - Never try to define secrets as ENV, ARG, or using ``--build-arg``
+ - Do not try to get the secret into the container via a "temporary" file that you remove (it in fact will still exist in a layer)
+
+Instead, it's recommended to use buildkit `as explained here <https://pythonspeed.com/articles/docker-build-secrets/>`_.
+You'll need to again export environment variables for your spack monitor server:
+
+.. code-block:: console
+
+    $ export SPACKMON_TOKEN=50445263afd8f67e59bd79bff597836ee6c05438
+    $ export SPACKMON_USER=spacky
+
+And then use buildkit along with your build and identifying the name of the secret:
+
+.. code-block:: console
+
+    $ DOCKER_BUILDKIT=1 docker build --secret id=st,env=SPACKMON_TOKEN --secret id=su,env=SPACKMON_USER -t spack/container . 
+    
+The secrets are expected to come from your environment, and then will be temporarily mounted and available
+at ``/run/secrets/<name>``. If you forget to supply them (and authentication is required) the build
+will fail. If you need to build on your host (and interact with a spack monitor at localhost) you'll
+need to tell Docker to use the host network:
+
+.. code-block:: console
+
+    $ DOCKER_BUILDKIT=1 docker build --network="host" --secret id=st,env=SPACKMON_TOKEN --secret id=su,env=SPACKMON_USER -t spack/container . 
+    
+
+^^^^^^^^^^^
+Singularity
+^^^^^^^^^^^
+
+To add monitoring to a Singularity container build, the spack.yaml needs to
+be modified slightly to specify wanting a different format:
+
+
+.. code-block:: yaml
+
+   spack:
+     specs:
+       - samtools
+     container:
+       format: singularity
+       
+       
+Again, generate the recipe:
+
+
+.. code-block:: console
+
+    # preview first
+    $ spack containerize --monitor
+    
+    # then write to a Singularity recipe
+    $ spack containerize --monitor > Singularity
+
+
+Singularity doesn't have a direct way to define secrets at build time, so we have
+to do a bit of a manual command to add a file, source secrets in it, and remove it.
+Since Singularity doesn't have layers like Docker, deleting a file will truly
+remove it from the container and history. So let's say we have this file,
+``secrets.sh``:
+
+.. code-block:: console
+
+    # secrets.sh
+    export SPACKMON_USER=spack
+    export SPACKMON_TOKEN=50445263afd8f67e59bd79bff597836ee6c05438
+
+
+We would then generate the Singularity recipe, and add a files section,
+a source of that file at the start of ``%post``, and **importantly**
+a removal of the final at the end of that same section.
+
+.. code-block::
+
+    Bootstrap: docker
+    From: spack/ubuntu-bionic:latest
+    Stage: build
+
+    %files
+      secrets.sh /opt/secrets.sh 
+
+    %post
+      . /opt/secrets.sh
+
+      # spack install commands are here
+      ...
+      
+      # Don't forget to remove here!
+      rm /opt/secrets.sh
+
+
+You can then build the container as your normally would.
+
+.. code-block:: console
+
+    $ sudo singularity build container.sif Singularity 
+
+
 ------------------
 Monitoring Offline
 ------------------