Add better tests for web.py; fix some bugs found with spidering.

- _spider in web.py was actually failing to spider deeper than a certain
  point.
- Fixed multiprocessing pools to not use daemons and to allow recursive
  spawning.
- Added detailed tests for spidering and for finding archive versions.
- left some xfail URL finding exercises for the reader.
- Fix noqa annotations for some @when decorators

var/spack/repos/builtin/packages/autogen/package.py

@@ -34,7 +34,7 @@ class Autogen(AutotoolsPackage):
     homepage = "https://www.gnu.org/software/autogen/index.html"
     url      = "https://ftp.gnu.org/gnu/autogen/rel5.18.12/autogen-5.18.12.tar.gz"
     list_url = "https://ftp.gnu.org/gnu/autogen"
-    list_depth = 2
+    list_depth = 1
 
     version('5.18.12', '551d15ccbf5b5fc5658da375d5003389')