Add better tests for web.py; fix some bugs found with spidering.

- _spider in web.py was actually failing to spider deeper than a certain
  point.
- Fixed multiprocessing pools to not use daemons and to allow recursive
  spawning.
- Added detailed tests for spidering and for finding archive versions.
- left some xfail URL finding exercises for the reader.
- Fix noqa annotations for some @when decorators

var/spack/repos/builtin/packages/hydra/package.py

@@ -34,6 +34,6 @@ class Hydra(AutotoolsPackage):
     homepage = "http://www.mpich.org"
     url      = "http://www.mpich.org/static/downloads/3.2/hydra-3.2.tar.gz"
     list_url = "http://www.mpich.org/static/downloads/"
-    list_depth = 2
+    list_depth = 1
 
     version('3.2', '4d670916695bf7e3a869cc336a881b39')