specs: move to new spec.json format with build provenance (#22845)

This is a major rework of Spack's core core `spec.yaml` metadata format.  It moves from `spec.yaml` to `spec.json` for speed, and it changes the format in several ways. Specifically:

1. The spec format now has a `_meta` section with a version (now set to version `2`).  This will simplify major changes like this one in the future.
2. The node list in spec dictionaries is no longer keyed by name. Instead, it is a list of records with no required key. The name, hash, etc. are fields in the dictionary records like any other.
3. Dependencies can be keyed by any hash (`hash`, `full_hash`, `build_hash`).
4. `build_spec` provenance from #20262 is included in the spec format. This means that, for spliced specs, we preserve the *full* provenance of how to build, and we can reproduce a spliced spec from the original builds that produced it.

**NOTE**: Because we have switched the spec format, this PR changes Spack's hashing algorithm.  This means that after this commit, Spack will think a lot of things need rebuilds.

There are two major benefits this PR provides:
* The switch to JSON format speeds up Spack significantly, as Python's builtin JSON implementation is orders of magnitude faster than YAML. 
* The new Spec format will soon allow us to represent DAGs with potentially multiple versions of the same dependency -- e.g., for build dependencies or for compilers-as-dependencies.  This PR lays the necessary groundwork for those features.

The old `spec.yaml` format continues to be supported, but is now considered a legacy format, and Spack will opportunistically convert these to the new `spec.json` format.

var/spack/repos/builtin.mock/packages/zlib/package.py

@@ -0,0 +1,55 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+
+# Although zlib comes with a configure script, it does not use Autotools
+# The AutotoolsPackage causes zlib to fail to build with PGI
+class Zlib(Package):
+    """A free, general-purpose, legally unencumbered lossless
+    data-compression library.
+    """
+
+    homepage = "http://zlib.net"
+    # URL must remain http:// so Spack can bootstrap curl
+    url = "http://zlib.net/fossils/zlib-1.2.11.tar.gz"
+
+    version('1.2.11', sha256='c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1')
+    # Due to the bug fixes, any installations of 1.2.9 or 1.2.10 should be
+    # immediately replaced with 1.2.11.
+    version('1.2.8', sha256='36658cb768a54c1d4dec43c3116c27ed893e88b02ecfcb44f2166f9c0b7f2a0d')
+    version('1.2.3', sha256='1795c7d067a43174113fdf03447532f373e1c6c57c08d61d9e4e9be5e244b05e')
+
+    variant('pic', default=True,
+            description='Produce position-independent code (for shared libs)')
+    variant('shared', default=True,
+            description='Enables the build of shared libraries.')
+    variant('optimize', default=True,
+            description='Enable -O2 for a more optimized lib')
+
+    patch('w_patch.patch', when="@1.2.11%cce")
+
+    @property
+    def libs(self):
+        shared = '+shared' in self.spec
+        return find_libraries(
+            ['libz'], root=self.prefix, recursive=True, shared=shared
+        )
+
+    def setup_build_environment(self, env):
+        if '+pic' in self.spec:
+            env.append_flags('CFLAGS', self.compiler.cc_pic_flag)
+        if '+optimize' in self.spec:
+            env.append_flags('CFLAGS', '-O2')
+
+    def install(self, spec, prefix):
+        config_args = []
+        if '~shared' in spec:
+            config_args.append('--static')
+        configure('--prefix={0}'.format(prefix), *config_args)
+
+        make()
+        if self.run_tests:
+            make('check')
+        make('install')

var/spack/repos/builtin.mock/packages/zlib/w_patch.patch

@@ -0,0 +1,13 @@
+diff --git a/configure b/configure
+index e974d1f..ed26a63 100755
+--- a/configure
++++ b/configure
+@@ -409,7 +409,7 @@ EOF
+ if test $shared -eq 1; then
+   echo Checking for shared library support... | tee -a configure.log
+   # we must test in two steps (cc then ld), required at least on SunOS 4.x
+-  if try $CC -w -c $SFLAGS $test.c &&
++  if try $CC -c $SFLAGS $test.c &&
+      try $LDSHARED $SFLAGS -o $test$shared_ext $test.o; then
+     echo Building shared library $SHAREDLIBV with $CC. | tee -a configure.log
+   elif test -z "$old_cc" -a -z "$old_cflags"; then