diff --git a/lib/spack/docs/build_systems.rst b/lib/spack/docs/build_systems.rst index 8061b158584..b123704c229 100644 --- a/lib/spack/docs/build_systems.rst +++ b/lib/spack/docs/build_systems.rst @@ -63,7 +63,6 @@ on these ideas for each distinct build system that Spack supports: build_systems/cudapackage build_systems/custompackage build_systems/inteloneapipackage - build_systems/intelpackage build_systems/rocmpackage build_systems/sourceforgepackage diff --git a/lib/spack/docs/build_systems/inteloneapipackage.rst b/lib/spack/docs/build_systems/inteloneapipackage.rst index 5f6c29e7450..22f3f2929c5 100644 --- a/lib/spack/docs/build_systems/inteloneapipackage.rst +++ b/lib/spack/docs/build_systems/inteloneapipackage.rst @@ -33,9 +33,6 @@ For more information on a specific package, do:: spack info --all -Intel no longer releases new versions of Parallel Studio, which can be -used in Spack via the :ref:`intelpackage`. All of its components can -now be found in oneAPI. Examples ======== @@ -50,34 +47,8 @@ Install the oneAPI compilers:: spack install intel-oneapi-compilers -Add the compilers to your ``compilers.yaml`` so spack can use them:: - spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin - -Verify that the compilers are available:: - - spack compiler list - -Note that 2024 and later releases do not include ``icc``. Before 2024, -the package layout was different:: - - spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin/intel64 - spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin - -The ``intel-oneapi-compilers`` package includes 2 families of -compilers: - -* ``intel``: ``icc``, ``icpc``, ``ifort``. Intel's *classic* - compilers. 2024 and later releases contain ``ifort``, but not - ``icc`` and ``icpc``. -* ``oneapi``: ``icx``, ``icpx``, ``ifx``. Intel's new generation of - compilers based on LLVM. - -To build the ``patchelf`` Spack package with ``icc``, do:: - - spack install patchelf%intel - -To build with with ``icx``, do :: +To build the ``patchelf`` Spack package with ``icx``, do:: spack install patchelf%oneapi @@ -92,15 +63,6 @@ Install the oneAPI compilers:: spack install intel-oneapi-compilers -Add the compilers to your ``compilers.yaml`` so Spack can use them:: - - spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin - spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin - -Verify that the compilers are available:: - - spack compiler list - Clone `spack-configs `_ repo and activate Intel oneAPI CPU environment:: git clone https://github.com/spack/spack-configs @@ -149,7 +111,7 @@ Compilers --------- To use the compilers, add some information about the installation to -``compilers.yaml``. For most users, it is sufficient to do:: +``packages.yaml``. For most users, it is sufficient to do:: spack compiler add /opt/intel/oneapi/compiler/latest/bin @@ -157,7 +119,7 @@ Adapt the paths above if you did not install the tools in the default location. After adding the compilers, using them is the same as if you had installed the ``intel-oneapi-compilers`` package. Another option is to manually add the configuration to -``compilers.yaml`` as described in :ref:`Compiler configuration +``packages.yaml`` as described in :ref:`Compiler configuration `. Before 2024, the directory structure was different:: @@ -200,15 +162,5 @@ You can also use Spack-installed libraries. For example:: Will update your environment CPATH, LIBRARY_PATH, and other environment variables for building an application with oneMKL. -More information -================ - -This section describes basic use of oneAPI, especially if it has -changed compared to Parallel Studio. See :ref:`intelpackage` for more -information on :ref:`intel-virtual-packages`, -:ref:`intel-unrelated-packages`, -:ref:`intel-integrating-external-libraries`, and -:ref:`using-mkl-tips`. - .. _`Intel installers`: https://software.intel.com/content/www/us/en/develop/documentation/installation-guide-for-intel-oneapi-toolkits-linux/top.html diff --git a/lib/spack/docs/build_systems/intelpackage.rst b/lib/spack/docs/build_systems/intelpackage.rst deleted file mode 100644 index 733c8b0f1d2..00000000000 --- a/lib/spack/docs/build_systems/intelpackage.rst +++ /dev/null @@ -1,1077 +0,0 @@ -.. Copyright Spack Project Developers. See COPYRIGHT file for details. - - SPDX-License-Identifier: (Apache-2.0 OR MIT) - -.. _intelpackage: - ------ -Intel ------ - -.. contents:: - -^^^^^^^^^^^^^^^^^^^^^^^^ -Intel packages in Spack -^^^^^^^^^^^^^^^^^^^^^^^^ - -This is an earlier version of Intel software development tools and has -now been replaced by Intel oneAPI Toolkits. - -Spack can install and use several software development products offered by Intel. -Some of these are available under no-cost terms, others require a paid license. -All share the same basic steps for configuration, installation, and, where -applicable, license management. The Spack Python class ``IntelPackage`` implements -these steps. - -Spack interacts with Intel tools in several routes, like it does for any -other package: - -.. _`route 1`: - -1. Accept system-provided tools after you declare them to Spack as *external packages*. - -.. _`route 2`: - -2. Install the products for you as *internal packages* in Spack. - -.. _`route 3`: - -3. *Use* the packages, regardless of installation route, to install what we'll - call *client packages* for you, this being Spack's primary purpose. - -An auxiliary route follows from route 2, as it would for most Spack -packages, namely: - -.. _`route 4`: - -4. Make Spack-installed Intel tools available outside of Spack for ad-hoc use, - typically through Spack-managed modulefiles. - -This document covers routes 1 through 3. - - -"""""""""""""""""""""""""""""""""" -Packages under no-cost license -"""""""""""""""""""""""""""""""""" - -Intel's standalone performance library products, notably MKL and MPI, are -available for use under a `simplified license -`_ -since 2017 [fn1]_. They are packaged in Spack as: - -* ``intel-mkl`` -- Math Kernel Library (linear algebra and FFT), -* ``intel-mpi`` -- The Intel-MPI implementation (derived from MPICH), -* ``intel-ipp`` -- Primitives for image-, signal-, and data-processing, -* ``intel-daal`` -- Machine learning and data analytics. - -Some earlier versions of these libraries were released under a paid license. -For these older versions, the license must be available at installation time of -the products and during compilation of client packages. - -The library packages work well with the Intel compilers but do not require them --- those packages can just as well be used with other compilers. The Intel -compiler invocation commands offer custom options to simplify linking Intel -libraries (sometimes considerably), but Spack always uses fairly explicit -linkage anyway. - - -"""""""""""""""""" -Licensed packages -"""""""""""""""""" - -Intel's core software development products that provide compilers, analyzers, -and optimizers do require a paid license. In Spack, they are packaged as: - -* ``intel-parallel-studio`` -- the entire suite of compilers and libraries, -* ``intel`` -- a subset containing just the compilers and the Intel-MPI runtime [fn2]_. - -.. - TODO: Confirm and possible change(!) the scope of MPI components (runtime - vs. devel) in current (and previous?) *cluster/professional/composer* - editions, i.e., presence in downloads, possibly subject to license - coverage(!); see `discussion in PR #4300 - `_. [NB: - An "mpi" subdirectory is not indicative of the full MPI SDK being present - (i.e., ``mpicc``, ..., and header files). The directory may just as well - contain only the MPI runtime (``mpirun`` and shared libraries) .] - See also issue #8632. - -The license is needed at installation time and to compile client packages, but -never to merely run any resulting binaries. The license status for a given -Spack package is normally specified in the *package code* through directives like -`license_required` (see :ref:`Licensed software `). -For the Intel packages, however, the *class code* provides these directives (in -exchange of forfeiting a measure of OOP purity) and takes care of idiosyncasies -like historic version dependence. - -The libraries that are provided in the standalone packages are also included in the -all-encompassing ``intel-parallel-studio``. To complicate matters a bit, that -package is sold in 3 "editions", of which only the upper-tier ``cluster`` -edition supports *compiling* MPI applications, and hence only that edition can -provide the ``mpi`` virtual package. (As mentioned [fn2]_, all editions -provide support for *running* MPI applications.) - -The edition forms the leading part of the version number for Spack's -``intel*`` packages discussed here. This differs from the primarily numeric -version numbers seen with most other Spack packages. For example, we have: - - -.. code-block:: console - - $ spack info intel-parallel-studio - ... - Preferred version: - professional.2018.3 http:... - - Safe versions: - professional.2018.3 http:... - ... - composer.2018.3 http:... - ... - cluster.2018.3 http:... - ... - ... - -The full studio suite, capable of compiling MPI applications, currently -requires about 12 GB of disk space when installed (see section `Install steps -for packages with compilers and libraries`_ for detailed instructions). -If you need to save disk space or installation time, you could install the -``intel`` compilers-only subset (0.6 GB) and just the library packages you -need, for example ``intel-mpi`` (0.5 GB) and ``intel-mkl`` (2.5 GB). - -.. _intel-unrelated-packages: - -"""""""""""""""""""" -Unrelated packages -"""""""""""""""""""" - -The following packages do not use the Intel installer and are not in class ``IntelPackage`` -that is discussed here: - -* ``intel-gpu-tools`` -- Test suite and low-level tools for the Linux `Direct - Rendering Manager `_ -* ``intel-mkl-dnn`` -- Math Kernel Library for Deep Neural Networks (``CMakePackage``) -* ``intel-xed`` -- X86 machine instructions encoder/decoder -* ``intel-tbb`` -- Standalone version of Intel Threading Building Blocks. Note that - a TBB runtime version is included with ``intel-mkl``, and development - versions are provided by the packages ``intel-parallel-studio`` (all - editions) and its ``intel`` subset. - -"""""""""""""""""""""""""""""""""""""""""" -Configuring Spack to use Intel licenses -"""""""""""""""""""""""""""""""""""""""""" - -If you wish to integrate licensed Intel products into Spack as external packages -(`route 1`_ above) we assume that their license configuration is in place and -is working [fn3]_. In this case, skip to section `Integration of Intel tools -installed external to Spack`_. - -If you plan to have Spack install licensed products for you (`route 2`_ above), -the Intel product installer that Spack will run underneath must have access to -a license that is either provided by a *license server* or as a *license file*. -The installer may be able to locate a license that is already configured on -your system. If it cannot, you must configure Spack to provide either the -server location or the license file. - -For authoritative information on Intel licensing, see: - -* https://software.intel.com/en-us/faq/licensing -* https://software.intel.com/en-us/articles/how-do-i-manage-my-licenses - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Pointing to an existing license server -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Installing and configuring a license server is outside the scope of Spack. We -assume that: - -* Your system administrator has a license server running. -* The license server offers valid licenses for the Intel packages of interest. -* You can access these licenses under the user id running Spack. - -Be aware of the difference between (a) installing and configuring a license -server, and (b) configuring client software to *use* a server's -so-called floating licenses. We are concerned here with (b) only. The -process of obtaining a license from a server for temporary use is called -"checking out a license". For that, a client application such as the Intel -package installer or a compiler needs to know the host name and port number of -one or more license servers that it may query [fn4]_. - -Follow one of three methods to `point client software to a floating license server -`_. -Ideally, your license administrator will already have implemented one that can -be used unchanged in Spack: Look for the environment variable -``INTEL_LICENSE_FILE`` or for files -``/opt/intel/licenses/*.lic`` that contain:: - - SERVER hostname hostid_or_ANY portnum - USE_SERVER - -The relevant tokens, among possibly others, are the ``USE_SERVER`` line, -intended specifically for clients, and one or more ``SERVER`` lines above it -which give the network address. - -If you cannot find pre-existing ``/opt/intel/licenses/*.lic`` files and the -``INTEL_LICENSE_FILE`` environment variable is not set (even after you loaded -any relevant modulefiles), ask your license administrator for the server -address(es) and place them in a "global" license file within your Spack -directory tree `as shown below `_). - - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Installing a standalone license file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you purchased a user-specific license, follow `Intel's instructions -`_ -to "activate" it for your serial number, then download the resulting license file. -If needed, `request to have the file re-sent -`_ to you. - -Intel's license files are text files that contain tokens in the proprietary -"FLEXlm" format and whose name ends in ``.lic``. -Intel installers and compilers look for license files in several locations when they run. -Place your license by one of the following means, in order of decreasing preference: - -* Default directory - - Install your license file in the directory ``/opt/intel/licenses/`` if you - have write permission to it. This directory is inspected by all Intel tools - and is therefore preferred, as no further configuration will be needed. - Create the directory if it does not yet exist. For the file name, either - keep the downloaded name or use another suitably plain yet descriptive - name that ends in ``.lic``. Adjust file permissions for access by licensed - users. - - -* Directory given in environment variable - - If you cannot use the default directory, but your system already has set the - environment variable ``INTEL_LICENSE_FILE`` independent from Spack [fn5]_, - then, if you have the necessary write permissions, place your license file in - one of the directories mentioned in this environment variable. Adjust file - permissions to match licensed users. - - .. tip:: - - If your system has not yet set and used the environment variable - ``INTEL_LICENSE_FILE``, you could start using it with the ``spack - install`` stage of licensed tools and subsequent client packages. You - would, however, be in a bind to always set that variable in the same - manner, across updates and re-installations, and perhaps accommodate - additions to it. As this may be difficult in the long run, we recommend - that you do *not* attempt to start using the variable solely for Spack. - -.. _`Spack-managed file`: - -* Spack-managed file - - The first time Spack encounters an Intel package that requires a license, it - will initialize a Spack-global Intel-specific license file for you, as a - template with instructional comments, and bring up an editor [fn6]_. Spack - will do this *even if you have a working license elsewhere* on the system. - - * To proceed with an externally configured license, leave the newly templated - file as is (containing comments only) and close the editor. You do not need - to touch the file again. - - * To configure your own standalone license, copy the contents of your - downloaded license file into the opened file, save it, and close the editor. - - * To use a license server (i.e., a floating network license) that is not - already configured elsewhere on the system, supply your license server - address(es) in the form of ``SERVER`` and ``USE_SERVER`` lines at the - *beginning of the file* [fn7]_, in the format shown in section `Pointing to - an existing license server`_. Save the file and close the editor. - - To revisit and manually edit this file, such as prior to a subsequent - installation attempt, find it at - ``$SPACK_ROOT/etc/spack/licenses/intel/intel.lic`` . - - Spack will place symbolic links to this file in each directory where licensed - Intel binaries were installed. If you kept the template unchanged, Intel tools - will simply ignore it. - - -.. _integrate-external-intel: - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Integration of Intel tools installed *external* to Spack -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This section discusses `route 1`_ from the introduction. - -A site that already uses Intel tools, especially licensed ones, will likely -have some versions already installed on the system, especially at a time when -Spack is just being introduced. It will be useful to make such previously -installed tools available for use by Spack as they are. How to do this varies -depending on the type of the tools: - -"""""""""""""""""""""""""""""""""" -Integrating external compilers -"""""""""""""""""""""""""""""""""" - -For Spack to use external Intel compilers, you must tell it both *where* to -find them and *when* to use them. The present section documents the "where" -aspect, involving ``compilers.yaml`` and, in most cases, long absolute paths. -The "when" aspect actually relates to `route 3`_ and requires explicitly -stating the compiler as a spec component (in the form ``foo %intel`` or ``foo -%intel@compilerversion``) when installing client packages or altering Spack's -compiler default in ``packages.yaml``. -See section `Selecting Intel compilers `_ for details. - -To integrate a new set of externally installed Intel compilers into Spack -follow section -:ref:`Compiler configuration `. -Briefly, prepare your shell environment like you would if you were to use these -compilers normally, i.e., typically by a ``module load ...`` or a shell -``source ...`` command, then use ``spack compiler find`` to make Spack aware of -these compilers. This will create a new entry in a suitably scoped and possibly new -``compilers.yaml`` file. You could certainly create such a compiler entry -manually, but this is error-prone due to the indentation and different data -types involved. - -The Intel compilers need and use the system's native GCC compiler (``gcc`` on -most systems, ``clang`` on macOS) to provide certain functionality, notably to -support C++. To provide a different GCC compiler for the Intel tools, or more -generally set persistent flags for all invocations of the Intel compilers, locate -the ``compilers.yaml`` entry that defines your Intel compiler, and, using a -text editor, change one or both of the following: - -1. At the ``modules:`` tag, add a ``gcc`` module to the list. -2. At the ``flags:`` tag, add ``cflags:``, ``cxxflags:``, and ``fflags:`` key-value entries. - -Consult the examples under -:ref:`Compiler configuration ` -and -:ref:`Vendor-Specific Compiler Configuration ` -in the Spack documentation. -When done, validate your compiler definition by running -``spack compiler info intel@compilerversion`` (replacing ``compilerversion`` by -the version that you defined). - -Be aware that both the GCC integration and persistent compiler flags can also be -affected by an advanced third method: - -3. A modulefile that provides the Intel compilers for you - could, for the benefit of users outside of Spack, implicitly - integrate a specific ``gcc`` version via compiler flag environment variables - or (hopefully not) via a sneaky extra ``PATH`` addition. - -Next, visit section `Selecting Intel Compilers`_ to learn how to tell -Spack to use the newly configured compilers. - -.. _intel-integrating-external-libraries: - -"""""""""""""""""""""""""""""""""" -Integrating external libraries -"""""""""""""""""""""""""""""""""" - -Configure external library-type packages (as opposed to compilers) -in the files ``$SPACK_ROOT/etc/spack/packages.yaml`` or -``~/.spack/packages.yaml``, following the Spack documentation under -:ref:`External Packages `. - -Similar to ``compilers.yaml``, the ``packages.yaml`` files define a package -external to Spack in terms of a Spack spec and resolve each such spec via -either the ``paths`` or ``modules`` tokens to a specific pre-installed package -version on the system. Since Intel tools generally need environment variables -to interoperate, which cannot be conveyed in a mere ``paths`` specification, -the ``modules`` token will be more sensible to use. It resolves the Spack-side -spec to a modulefile generated and managed outside of Spack's purview, -which Spack will load internally and transiently when the corresponding spec is -called upon to compile client packages. - -Unlike for compilers, where ``spack find compilers [spec]`` generates an entry -in an existing or new ``compilers.yaml`` file, Spack does not offer a command -to generate an entirely new ``packages.yaml`` entry. You must create -new entries yourself in a text editor, though the command ``spack config -[--scope=...] edit packages`` can help with selecting the proper file. -See section -:ref:`Configuration Scopes ` -for an explanation about the different files -and section -:ref:`Build customization ` -for specifics and examples for ``packages.yaml`` files. - -.. If your system administrator did not provide modules for pre-installed Intel - tools, you could do well to ask for them, because installing multiple copies - of the Intel tools, as is won't to happen once Spack is in the picture, is - bound to stretch disk space and patience thin. If you *are* the system - administrator and are still new to modules, then perhaps it's best to follow - the `next section `_ and install the tools - solely within Spack. - -The following example integrates packages embodied by hypothetical -external modulefiles ``intel-mkl/18/...`` into -Spack as packages ``intel-mkl@...``: - -.. code-block:: console - - $ spack config edit packages - -Make sure the file begins with: - -.. code-block:: yaml - - packages: - -Adapt the following example. Be sure to maintain the indentation: - -.. code-block:: yaml - - # other content ... - - intel-mkl: - externals: - - spec: "intel-mkl@2018.2.199 arch=linux-centos6-x86_64" - modules: - - intel-mkl/18/18.0.2 - - spec: "intel-mkl@2018.3.222 arch=linux-centos6-x86_64" - modules: - - intel-mkl/18/18.0.3 - -The version numbers for the ``intel-mkl`` specs defined here correspond to file -and directory names that Intel uses for its products because they were adopted -and declared as such within Spack's package repository. You can inspect the -versions known to your current Spack installation by: - -.. code-block:: console - - $ spack info intel-mkl - -Using the same version numbers for external packages as for packages known -internally is useful for clarity, but not strictly necessary. Moreover, with a -``packages.yaml`` entry, you can go beyond internally known versions. - -.. _compiler-neutral-package: - -Note that the Spack spec in the example does not contain a compiler -specification. This is intentional, as the Intel library packages can be used -unmodified with different compilers. - -A slightly more advanced example illustrates how to provide -:ref:`variants ` -and how to use the ``buildable: False`` directive to prevent Spack from installing -other versions or variants of the named package through its normal internal -mechanism. - -.. code-block:: yaml - - packages: - intel-parallel-studio: - externals: - - spec: "intel-parallel-studio@cluster.2018.2.199 +mkl+mpi+ipp+tbb+daal arch=linux-centos6-x86_64" - modules: - - intel/18/18.0.2 - - spec: "intel-parallel-studio@cluster.2018.3.222 +mkl+mpi+ipp+tbb+daal arch=linux-centos6-x86_64" - modules: - - intel/18/18.0.3 - buildable: False - -One additional example illustrates the use of ``prefix:`` instead of -``modules:``, useful when external modulefiles are not available or not -suitable: - -.. code-block:: yaml - - packages: - intel-parallel-studio: - externals: - - spec: "intel-parallel-studio@cluster.2018.2.199 +mkl+mpi+ipp+tbb+daal" - prefix: /opt/intel - - spec: "intel-parallel-studio@cluster.2018.3.222 +mkl+mpi+ipp+tbb+daal" - prefix: /opt/intel - buildable: False - -Note that for the Intel packages discussed here, the directory values in the -``prefix:`` entries must be the high-level and typically version-less -"installation directory" that has been used by Intel's product installer. -Such a directory will typically accumulate various product versions. Amongst -them, Spack will select the correct version-specific product directory based on -the ``@version`` spec component that each path is being defined for. - -For further background and details, see -:ref:`External Packages `. - - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Installing Intel tools *within* Spack -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This section discusses `route 2`_ from the introduction. - -When a system does not yet have Intel tools installed already, or the installed -versions are undesirable, Spack can install these tools like any regular Spack -package for you and, with appropriate pre- and post-install configuration, use its -compilers and/or libraries to install client packages. - -.. _intel-install-studio: - -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Install steps for packages with compilers and libraries -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -The packages ``intel-parallel-studio`` and ``intel`` (which is a subset of the -former) are many-in-one products that contain both compilers and a set of -library packages whose scope depends on the edition. -Because they are general products geared towards shell environments, -it can be somewhat involved to integrate these packages at their full extent -into Spack. - -Note: To install library-only packages like ``intel-mkl``, ``intel-mpi``, and ``intel-daal`` -follow `the next section `_ instead. - -1. Review the section `Configuring spack to use intel licenses`_. - -.. _intel-compiler-anticipation: - -2. To install a version of ``intel-parallel-studio`` that provides Intel - compilers at a version that you have *not yet declared in Spack*, - the following preparatory steps are recommended: - - A. Determine the compiler spec that the new ``intel-parallel-studio`` package - will provide, as follows: From the package version, combine the last two - digits of the version year, a literal "0" (zero), and the version component - that immediately follows the year. - - ========================================== ====================== - Package version Compiler spec provided - ------------------------------------------ ---------------------- - ``intel-parallel-studio@edition.YYyy.u`` ``intel@yy.0.u`` - ========================================== ====================== - - Example: The package ``intel-parallel-studio@cluster.2018.3`` will provide - the compiler with spec ``intel@18.0.3``. - - .. _`config-compiler-anticipated`: - - B. Add a new compiler section with the newly anticipated version at the - end of a ``compilers.yaml`` file in a suitable scope. For example, run: - - .. code-block:: console - - $ spack config --scope=user/linux edit compilers - - and append a stub entry: - - .. code-block:: yaml - - - compiler: - target: x86_64 - operating_system: centos6 - modules: [] - spec: intel@18.0.3 - paths: - cc: /usr/bin/true - cxx: /usr/bin/true - f77: /usr/bin/true - fc: /usr/bin/true - - Replace ``18.0.3`` with the version that you determined in the preceding - step. The exact contents under ``paths:`` do not matter yet, but the paths must exist. - - This temporary stub is required such that the ``intel-parallel-studio`` package - can be installed for the ``intel`` compiler (which the package itself is going - to provide after the installation) rather than an arbitrary system compiler. - The paths given in ``cc``, ``cxx``, ``f77``, ``fc`` must exist, but will - never be used to build anything during the installation of ``intel-parallel-studio``. - - The reason for this stub is that ``intel-parallel-studio`` also provides the - ``mpi`` and ``mkl`` packages and when concretizing a spec, Spack ensures - strong consistency of the used compiler across all dependencies: [fn8]_. - Installing a package ``foo +mkl %intel`` will make Spack look for a package - ``mkl %intel``, which can be provided by ``intel-parallel-studio+mkl %intel``, - but not by ``intel-parallel-studio+mkl %gcc``. - - Failure to do so may result in additional installations of ``mkl``, ``intel-mpi`` or - even ``intel-parallel-studio`` as dependencies for other packages. - - .. _`verify-compiler-anticipated`: - -3. Verify that the compiler version provided by the new ``studio`` version - would be used as expected if you were to compile a client package: - - .. code-block:: console - - $ spack spec zlib %intel - - If the version does not match, explicitly state the anticipated compiler version, e.g.: - - .. code-block:: console - - $ spack spec zlib %intel@18.0.3 - - if there are problems, review and correct the compiler's ``compilers.yaml`` - entry, be it still in stub form or already complete (as it would be for a - re-installation). - -4. Install the new ``studio`` package using Spack's regular ``install`` - command. - It may be wise to provide the anticipated compiler (`see above - `_) as an explicit concretization - element: - - .. code-block:: console - - $ spack install intel-parallel-studio@cluster.2018.3 %intel@18.0.3 - -5. Follow the same steps as under `Integrating external compilers`_ to tell - Spack the minutiae for actually using those compilers with client packages. - If you placed a stub entry in a ``compilers.yaml`` file, now is the time to - edit it and fill in the particulars. - - * Under ``paths:``, give the full paths to the actual compiler binaries (``icc``, - ``ifort``, etc.) located within the Spack installation tree, in all their - unsightly length [fn9]_. - - To determine the full path to the C compiler, adapt and run: - - .. code-block:: console - - $ find `spack location -i intel-parallel-studio@cluster.2018.3` \ - -name icc -type f -ls - - If you get hits for both ``intel64`` and ``ia32``, you almost certainly will - want to use the ``intel64`` variant. The ``icpc`` and ``ifort`` compilers - will be located in the same directory as ``icc``. - - * Make sure to specify ``modules: ['intel-parallel-studio-cluster2018.3-intel-18.0.3-HASH']`` - (with ``HASH`` being the short hash as displayed when running - ``spack find -l intel-parallel-studio@cluster.2018.3`` and the versions adapted accordingly) - to ensure that the correct and complete environment for the Intel compilers gets - loaded when running them. With modern versions of the Intel compiler you may otherwise see - issues about missing libraries. Please also note that module name must exactly match - the name as returned by ``module avail`` (and shown in the example above). - - * Use the ``modules:`` and/or ``cflags:`` tokens to further specify a suitable accompanying - ``gcc`` version to help pacify picky client packages that ask for C++ - standards more recent than supported by your system-provided ``gcc`` and its - ``libstdc++.so``. - - * If you specified a custom variant (for example ``+vtune``) you may want to add this as your - preferred variant in the packages configuration for the ``intel-parallel-studio`` package - as described in :ref:`package-preferences`. Otherwise you will have to specify - the variant every time ``intel-parallel-studio`` is being used as ``mkl``, ``fftw`` or ``mpi`` - implementation to avoid pulling in a different variant. - - * To set the Intel compilers for default use in Spack, instead of the usual ``%gcc``, - follow section `Selecting Intel compilers`_. - -.. tip:: - - Compiler packages like ``intel-parallel-studio`` can easily be above 10 GB - in size, which can tax the disk space available for temporary files on - small, busy, or restricted systems (like virtual machines). The Intel - installer will stop and report insufficient space as:: - - ==> './install.sh' '--silent' 'silent.cfg' - ... - Missing critical prerequisite - -- Not enough disk space - - As first remedy, clean Spack's existing staging area: - - .. code-block:: console - - $ spack clean --stage - - then retry installing the large package. Spack normally cleans staging - directories but certain failures may prevent it from doing so. - - If the error persists, tell Spack to use an alternative location for - temporary files: - - 1. Run ``df -h`` to identify an alternative location on your system. - - 2. Tell Spack to use that location for staging. Do **one** of the following: - - * Run Spack with the environment variable ``TMPDIR`` altered for just a - single command. For example, to use your ``$HOME`` directory: - - .. code-block:: console - - $ TMPDIR="$HOME/spack-stage" spack install .... - - This example uses Bourne shell syntax. Adapt for other shells as needed. - - * Alternatively, customize - Spack's ``build_stage`` :ref:`configuration setting `. - - .. code-block:: console - - $ spack config edit config - - Append: - - .. code-block:: yaml - - config: - build_stage: - - /home/$user/spack-stage - - Do not duplicate the ``config:`` line if it already is present. - Adapt the location, which here is the same as in the preceding example. - - 3. Retry installing the large package. - - -.. _intel-install-libs: - -"""""""""""""""""""""""""""""""""""""""""""""""""""""""" -Install steps for library-only packages -"""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -To install library-only packages like ``intel-mkl``, ``intel-mpi``, and ``intel-daal`` -follow the steps given here. -For packages that contain a compiler, follow `the previous section -`_ instead. - -1. For pre-2017 product releases, review the section `Configuring Spack to use Intel licenses`_. - -2. Inspect the package spec. Specify an explicit compiler if necessary, e.g.: - - .. code-block:: console - - $ spack spec intel-mpi@2018.3.199 - $ spack spec intel-mpi@2018.3.199 %intel - - Check that the package will use the compiler flavor and version that you expect. - -3. Install the package normally within Spack. Use the same spec as in the - previous command, i.e., as general or as specific as needed: - - .. code-block:: console - - $ spack install intel-mpi@2018.3.199 - $ spack install intel-mpi@2018.3.199 %intel@18 - -4. To prepare the new packages for use with client packages, - follow `Selecting libraries to satisfy virtual packages`_. - - -"""""""""""""""" -Debug notes -"""""""""""""""" - -* You can trigger a wall of additional diagnostics using Spack options, e.g.: - - .. code-block:: console - - $ spack --debug -v install intel-mpi - - The ``--debug`` option can also be useful while installing client - packages `(see below) `_ to confirm the integration of the Intel tools in Spack, notably - MKL and MPI. - -* The ``.spack/`` subdirectory of an installed ``IntelPackage`` will contain, - besides Spack's usual archival items, a copy of the ``silent.cfg`` file that - was passed to the Intel installer: - - .. code-block:: console - - $ grep COMPONENTS ...intel-mpi.../.spack/silent.cfg - COMPONENTS=ALL - -* If an installation error occurs, Spack will normally clean up and remove a - partially installed target directory. You can direct Spack to keep it using - ``--keep-prefix``, e.g.: - - .. code-block:: console - - $ spack install --keep-prefix intel-mpi - - You must, however, *remove such partial installations* prior to subsequent - installation attempts. Otherwise, the Intel installer will behave - incorrectly. - - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Using Intel tools in Spack to install client packages -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Finally, this section pertains to `route 3`_ from the introduction. - -Once Intel tools are installed within Spack as external or internal packages -they can be used as intended for installing client packages. - - -.. _`select-intel-compilers`: - -"""""""""""""""""""""""""" -Selecting Intel compilers -"""""""""""""""""""""""""" - -Select Intel compilers to compile client packages, like any compiler in Spack, -by one of the following means: - -* Request the Intel compilers explicitly in the client spec, e.g.: - - .. code-block:: console - - $ spack install libxc@3.0.0%intel - - -* Alternatively, request Intel compilers implicitly by package preferences. - Configure the order of compilers in the appropriate ``packages.yaml`` file, - under either an ``all:`` or client-package-specific entry, in a - ``compiler:`` list. Consult the Spack documentation for - `Configuring Package Preferences `_ - and - :ref:`Package Preferences `. - -Example: ``etc/spack/packages.yaml`` might simply contain: - -.. code-block:: yaml - - packages: - all: - compiler: [ intel, gcc, ] - -To be more specific, you can state partial or full compiler version numbers, -for example: - -.. code-block:: yaml - - packages: - all: - compiler: [ intel@18, intel@17, gcc@4.4.7, gcc@4.9.3, gcc@7.3.0, ] - - -.. _intel-virtual-packages: - -"""""""""""""""""""""""""""""""""""""""""""""""" -Selecting libraries to satisfy virtual packages -"""""""""""""""""""""""""""""""""""""""""""""""" - -Intel packages, whether integrated into Spack as external packages or -installed within Spack, can be called upon to satisfy the requirement of a -client package for a library that is available from different providers. -The relevant virtual packages for Intel are ``blas``, ``lapack``, -``scalapack``, and ``mpi``. - -In both integration routes, Intel packages can have optional -:ref:`variants ` -which alter the list of virtual packages they can satisfy. For Spack-external -packages, the active variants are a combination of the defaults declared in -Spack's package repository and the spec it is declared as in ``packages.yaml``. -Needless to say, those should match the components that are actually present in -the external product installation. Likewise, for Spack-internal packages, the -active variants are determined, persistently at installation time, from the -defaults in the repository and the spec selected to be installed. - -To have Intel packages satisfy virtual package requests for all or selected -client packages, edit the ``packages.yaml`` file. Customize, either in the -``all:`` or a more specific entry, a ``providers:`` dictionary whose keys are -the virtual packages and whose values are the Spack specs that satisfy the -virtual package, in order of decreasing preference. To learn more about the -``providers:`` settings, see the Spack tutorial for -`Configuring Package Preferences `_ -and the section -:ref:`Package Preferences `. - -Example: The following fairly minimal example for ``packages.yaml`` shows how -to exclusively use the standalone ``intel-mkl`` package for all the linear -algebra virtual packages in Spack, and ``intel-mpi`` as the preferred MPI -implementation. Other providers can still be chosen on a per-package basis. - -.. code-block:: yaml - - packages: - all: - providers: - mpi: [intel-mpi] - blas: [intel-mkl] - lapack: [intel-mkl] - scalapack: [intel-mkl] - -If you have access to the ``intel-parallel-studio@cluster`` edition, you can -use instead: - -.. code-block:: yaml - - all: - providers: - mpi: [intel-parallel-studio+mpi] - # Note: +mpi vs. +mkl - blas: [intel-parallel-studio+mkl] - lapack: [intel-parallel-studio+mkl] - scalapack: [intel-parallel-studio+mkl] - -If you installed ``intel-parallel-studio`` within Spack ("`route 2`_"), make -sure you followed the `special installation step -`_ to ensure that its virtual packages match the -compilers it provides. - - -"""""""""""""""""""""""""""""""""""""""""""" -Using Intel tools as explicit dependency -"""""""""""""""""""""""""""""""""""""""""""" - -With the proper installation as detailed above, no special steps should be -required when a client package specifically (and thus deliberately) requests an -Intel package as dependency, this being one of the target use cases for Spack. - -.. _using-mkl-tips: - -""""""""""""""""""""""""""""""""""""""""""""""" -Tips for configuring client packages to use MKL -""""""""""""""""""""""""""""""""""""""""""""""" - -The Math Kernel Library (MKL) is provided by several Intel packages, currently -``intel-parallel-studio`` when variant ``+mkl`` is active (it is by default) -and the standalone ``intel-mkl``. Because of these different provider packages, -a *virtual* ``mkl`` package is declared in Spack. - -* To use MKL-specific APIs in a client package: - - Declare a dependency on ``mkl``, rather than a specific provider like - ``intel-mkl``. Declare the dependency either absolutely or conditionally - based on variants that your package might have declared: - - .. code-block:: python - - # Examples for absolute and conditional dependencies: - depends_on("mkl") - depends_on("mkl", when="+mkl") - depends_on("mkl", when="fftw=mkl") - - The ``MKLROOT`` environment variable (part of the documented API) will be set - during all stages of client package installation, and is available to both - the Spack packaging code and the client code. - -* To use MKL as provider for BLAS, LAPACK, or ScaLAPACK: - - The packages that provide ``mkl`` also provide the narrower - virtual ``blas``, ``lapack``, and ``scalapack`` packages. - See the relevant :ref:`Packaging Guide section ` - for an introduction. - To portably use these virtual packages, construct preprocessor and linker - option strings in your package configuration code using the package functions - ``.headers`` and ``.libs`` in conjunction with utility functions from the - following classes: - - * :py:class:`llnl.util.filesystem.FileList`, - * :py:class:`llnl.util.filesystem.HeaderList`, - * :py:class:`llnl.util.filesystem.LibraryList`. - - .. tip:: - *Do not* use constructs like ``.prefix.include`` or ``.prefix.lib``, with - Intel or any other implementation of ``blas``, ``lapack``, and - ``scalapack``. - - For example, for an - :ref:`AutotoolsPackage ` - use ``.libs.ld_flags`` to transform the library file list into linker options - passed to ``./configure``: - - .. code-block:: python - - def configure_args(self): - args = [] - ... - args.append("--with-blas=%s" % self.spec["blas"].libs.ld_flags) - args.append("--with-lapack=%s" % self.spec["lapack"].libs.ld_flags) - ... - - .. tip:: - Even though ``.ld_flags`` will return a string of multiple words, *do not* - use quotes for options like ``--with-blas=...`` because Spack passes them - to ``./configure`` without invoking a shell. - - Likewise, in a - :ref:`MakefilePackage ` - or similar package that does not use AutoTools you may need to provide include - and link options for use on command lines or in environment variables. - For example, to generate an option string of the form ``-I``, use: - - .. code-block:: python - - self.spec["blas"].headers.include_flags - - and to generate linker options (``-L -llibname ...``), use the same as above, - - .. code-block:: python - - self.spec["blas"].libs.ld_flags - - See - :ref:`MakefilePackage ` - and more generally the - :ref:`Packaging Guide ` - for background and further examples. - - -^^^^^^^^^^ -Footnotes -^^^^^^^^^^ - -.. [fn1] Strictly speaking, versions from ``2017.2`` onward. - -.. [fn2] The package ``intel`` intentionally does not have a ``+mpi`` variant since - it is meant to be small. The native installer will always add MPI *runtime* - components because it follows defaults defined in the download package, even - when ``intel-parallel-studio ~mpi`` has been requested. - - For ``intel-parallel-studio +mpi``, the class function - :py:func:``.IntelPackage.pset_components`` - will include ``"intel-mpi intel-imb"`` in a list of component patterns passed - to the Intel installer. The installer will extend each pattern word with an - implied glob-like ``*`` to resolve it to package names that are - *actually present in the product BOM*. - As a side effect, this pattern approach accommodates occasional package name - changes, e.g., capturing both ``intel-mpirt`` and ``intel-mpi-rt`` . - -.. [fn3] How could the external installation have succeeded otherwise? - -.. [fn4] According to Intel's documentation, there is supposedly a way to install a - product using a network license even `when a FLEXlm server is not running - `_: - Specify the license in the form ``port@serverhost`` in the - ``INTEL_LICENSE_FILE`` environment variable. All other means of specifying a - network license require that the license server be up. - -.. [fn5] Despite the name, ``INTEL_LICENSE_FILE`` can hold several and diverse entries. - They can be either directories (presumed to contain ``*.lic`` files), file - names, or network locations in the form ``port@host`` (on Linux and Mac), - with all items separated by ":" (on Linux and Mac). - -.. [fn6] Should said editor turn out to be ``vi``, you better be in a position - to know how to use it. - -.. [fn7] Comment lines in FLEXlm files, indicated by ``#`` as the first - non-whitespace character on the line, are generally allowed anywhere in the file. - There `have been reports `_, - however, that as of 2018, ``SERVER`` and ``USE_SERVER`` lines must precede - any comment lines. - -.. - .. [fnX] The name component ``intel`` of the compiler spec is separate from (in - a different namespace than) the names of the Spack packages - ``intel-parallel-studio`` and ``intel``. Both of the latter provide the former. - -.. [fn8] Spack's close coupling of installed packages to compilers, which both - necessitates the detour for installing ``intel-parallel-studio``, and - largely limits any of its provided virtual packages to a single compiler, heavily - favors `recommending to install Intel Parallel Studio outside of Spack - `_ and declare it for Spack in ``packages.yaml`` - by a `compiler-less spec `_. - -.. [fn9] With some effort, you can convince Spack to use shorter paths. - - .. warning:: Altering the naming scheme means that Spack will lose track of - all packages it has installed for you so far. - That said, the time is right for this kind of customization - when you are defining a new set of compilers. - - The relevant tunables are: - - 1. Set the ``install_tree`` location in ``config.yaml`` - (:ref:`see doc `). - 2. Set the hash length in ``install-path-scheme``, also in ``config.yaml`` - (:ref:`q.v. `). - 3. You will want to set the *same* hash length for - :ref:`module files ` - if you have Spack produce them for you, under ``projections`` in - ``modules.yaml``. diff --git a/lib/spack/docs/build_systems/rocmpackage.rst b/lib/spack/docs/build_systems/rocmpackage.rst index 36ced308abb..41a6f43e22b 100644 --- a/lib/spack/docs/build_systems/rocmpackage.rst +++ b/lib/spack/docs/build_systems/rocmpackage.rst @@ -12,8 +12,7 @@ The ``ROCmPackage`` is not a build system but a helper package. Like ``CudaPacka it provides standard variants, dependencies, and conflicts to facilitate building packages using GPUs though for AMD in this case. -You can find the source for this package (and suggestions for setting up your -``compilers.yaml`` and ``packages.yaml`` files) at +You can find the source for this package (and suggestions for setting up your ``packages.yaml`` file) at ``__. ^^^^^^^^ diff --git a/lib/spack/docs/configuration.rst b/lib/spack/docs/configuration.rst index 700e7455a52..3f092291a92 100644 --- a/lib/spack/docs/configuration.rst +++ b/lib/spack/docs/configuration.rst @@ -11,7 +11,7 @@ Configuration Files Spack has many configuration files. Here is a quick list of them, in case you want to skip directly to specific docs: -* :ref:`compilers.yaml ` +* :ref:`packages.yaml ` * :ref:`concretizer.yaml ` * :ref:`config.yaml ` * :ref:`include.yaml ` @@ -95,7 +95,7 @@ are six configuration scopes. From lowest to highest: precedence over all other scopes. Each configuration directory may contain several configuration files, -such as ``config.yaml``, ``compilers.yaml``, or ``mirrors.yaml``. When +such as ``config.yaml``, ``packages.yaml``, or ``mirrors.yaml``. When configurations conflict, settings from higher-precedence scopes override lower-precedence settings. diff --git a/lib/spack/docs/environments.rst b/lib/spack/docs/environments.rst index b73918c5211..3221fc58e02 100644 --- a/lib/spack/docs/environments.rst +++ b/lib/spack/docs/environments.rst @@ -686,7 +686,7 @@ the environment. spack: include: - environment/relative/path/to/config.yaml - - https://github.com/path/to/raw/config/compilers.yaml + - https://github.com/path/to/raw/config/packages.yaml - /absolute/path/to/packages.yaml - path: /path/to/$os/$target/environment optional: true diff --git a/lib/spack/docs/getting_started.rst b/lib/spack/docs/getting_started.rst index 544410fe8f1..a0870ec86e4 100644 --- a/lib/spack/docs/getting_started.rst +++ b/lib/spack/docs/getting_started.rst @@ -254,12 +254,11 @@ directory. Compiler configuration ---------------------- -Spack has the ability to build packages with multiple compilers and -compiler versions. Compilers can be made available to Spack by -specifying them manually in ``compilers.yaml`` or ``packages.yaml``, -or automatically by running ``spack compiler find``, but for -convenience Spack will automatically detect compilers the first time -it needs them. +Spack has the ability to build packages with multiple compilers and compiler versions. +Compilers can be made available to Spack by specifying them manually in ``packages.yaml``, +or automatically by running ``spack compiler find``. +For convenience, Spack will automatically detect compilers the first time it needs them, +if none is available. .. _cmd-spack-compilers: @@ -274,16 +273,11 @@ compilers`` or ``spack compiler list``: $ spack compilers ==> Available compilers - -- gcc --------------------------------------------------------- - gcc@4.9.0 gcc@4.8.0 gcc@4.7.0 gcc@4.6.2 gcc@4.4.7 - gcc@4.8.2 gcc@4.7.1 gcc@4.6.3 gcc@4.6.1 gcc@4.1.2 - -- intel ------------------------------------------------------- - intel@15.0.0 intel@14.0.0 intel@13.0.0 intel@12.1.0 intel@10.0 - intel@14.0.3 intel@13.1.1 intel@12.1.5 intel@12.0.4 intel@9.1 - intel@14.0.2 intel@13.1.0 intel@12.1.3 intel@11.1 - intel@14.0.1 intel@13.0.1 intel@12.1.2 intel@10.1 - -- clang ------------------------------------------------------- - clang@3.4 clang@3.3 clang@3.2 clang@3.1 + -- gcc ubuntu20.04-x86_64 --------------------------------------- + gcc@9.4.0 gcc@8.4.0 gcc@10.5.0 + + -- llvm ubuntu20.04-x86_64 -------------------------------------- + llvm@12.0.0 llvm@11.0.0 llvm@10.0.0 Any of these compilers can be used to build Spack packages. More on how this is done is in :ref:`sec-specs`. @@ -302,16 +296,22 @@ An alias for ``spack compiler find``. ``spack compiler find`` ^^^^^^^^^^^^^^^^^^^^^^^ -Lists the compilers currently available to Spack. If you do not see -a compiler in this list, but you want to use it with Spack, you can -simply run ``spack compiler find`` with the path to where the -compiler is installed. For example: +If you do not see a compiler in the list shown by: .. code-block:: console - $ spack compiler find /usr/local/tools/ic-13.0.079 - ==> Added 1 new compiler to ~/.spack/linux/compilers.yaml - intel@13.0.079 + $ spack compiler list + +but you want to use it with Spack, you can simply run ``spack compiler find`` with the +path to where the compiler is installed. For example: + +.. code-block:: console + + $ spack compiler find /opt/intel/oneapi/compiler/2025.1/bin/ + ==> Added 1 new compiler to /home/user/.spack/packages.yaml + intel-oneapi-compilers@2025.1.0 + ==> Compilers are defined in the following files: + /home/user/.spack/packages.yaml Or you can run ``spack compiler find`` with no arguments to force auto-detection. This is useful if you do not know where compilers are @@ -322,7 +322,7 @@ installed, but you know that new compilers have been added to your $ module load gcc/4.9.0 $ spack compiler find - ==> Added 1 new compiler to ~/.spack/linux/compilers.yaml + ==> Added 1 new compiler to /home/user/.spack/packages.yaml gcc@4.9.0 This loads the environment module for gcc-4.9.0 to add it to @@ -331,7 +331,7 @@ This loads the environment module for gcc-4.9.0 to add it to .. note:: By default, spack does not fill in the ``modules:`` field in the - ``compilers.yaml`` file. If you are using a compiler from a + ``packages.yaml`` file. If you are using a compiler from a module, then you should add this field manually. See the section on :ref:`compilers-requiring-modules`. @@ -341,91 +341,82 @@ This loads the environment module for gcc-4.9.0 to add it to ``spack compiler info`` ^^^^^^^^^^^^^^^^^^^^^^^ -If you want to see specifics on a particular compiler, you can run -``spack compiler info`` on it: +If you want to see additional information on some specific compilers, you can run ``spack compiler info`` on it: .. code-block:: console - $ spack compiler info intel@15 - intel@15.0.0: - paths: - cc = /usr/local/bin/icc-15.0.090 - cxx = /usr/local/bin/icpc-15.0.090 - f77 = /usr/local/bin/ifort-15.0.090 - fc = /usr/local/bin/ifort-15.0.090 - modules = [] - operating_system = centos6 - ... + $ spack compiler info gcc + gcc@=8.4.0 languages='c,c++,fortran' arch=linux-ubuntu20.04-x86_64: + prefix: /usr + compilers: + c: /usr/bin/gcc-8 + cxx: /usr/bin/g++-8 + fortran: /usr/bin/gfortran-8 -This shows which C, C++, and Fortran compilers were detected by Spack. -Notice also that we didn't have to be too specific about the -version. We just said ``intel@15``, and information about the only -matching Intel compiler was displayed. + gcc@=9.4.0 languages='c,c++,fortran' arch=linux-ubuntu20.04-x86_64: + prefix: /usr + compilers: + c: /usr/bin/gcc + cxx: /usr/bin/g++ + fortran: /usr/bin/gfortran + + gcc@=10.5.0 languages='c,c++,fortran' arch=linux-ubuntu20.04-x86_64: + prefix: /usr + compilers: + c: /usr/bin/gcc-10 + cxx: /usr/bin/g++-10 + fortran: /usr/bin/gfortran-10 + +This shows the details of the compilers that were detected by Spack. +Notice also that we didn't have to be too specific about the version. We just said ``gcc``, and we got information +about all the matching compilers. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Manual compiler configuration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -If auto-detection fails, you can manually configure a compiler by -editing your ``~/.spack//compilers.yaml`` file. You can do this by running -``spack config edit compilers``, which will open the file in +If auto-detection fails, you can manually configure a compiler by editing your ``~/.spack/packages.yaml`` file. +You can do this by running ``spack config edit packages``, which will open the file in :ref:`your favorite editor `. -Each compiler configuration in the file looks like this: +Each compiler has an "external" entry in the file with some ``extra_attributes``: .. code-block:: yaml - compilers: - - compiler: - modules: [] - operating_system: centos6 - paths: - cc: /usr/local/bin/icc-15.0.024-beta - cxx: /usr/local/bin/icpc-15.0.024-beta - f77: /usr/local/bin/ifort-15.0.024-beta - fc: /usr/local/bin/ifort-15.0.024-beta - spec: intel@15.0.0 + packages: + gcc: + externals: + - spec: gcc@10.5.0 languages='c,c++,fortran' + prefix: /usr + extra_attributes: + compilers: + c: /usr/bin/gcc-10 + cxx: /usr/bin/g++-10 + fortran: /usr/bin/gfortran-10 -For compilers that do not support Fortran (like ``clang``), put -``None`` for ``f77`` and ``fc``: +The compiler executables are listed under ``extra_attributes:compilers``, and are keyed by language. +Once you save the file, the configured compilers will show up in the list displayed by ``spack compilers``. -.. code-block:: yaml - - compilers: - - compiler: - modules: [] - operating_system: centos6 - paths: - cc: /usr/bin/clang - cxx: /usr/bin/clang++ - f77: None - fc: None - spec: clang@3.3svn - -Once you save the file, the configured compilers will show up in the -list displayed by ``spack compilers``. - -You can also add compiler flags to manually configured compilers. These -flags should be specified in the ``flags`` section of the compiler -specification. The valid flags are ``cflags``, ``cxxflags``, ``fflags``, +You can also add compiler flags to manually configured compilers. These flags should be specified in the +``flags`` section of the compiler specification. The valid flags are ``cflags``, ``cxxflags``, ``fflags``, ``cppflags``, ``ldflags``, and ``ldlibs``. For example: .. code-block:: yaml - compilers: - - compiler: - modules: [] - operating_system: centos6 - paths: - cc: /usr/bin/gcc - cxx: /usr/bin/g++ - f77: /usr/bin/gfortran - fc: /usr/bin/gfortran - flags: - cflags: -O3 -fPIC - cxxflags: -O3 -fPIC - cppflags: -O3 -fPIC - spec: gcc@4.7.2 + packages: + gcc: + externals: + - spec: gcc@10.5.0 languages='c,c++,fortran' + prefix: /usr + extra_attributes: + compilers: + c: /usr/bin/gcc-10 + cxx: /usr/bin/g++-10 + fortran: /usr/bin/gfortran-10 + flags: + cflags: -O3 -fPIC + cxxflags: -O3 -fPIC + cppflags: -O3 -fPIC These flags will be treated by spack as if they were entered from the command line each time this compiler is used. The compiler wrappers @@ -440,95 +431,44 @@ These variables should be specified in the ``environment`` section of the compil specification. The operations available to modify the environment are ``set``, ``unset``, ``prepend_path``, ``append_path``, and ``remove_path``. For example: -.. code-block:: yaml - - compilers: - - compiler: - modules: [] - operating_system: centos6 - paths: - cc: /opt/intel/oneapi/compiler/latest/linux/bin/icx - cxx: /opt/intel/oneapi/compiler/latest/linux/bin/icpx - f77: /opt/intel/oneapi/compiler/latest/linux/bin/ifx - fc: /opt/intel/oneapi/compiler/latest/linux/bin/ifx - spec: oneapi@latest - environment: - set: - MKL_ROOT: "/path/to/mkl/root" - unset: # A list of environment variables to unset - - CC - prepend_path: # Similar for append|remove_path - LD_LIBRARY_PATH: /ld/paths/added/by/setvars/sh - -.. note:: - - Spack is in the process of moving compilers from a separate - attribute to be handled like all other packages. As part of this - process, the ``compilers.yaml`` section will eventually be replaced - by configuration in the ``packages.yaml`` section. This new - configuration is now available, although it is not yet the default - behavior. - -Compilers can also be configured as external packages in the -``packages.yaml`` config file. Any external package for a compiler -(e.g. ``gcc`` or ``llvm``) will be treated as a configured compiler -assuming the paths to the compiler executables are determinable from -the prefix. - -If the paths to the compiler executable are not determinable from the -prefix, you can add them to the ``extra_attributes`` field. Similarly, -all other fields from the compilers config can be added to the -``extra_attributes`` field for an external representing a compiler. - -Note that the format for the ``paths`` field in the -``extra_attributes`` section is different than in the ``compilers`` -config. For compilers configured as external packages, the section is -named ``compilers`` and the dictionary maps language names (``c``, -``cxx``, ``fortran``) to paths, rather than using the names ``cc``, -``fc``, and ``f77``. - .. code-block:: yaml packages: - gcc: - external: - - spec: gcc@12.2.0 arch=linux-rhel8-skylake - prefix: /usr - extra_attributes: - environment: - set: - GCC_ROOT: /usr - external: - - spec: llvm+clang@15.0.0 arch=linux-rhel8-skylake - prefix: /usr + intel-oneapi-compilers: + externals: + - spec: intel-oneapi-compilers@2025.1.0 + prefix: /opt/intel/oneapi extra_attributes: compilers: - c: /usr/bin/clang-with-suffix - cxx: /usr/bin/clang++-with-extra-info - fortran: /usr/bin/gfortran - extra_rpaths: - - /usr/lib/llvm/ + c: /opt/intel/oneapi/compiler/2025.1/bin/icx + cxx: /opt/intel/oneapi/compiler/2025.1/bin/icpx + fortran: /opt/intel/oneapi/compiler/2025.1/bin/ifx + environment: + set: + MKL_ROOT: "/path/to/mkl/root" + unset: # A list of environment variables to unset + - CC + prepend_path: # Similar for append|remove_path + LD_LIBRARY_PATH: /ld/paths/added/by/setvars/sh ^^^^^^^^^^^^^^^^^^^^^^^ Build Your Own Compiler ^^^^^^^^^^^^^^^^^^^^^^^ -If you are particular about which compiler/version you use, you might -wish to have Spack build it for you. For example: +If you are particular about which compiler/version you use, you might wish to have Spack build it for you. +For example: .. code-block:: console - $ spack install gcc@4.9.3 + $ spack install gcc@14+binutils -Once that has finished, you will need to add it to your -``compilers.yaml`` file. You can then set Spack to use it by default -by adding the following to your ``packages.yaml`` file: +Once the compiler is installed, you can start using it without additional configuration: -.. code-block:: yaml +.. code-block:: console - packages: - all: - compiler: [gcc@4.9.3] + $ spack install hdf5~mpi %gcc@14 + +The same holds true for compilers that are made available from buildcaches, when reusing them is allowed. .. _compilers-requiring-modules: @@ -536,30 +476,26 @@ by adding the following to your ``packages.yaml`` file: Compilers Requiring Modules ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Many installed compilers will work regardless of the environment they -are called with. However, some installed compilers require -``$LD_LIBRARY_PATH`` or other environment variables to be set in order -to run; this is typical for Intel and other proprietary compilers. +Many installed compilers will work regardless of the environment they are called with. +However, some installed compilers require environment variables to be set in order to run; +this is typical for Intel and other proprietary compilers. -In such a case, you should tell Spack which module(s) to load in order -to run the chosen compiler (If the compiler does not come with a -module file, you might consider making one by hand). Spack will load -this module into the environment ONLY when the compiler is run, and -NOT in general for a package's ``install()`` method. See, for -example, this ``compilers.yaml`` file: +On typical HPC clusters, these environment modifications are usually delegated to some "module" system. +In such a case, you should tell Spack which module(s) to load in order to run the chosen compiler: .. code-block:: yaml - compilers: - - compiler: - modules: [other/comp/gcc-5.3-sp3] - operating_system: SuSE11 - paths: - cc: /usr/local/other/SLES11.3/gcc/5.3.0/bin/gcc - cxx: /usr/local/other/SLES11.3/gcc/5.3.0/bin/g++ - f77: /usr/local/other/SLES11.3/gcc/5.3.0/bin/gfortran - fc: /usr/local/other/SLES11.3/gcc/5.3.0/bin/gfortran - spec: gcc@5.3.0 + packages: + gcc: + externals: + - spec: gcc@10.5.0 languages='c,c++,fortran' + prefix: /opt/compilers + extra_attributes: + compilers: + c: /opt/compilers/bin/gcc-10 + cxx: /opt/compilers/bin/g++-10 + fortran: /opt/compilers/bin/gfortran-10 + modules: [gcc/10.5.0] Some compilers require special environment settings to be loaded not just to run, but also to execute the code they build, breaking packages that @@ -580,7 +516,7 @@ Licensed Compilers ^^^^^^^^^^^^^^^^^^ Some proprietary compilers require licensing to use. If you need to -use a licensed compiler (eg, PGI), the process is similar to a mix of +use a licensed compiler, the process is similar to a mix of build your own, plus modules: #. Create a Spack package (if it doesn't exist already) to install @@ -590,24 +526,21 @@ build your own, plus modules: using Spack to load the module it just created, and running simple builds (eg: ``cc helloWorld.c && ./a.out``) -#. Add the newly-installed compiler to ``compilers.yaml`` as shown - above. +#. Add the newly-installed compiler to ``packages.yaml`` as shown above. .. _mixed-toolchains: -^^^^^^^^^^^^^^^^ -Mixed Toolchains -^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^ +Fortran compilers on macOS +^^^^^^^^^^^^^^^^^^^^^^^^^^ Modern compilers typically come with related compilers for C, C++ and Fortran bundled together. When possible, results are best if the same compiler is used for all languages. -In some cases, this is not possible. For example, starting with macOS El -Capitan (10.11), many packages no longer build with GCC, but XCode -provides no Fortran compilers. The user is therefore forced to use a -mixed toolchain: XCode-provided Clang for C/C++ and GNU ``gfortran`` for -Fortran. +In some cases, this is not possible. For example, XCode on macOS provides no Fortran compilers. +The user is therefore forced to use a mixed toolchain: XCode-provided Clang for C/C++ and e.g. +GNU ``gfortran`` for Fortran. #. You need to make sure that Xcode is installed. Run the following command: @@ -660,45 +593,25 @@ Fortran. Note: the flag is ``-license``, not ``--license``. -#. Run ``spack compiler find`` to locate Clang. - #. There are different ways to get ``gfortran`` on macOS. For example, you can install GCC with Spack (``spack install gcc``), with Homebrew (``brew install gcc``), or from a `DMG installer `_. -#. The only thing left to do is to edit ``~/.spack/darwin/compilers.yaml`` to provide - the path to ``gfortran``: +#. Run ``spack compiler find`` to locate both Apple-Clang and GCC. - .. code-block:: yaml - - compilers: - - compiler: - # ... - paths: - cc: /usr/bin/clang - cxx: /usr/bin/clang++ - f77: /path/to/bin/gfortran - fc: /path/to/bin/gfortran - spec: apple-clang@11.0.0 - - - If you used Spack to install GCC, you can get the installation prefix by - ``spack location -i gcc`` (this will only work if you have a single version - of GCC installed). Whereas for Homebrew, GCC is installed in - ``/usr/local/Cellar/gcc/x.y.z``. With the DMG installer, the correct path - will be ``/usr/local/gfortran``. +Since languages in Spack are modeled as virtual packages, ``apple-clang`` will be used to provide +C and C++, while GCC will be used for Fortran. ^^^^^^^^^^^^^^^^^^^^^ Compiler Verification ^^^^^^^^^^^^^^^^^^^^^ -You can verify that your compilers are configured properly by installing a -simple package. For example: +You can verify that your compilers are configured properly by installing a simple package. For example: .. code-block:: console - $ spack install zlib%gcc@5.3.0 + $ spack install zlib-ng%gcc@5.3.0 .. _vendor-specific-compiler-configuration: @@ -707,9 +620,7 @@ simple package. For example: Vendor-Specific Compiler Configuration -------------------------------------- -With Spack, things usually "just work" with GCC. Not so for other -compilers. This section provides details on how to get specific -compilers working. +This section provides details on how to get vendor-specific compilers working. ^^^^^^^^^^^^^^^ Intel Compilers @@ -731,8 +642,8 @@ compilers: you have installed from the ``PATH`` environment variable. If you want use a version of ``gcc`` or ``g++`` other than the default - version on your system, you need to use either the ``-gcc-name`` - or ``-gxx-name`` compiler option to specify the path to the version of + version on your system, you need to use either the ``--gcc-install-dir`` + or ``--gcc-toolchain`` compiler option to specify the path to the version of ``gcc`` or ``g++`` that you want to use." -- `Intel Reference Guide `_ @@ -740,76 +651,12 @@ compilers: Intel compilers may therefore be configured in one of two ways with Spack: using modules, or using compiler flags. -"""""""""""""""""""""""""" -Configuration with Modules -"""""""""""""""""""""""""" - -One can control which GCC is seen by the Intel compiler with modules. -A module must be loaded both for the Intel Compiler (so it will run) -and GCC (so the compiler can find the intended GCC). The following -configuration in ``compilers.yaml`` illustrates this technique: - -.. code-block:: yaml - - compilers: - - compiler: - modules: [gcc-4.9.3, intel-15.0.24] - operating_system: centos7 - paths: - cc: /opt/intel-15.0.24/bin/icc-15.0.24-beta - cxx: /opt/intel-15.0.24/bin/icpc-15.0.24-beta - f77: /opt/intel-15.0.24/bin/ifort-15.0.24-beta - fc: /opt/intel-15.0.24/bin/ifort-15.0.24-beta - spec: intel@15.0.24.4.9.3 - - -.. note:: - - The version number on the Intel compiler is a combination of - the "native" Intel version number and the GNU compiler it is - targeting. - -"""""""""""""""""""""""""" -Command Line Configuration -"""""""""""""""""""""""""" - -One can also control which GCC is seen by the Intel compiler by adding -flags to the ``icc`` command: - -#. Identify the location of the compiler you just installed: - - .. code-block:: console - - $ spack location --install-dir gcc - ~/spack/opt/spack/linux-centos7-x86_64/gcc-4.9.3-iy4rw... - -#. Set up ``compilers.yaml``, for example: - - .. code-block:: yaml - - compilers: - - compiler: - modules: [intel-15.0.24] - operating_system: centos7 - paths: - cc: /opt/intel-15.0.24/bin/icc-15.0.24-beta - cxx: /opt/intel-15.0.24/bin/icpc-15.0.24-beta - f77: /opt/intel-15.0.24/bin/ifort-15.0.24-beta - fc: /opt/intel-15.0.24/bin/ifort-15.0.24-beta - flags: - cflags: -gcc-name ~/spack/opt/spack/linux-centos7-x86_64/gcc-4.9.3-iy4rw.../bin/gcc - cxxflags: -gxx-name ~/spack/opt/spack/linux-centos7-x86_64/gcc-4.9.3-iy4rw.../bin/g++ - fflags: -gcc-name ~/spack/opt/spack/linux-centos7-x86_64/gcc-4.9.3-iy4rw.../bin/gcc - spec: intel@15.0.24.4.9.3 - - ^^^ NAG ^^^ -The Numerical Algorithms Group provides a licensed Fortran compiler. Like Clang, -this requires you to set up a :ref:`mixed-toolchains`. It is recommended to use -GCC for your C/C++ compilers. +The Numerical Algorithms Group provides a licensed Fortran compiler. +It is recommended to use GCC for your C/C++ compilers. The NAG Fortran compilers are a bit more strict than other compilers, and many packages will fail to install with error messages like: @@ -826,44 +673,40 @@ the command line: $ spack install openmpi fflags="-mismatch" -Or it can be set permanently in your ``compilers.yaml``: +Or it can be set permanently in your ``packages.yaml``: .. code-block:: yaml - - compiler: - modules: [] - operating_system: centos6 - paths: - cc: /soft/spack/opt/spack/linux-x86_64/gcc-5.3.0/gcc-6.1.0-q2zosj3igepi3pjnqt74bwazmptr5gpj/bin/gcc - cxx: /soft/spack/opt/spack/linux-x86_64/gcc-5.3.0/gcc-6.1.0-q2zosj3igepi3pjnqt74bwazmptr5gpj/bin/g++ - f77: /soft/spack/opt/spack/linux-x86_64/gcc-4.4.7/nag-6.1-jt3h5hwt5myezgqguhfsan52zcskqene/bin/nagfor - fc: /soft/spack/opt/spack/linux-x86_64/gcc-4.4.7/nag-6.1-jt3h5hwt5myezgqguhfsan52zcskqene/bin/nagfor - flags: - fflags: -mismatch - spec: nag@6.1 - + packages: + nag: + externals: + - spec: nag@6.1 + prefix: /opt/nag/bin + extra_attributes: + compilers: + fortran: /opt/nag/bin/nagfor + flags: + fflags: -mismatch --------------- System Packages --------------- -Once compilers are configured, one needs to determine which -pre-installed system packages, if any, to use in builds. This is -configured in the file ``~/.spack/packages.yaml``. For example, to use -an OpenMPI installed in /opt/local, one would use: +Once compilers are configured, one needs to determine which pre-installed system packages, +if any, to use in builds. These are also configured in the ``~/.spack/packages.yaml`` file. +For example, to use an OpenMPI installed in /opt/local, one would use: .. code-block:: yaml - packages: - openmpi: - externals: - - spec: openmpi@1.10.1 - prefix: /opt/local - buildable: False + packages: + openmpi: + buildable: False + externals: + - spec: openmpi@1.10.1 + prefix: /opt/local -In general, Spack is easier to use and more reliable if it builds all of -its own dependencies. However, there are several packages for which one -commonly needs to use system versions: +In general, *Spack is easier to use and more reliable if it builds all of its own dependencies*. +However, there are several packages for which one commonly needs to use system versions: ^^^ MPI @@ -876,8 +719,7 @@ you are unlikely to get a working MPI from Spack. Instead, use an appropriate pre-installed MPI. If you choose a pre-installed MPI, you should consider using the -pre-installed compiler used to build that MPI; see above on -``compilers.yaml``. +pre-installed compiler used to build that MPI. ^^^^^^^ OpenSSL @@ -1441,9 +1283,9 @@ To configure Spack, first run the following command inside the Spack console: spack compiler find This creates a ``.staging`` directory in our Spack prefix, along with a ``windows`` subdirectory -containing a ``compilers.yaml`` file. On a fresh Windows install with the above packages +containing a ``packages.yaml`` file. On a fresh Windows install with the above packages installed, this command should only detect Microsoft Visual Studio and the Intel Fortran -compiler will be integrated within the first version of MSVC present in the ``compilers.yaml`` +compiler will be integrated within the first version of MSVC present in the ``packages.yaml`` output. Spack provides a default ``config.yaml`` file for Windows that it will use unless overridden.