808 lines
30 KiB
ReStructuredText
808 lines
30 KiB
ReStructuredText
.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
|
|
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
|
|
|
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
|
|
|
.. _build-systems-tutorial:
|
|
|
|
==============================
|
|
Spack Package Build Systems
|
|
==============================
|
|
|
|
You may begin to notice after writing a couple of package template files a
|
|
pattern emerge for some packages. For example, you may find yourself writing
|
|
an :code:`install()` method that invokes: :code:`configure`, :code:`cmake`,
|
|
:code:`make`, :code:`make install`. You may also find yourself writing
|
|
:code:`"prefix=" + prefix` as an argument to :code:`configure` or :code:`cmake`.
|
|
Rather than having you repeat these lines for all packages, Spack has
|
|
classes that can take care of these patterns. In addition,
|
|
these package files allow for finer grained control of these build systems.
|
|
In this section, we will describe each build system and give examples on
|
|
how these can be manipulated to install a package.
|
|
|
|
-----------------------
|
|
Package Class Hierarchy
|
|
-----------------------
|
|
|
|
.. graphviz::
|
|
|
|
digraph G {
|
|
|
|
node [
|
|
shape = "record"
|
|
]
|
|
edge [
|
|
arrowhead = "empty"
|
|
]
|
|
|
|
PackageBase -> Package [dir=back]
|
|
PackageBase -> MakefilePackage [dir=back]
|
|
PackageBase -> AutotoolsPackage [dir=back]
|
|
PackageBase -> CMakePackage [dir=back]
|
|
PackageBase -> PythonPackage [dir=back]
|
|
}
|
|
|
|
The above diagram gives a high level view of the class hierarchy and how each
|
|
package relates. Each subclass inherits from the :code:`PackageBaseClass`
|
|
super class. The bulk of the work is done in this super class which includes
|
|
fetching, extracting to a staging directory and installing. Each subclass
|
|
then adds additional build-system-specific functionality. In the following
|
|
sections, we will go over examples of how to utilize each subclass and to see
|
|
how powerful these abstractions are when packaging.
|
|
|
|
-----------------
|
|
Package
|
|
-----------------
|
|
|
|
We've already seen examples of a :code:`Package` class in our walkthrough for writing
|
|
package files, so we won't be spending much time with them here. Briefly,
|
|
the Package class allows for abitrary control over the build process, whereas
|
|
subclasses rely on certain patterns (e.g. :code:`configure` :code:`make`
|
|
:code:`make install`) to be useful. :code:`Package` classes are particularly useful
|
|
for packages that have a non-conventional way of being built since the packager
|
|
can utilize some of Spack's helper functions to customize the building and
|
|
installing of a package.
|
|
|
|
-------------------
|
|
Autotools
|
|
-------------------
|
|
|
|
As we have seen earlier, packages using :code:`Autotools` use :code:`configure`,
|
|
:code:`make` and :code:`make install` commands to execute the build and
|
|
install process. In our :code:`Package` class, your typical build incantation will
|
|
consist of the following:
|
|
|
|
.. code-block:: python
|
|
|
|
def install(self, spec, prefix):
|
|
configure("--prefix=" + prefix)
|
|
make()
|
|
make("install")
|
|
|
|
You'll see that this looks similar to what we wrote in our packaging tutorial.
|
|
|
|
The :code:`Autotools` subclass aims to simplify writing package files and provides
|
|
convenience methods to manipulate each of the different phases for a :code:`Autotools`
|
|
build system.
|
|
|
|
:code:`Autotools` packages consist of four phases:
|
|
|
|
1. :code:`autoreconf()`
|
|
2. :code:`configure()`
|
|
3. :code:`build()`
|
|
4. :code:`install()`
|
|
|
|
|
|
Each of these phases have sensible defaults. Let's take a quick look at some
|
|
the internals of the :code:`Autotools` class:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack edit --build-system autotools
|
|
|
|
|
|
This will open the :code:`AutotoolsPackage` file in your text editor.
|
|
|
|
.. note::
|
|
The examples showing code for these classes is abridged to avoid having
|
|
long examples. We only show what is relevant to the packager.
|
|
|
|
|
|
.. literalinclude:: ../../../lib/spack/spack/build_systems/autotools.py
|
|
:language: python
|
|
:emphasize-lines: 33,36,54
|
|
:lines: 30-76,240-248
|
|
:linenos:
|
|
|
|
|
|
Important to note are the highlighted lines. These properties allow the
|
|
packager to set what build targets and install targets they want for their
|
|
package. If, for example, we wanted to add as our build target :code:`foo`
|
|
then we can append to our :code:`build_targets` property:
|
|
|
|
.. code-block:: python
|
|
|
|
build_targets = ["foo"]
|
|
|
|
Which is similiar to invoking make in our Package
|
|
|
|
.. code-block:: python
|
|
|
|
make("foo")
|
|
|
|
This is useful if we have packages that ignore environment variables and need
|
|
a command-line argument.
|
|
|
|
Another thing to take note of is in the :code:`configure()` method.
|
|
Here we see that the :code:`prefix` argument is already included since it is a
|
|
common pattern amongst packages using :code:`Autotools`. We then only have to
|
|
override :code:`configure_args()`, which will then return it's output to
|
|
to :code:`configure()`. Then, :code:`configure()` will append the common
|
|
arguments
|
|
|
|
Packagers also have the option to run :code:`autoreconf` in case a package
|
|
needs to update the build system and generate a new :code:`configure`. Though,
|
|
for the most part this will be unnecessary.
|
|
|
|
Let's look at the :code:`mpileaks` package.py file that we worked on earlier:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack edit mpileaks
|
|
|
|
Notice that mpileaks is a :code:`Package` class but uses the :code:`Autotools`
|
|
build system. Although this package is acceptable let's make this into an
|
|
:code:`AutotoolsPackage` class and simplify it further.
|
|
|
|
.. literalinclude:: tutorial/examples/Autotools/0.package.py
|
|
:language: python
|
|
:emphasize-lines: 9
|
|
:linenos:
|
|
|
|
We first inherit from the :code:`AutotoolsPackage` class.
|
|
|
|
|
|
Although we could keep the :code:`install()` method, most of it can be handled
|
|
by the :code:`AutotoolsPackage` base class. In fact, the only thing that needs
|
|
to be overridden is :code:`configure_args()`.
|
|
|
|
.. literalinclude:: tutorial/examples/Autotools/1.package.py
|
|
:language: python
|
|
:emphasize-lines: 25,26,27,28,29,30,31,32
|
|
:linenos:
|
|
|
|
Since Spack takes care of setting the prefix for us we can exclude that as
|
|
an argument to :code:`configure`. Our packages look simpler, and the packager
|
|
does not need to worry about whether they have properly included :code:`configure`
|
|
and :code:`make`.
|
|
|
|
This version of the :code:`mpileaks` package installs the same as the previous,
|
|
but the :code:`AutotoolsPackage` class lets us do it with a cleaner looking
|
|
package file.
|
|
|
|
-----------------
|
|
Makefile
|
|
-----------------
|
|
|
|
Packages that utilize :code:`Make` or a :code:`Makefile` usually require you
|
|
to edit a :code:`Makefile` to set up platform and compiler specific variables.
|
|
These packages are handled by the :code:`Makefile` subclass which provides
|
|
convenience methods to help write these types of packages.
|
|
|
|
A :code:`MakefilePackage` class has three phases that can be overridden. These include:
|
|
|
|
1. :code:`edit()`
|
|
2. :code:`build()`
|
|
3. :code:`install()`
|
|
|
|
Packagers then have the ability to control how a :code:`Makefile` is edited, and
|
|
what targets to include for the build phase or install phase.
|
|
|
|
Let's also take a look inside the :code:`MakefilePackage` class:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack edit --build-system makefile
|
|
|
|
Take note of the following:
|
|
|
|
|
|
.. literalinclude:: ../../../lib/spack/spack/build_systems/makefile.py
|
|
:language: python
|
|
:lines: 14,43-61,70-88
|
|
:emphasize-lines: 21,27,34
|
|
:linenos:
|
|
|
|
Similar to :code:`Autotools`, :code:`MakefilePackage` class has properties
|
|
that can be set by the packager. We can also override the different
|
|
methods highlighted.
|
|
|
|
|
|
Let's try to recreate the Bowtie_ package:
|
|
|
|
.. _Bowtie: http://bowtie-bio.sourceforge.net/index.shtml
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack create -f https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip
|
|
==> This looks like a URL for bowtie
|
|
==> Found 1 version of bowtie:
|
|
|
|
1.2.1.1 https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip
|
|
|
|
==> How many would you like to checksum? (default is 1, q to abort) 1
|
|
==> Downloading...
|
|
==> Fetching https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip
|
|
######################################################################## 100.0%
|
|
==> Checksummed 1 version of bowtie
|
|
==> This package looks like it uses the makefile build system
|
|
==> Created template for bowtie package
|
|
==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/bowtie/package.py
|
|
|
|
Once the fetching is completed, Spack will open up your text editor in the
|
|
usual fashion and create a template of a :code:`MakefilePackage` package.py.
|
|
|
|
.. literalinclude:: tutorial/examples/Makefile/0.package.py
|
|
:language: python
|
|
:linenos:
|
|
|
|
Spack was successfully able to detect that :code:`Bowtie` uses :code:`Make`.
|
|
Let's add in the rest of our details for our package:
|
|
|
|
.. literalinclude:: tutorial/examples/Makefile/1.package.py
|
|
:language: python
|
|
:emphasize-lines: 10,11,13,14,18,20
|
|
:linenos:
|
|
|
|
As we mentioned earlier, most packages using a :code:`Makefile` have hard-coded
|
|
variables that must be edited. These variables are fine if you happen to not
|
|
care about setup or types of compilers used but Spack is designed to work with
|
|
any compiler. The :code:`MakefilePackage` subclass makes it easy to edit
|
|
these :code:`Makefiles` by having an :code:`edit()` method that
|
|
can be overridden.
|
|
|
|
Let's take a look at the default :code:`Makefile` that :code:`Bowtie` provides.
|
|
If we look inside, we see that :code:`CC` and :code:`CXX` point to our GNU
|
|
compiler:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack stage bowtie
|
|
|
|
.. note::
|
|
As usual make sure you have shell support activated with spack:
|
|
:code:`source /path/to/spack_root/spack/share/spack/setup-env.sh`
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack cd -s bowtie
|
|
$ cd bowtie-1.2
|
|
$ vim Makefile
|
|
|
|
|
|
.. code-block:: make
|
|
|
|
CPP = g++ -w
|
|
CXX = $(CPP)
|
|
CC = gcc
|
|
LIBS = $(LDFLAGS) -lz
|
|
HEADERS = $(wildcard *.h)
|
|
|
|
To fix this, we need to use the :code:`edit()` method to write our custom
|
|
:code:`Makefile`.
|
|
|
|
.. literalinclude:: tutorial/examples/Makefile/2.package.py
|
|
:language: python
|
|
:emphasize-lines: 23,24,25
|
|
:linenos:
|
|
|
|
Here we use a :code:`FileFilter` object to edit our :code:`Makefile`. It takes
|
|
in a regular expression and then replaces :code:`CC` and :code:`CXX` to whatever
|
|
Spack sets :code:`CC` and :code:`CXX` environment variables to. This allows us to
|
|
build :code:`Bowtie` with whatever compiler we specify through Spack's
|
|
:code:`spec` syntax.
|
|
|
|
Let's change the build and install phases of our package:
|
|
|
|
.. literalinclude:: tutorial/examples/Makefile/3.package.py
|
|
:language: python
|
|
:emphasize-lines: 28,29,30,31,32,35,36
|
|
:linenos:
|
|
|
|
Here demonstrate another strategy that we can use to manipulate our package
|
|
We can provide command-line arguments to :code:`make()`. Since :code:`Bowtie`
|
|
can use :code:`tbb` we can either add :code:`NO_TBB=1` as a argument to prevent
|
|
:code:`tbb` support or we can just invoke :code:`make` with no arguments.
|
|
|
|
:code:`Bowtie` requires our :code:`install_target` to provide a path to
|
|
the install directory. We can do this by providing :code:`prefix=` as a command
|
|
line argument to :code:`make()`.
|
|
|
|
Let's look at a couple of other examples and go through them:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack edit esmf
|
|
|
|
Some packages allow environment variables to be set and will honor them.
|
|
Packages that use :code:`?=` for assignment in their :code:`Makefile`
|
|
can be set using environment variables. In our :code:`esmf` example we
|
|
set two environment variables in our :code:`edit()` method:
|
|
|
|
.. code-block:: python
|
|
|
|
def edit(self, spec, prefix):
|
|
for var in os.environ:
|
|
if var.startswith('ESMF_'):
|
|
os.environ.pop(var)
|
|
|
|
# More code ...
|
|
|
|
if self.compiler.name == 'gcc':
|
|
os.environ['ESMF_COMPILER'] = 'gfortran'
|
|
elif self.compiler.name == 'intel':
|
|
os.environ['ESMF_COMPILER'] = 'intel'
|
|
elif self.compiler.name == 'clang':
|
|
os.environ['ESMF_COMPILER'] = 'gfortranclang'
|
|
elif self.compiler.name == 'nag':
|
|
os.environ['ESMF_COMPILER'] = 'nag'
|
|
elif self.compiler.name == 'pgi':
|
|
os.environ['ESMF_COMPILER'] = 'pgi'
|
|
else:
|
|
msg = "The compiler you are building with, "
|
|
msg += "'{0}', is not supported by ESMF."
|
|
raise InstallError(msg.format(self.compiler.name))
|
|
|
|
As you may have noticed, we didn't really write anything to the :code:`Makefile`
|
|
but rather we set environment variables that will override variables set in
|
|
the :code:`Makefile`.
|
|
|
|
Some packages include a configuration file that sets certain compiler variables,
|
|
platform specific variables, and the location of dependencies or libraries.
|
|
If the file is simple and only requires a couple of changes, we can overwrite
|
|
those entries with our :code:`FileFilter` object. If the configuration involves
|
|
complex changes, we can write a new configuration file from scratch.
|
|
|
|
Let's look at an example of this in the :code:`elk` package:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack edit elk
|
|
|
|
.. code-block:: python
|
|
|
|
def edit(self, spec, prefix):
|
|
# Dictionary of configuration options
|
|
config = {
|
|
'MAKE': 'make',
|
|
'AR': 'ar'
|
|
}
|
|
|
|
# Compiler-specific flags
|
|
flags = ''
|
|
if self.compiler.name == 'intel':
|
|
flags = '-O3 -ip -unroll -no-prec-div'
|
|
elif self.compiler.name == 'gcc':
|
|
flags = '-O3 -ffast-math -funroll-loops'
|
|
elif self.compiler.name == 'pgi':
|
|
flags = '-O3 -lpthread'
|
|
elif self.compiler.name == 'g95':
|
|
flags = '-O3 -fno-second-underscore'
|
|
elif self.compiler.name == 'nag':
|
|
flags = '-O4 -kind=byte -dusty -dcfuns'
|
|
elif self.compiler.name == 'xl':
|
|
flags = '-O3'
|
|
config['F90_OPTS'] = flags
|
|
config['F77_OPTS'] = flags
|
|
|
|
# BLAS/LAPACK support
|
|
# Note: BLAS/LAPACK must be compiled with OpenMP support
|
|
# if the +openmp variant is chosen
|
|
blas = 'blas.a'
|
|
lapack = 'lapack.a'
|
|
if '+blas' in spec:
|
|
blas = spec['blas'].libs.joined()
|
|
if '+lapack' in spec:
|
|
lapack = spec['lapack'].libs.joined()
|
|
# lapack must come before blas
|
|
config['LIB_LPK'] = ' '.join([lapack, blas])
|
|
|
|
# FFT support
|
|
if '+fft' in spec:
|
|
config['LIB_FFT'] = join_path(spec['fftw'].prefix.lib,
|
|
'libfftw3.so')
|
|
config['SRC_FFT'] = 'zfftifc_fftw.f90'
|
|
else:
|
|
config['LIB_FFT'] = 'fftlib.a'
|
|
config['SRC_FFT'] = 'zfftifc.f90'
|
|
|
|
# MPI support
|
|
if '+mpi' in spec:
|
|
config['F90'] = spec['mpi'].mpifc
|
|
config['F77'] = spec['mpi'].mpif77
|
|
else:
|
|
config['F90'] = spack_fc
|
|
config['F77'] = spack_f77
|
|
config['SRC_MPI'] = 'mpi_stub.f90'
|
|
|
|
# OpenMP support
|
|
if '+openmp' in spec:
|
|
config['F90_OPTS'] += ' ' + self.compiler.openmp_flag
|
|
config['F77_OPTS'] += ' ' + self.compiler.openmp_flag
|
|
else:
|
|
config['SRC_OMP'] = 'omp_stub.f90'
|
|
|
|
# Libxc support
|
|
if '+libxc' in spec:
|
|
config['LIB_libxc'] = ' '.join([
|
|
join_path(spec['libxc'].prefix.lib, 'libxcf90.so'),
|
|
join_path(spec['libxc'].prefix.lib, 'libxc.so')
|
|
])
|
|
config['SRC_libxc'] = ' '.join([
|
|
'libxc_funcs.f90',
|
|
'libxc.f90',
|
|
'libxcifc.f90'
|
|
])
|
|
else:
|
|
config['SRC_libxc'] = 'libxcifc_stub.f90'
|
|
|
|
# Write configuration options to include file
|
|
with open('make.inc', 'w') as inc:
|
|
for key in config:
|
|
inc.write('{0} = {1}\n'.format(key, config[key]))
|
|
|
|
:code:`config` is just a dictionary that we can add key-value pairs to. By the
|
|
end of the :code:`edit()` method we write the contents of our dictionary to
|
|
:code:`make.inc`.
|
|
|
|
---------------
|
|
CMake
|
|
---------------
|
|
|
|
CMake_ is another common build system that has been gaining popularity. It works
|
|
in a similar manner to :code:`Autotools` but with differences in variable names,
|
|
the number of configuration options available, and the handling of shared libraries.
|
|
Typical build incantations look like this:
|
|
|
|
.. _CMake: https://cmake.org
|
|
|
|
.. code-block:: python
|
|
|
|
def install(self, spec, prefix):
|
|
cmake("-DCMAKE_INSTALL_PREFIX:PATH=/path/to/install_dir ..")
|
|
make()
|
|
make("install")
|
|
|
|
As you can see from the example above, it's very similar to invoking
|
|
:code:`configure` and :code:`make` in an :code:`Autotools` build system. However,
|
|
the variable names and options differ. Most options in CMake are prefixed
|
|
with a :code:`'-D'` flag to indicate a configuration setting.
|
|
|
|
In the :code:`CMakePackage` class we can override the following phases:
|
|
|
|
1. :code:`cmake()`
|
|
2. :code:`build()`
|
|
3. :code:`install()`
|
|
|
|
The :code:`CMakePackage` class also provides sensible defaults so we only need to
|
|
override :code:`cmake_args()`.
|
|
|
|
Let's look at these defaults in the :code:`CMakePackage` class in the :code:`_std_args()` method:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack edit --build-system cmake
|
|
|
|
.. literalinclude:: ../../../lib/spack/spack/build_systems/cmake.py
|
|
:language: python
|
|
:lines: 102-147
|
|
:emphasize-lines: 10,18,24,36,37,38,44
|
|
:linenos:
|
|
|
|
Some :code:`CMake` packages use different generators. Spack is able to support
|
|
Unix-Makefile_ generators as well as Ninja_ generators.
|
|
|
|
.. _Unix-Makefile: https://cmake.org/cmake/help/v3.4/generator/Unix%20Makefiles.html
|
|
.. _Ninja: https://cmake.org/cmake/help/v3.4/generator/Ninja.html
|
|
|
|
If no generator is specified Spack will default to :code:`Unix Makefiles`.
|
|
|
|
Next we setup the build type. In :code:`CMake` you can specify the build type
|
|
that you want. Options include:
|
|
|
|
1. :code:`empty`
|
|
2. :code:`Debug`
|
|
3. :code:`Release`
|
|
4. :code:`RelWithDebInfo`
|
|
5. :code:`MinSizeRel`
|
|
|
|
With these options you can specify whether you want your executable to have
|
|
the debug version only, release version or the release with debug information.
|
|
Release executables tend to be more optimized than Debug. In Spack, we set
|
|
the default as RelWithDebInfo unless otherwise specified through a variant.
|
|
|
|
Spack then automatically sets up the :code:`-DCMAKE_INSTALL_PREFIX` path,
|
|
appends the build type (:code:`RelWithDebInfo` default), and then specifies a verbose
|
|
:code:`Makefile`.
|
|
|
|
Next we add the :code:`rpaths` to :code:`-DCMAKE_INSTALL_RPATH:STRING`.
|
|
|
|
|
|
Finally we add to :code:`-DCMAKE_PREFIX_PATH:STRING` the locations of all our
|
|
dependencies so that :code:`CMake` can find them.
|
|
|
|
In the end our :code:`cmake` line will look like this (example is :code:`xrootd`):
|
|
|
|
.. code-block:: console
|
|
|
|
$ cmake $HOME/spack/var/spack/stage/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/xrootd-4.6.0 -G Unix Makefiles -DCMAKE_INSTALL_PREFIX:PATH=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_FIND_FRAMEWORK:STRING=LAST -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=FALSE -DCMAKE_INSTALL_RPATH:STRING=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/lib:$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/lib64 -DCMAKE_PREFIX_PATH:STRING=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/cmake-3.9.4-hally3vnbzydiwl3skxcxcbzsscaasx5
|
|
|
|
We can see now how :code:`CMake` takes care of a lot of the boilerplate code
|
|
that would have to be otherwise typed in.
|
|
|
|
Let's try to recreate callpath_:
|
|
|
|
.. _callpath: https://github.com/LLNL/callpath.git
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack create -f https://github.com/llnl/callpath/archive/v1.0.3.tar.gz
|
|
==> This looks like a URL for callpath
|
|
==> Found 4 versions of callpath:
|
|
|
|
1.0.3 https://github.com/LLNL/callpath/archive/v1.0.3.tar.gz
|
|
1.0.2 https://github.com/LLNL/callpath/archive/v1.0.2.tar.gz
|
|
1.0.1 https://github.com/LLNL/callpath/archive/v1.0.1.tar.gz
|
|
1.0 https://github.com/LLNL/callpath/archive/v1.0.tar.gz
|
|
|
|
==> How many would you like to checksum? (default is 1, q to abort) 1
|
|
==> Downloading...
|
|
==> Fetching https://github.com/LLNL/callpath/archive/v1.0.3.tar.gz
|
|
######################################################################## 100.0%
|
|
==> Checksummed 1 version of callpath
|
|
==> This package looks like it uses the cmake build system
|
|
==> Created template for callpath package
|
|
==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/callpath/package.py
|
|
|
|
|
|
which then produces the following template:
|
|
|
|
.. literalinclude:: tutorial/examples/Cmake/0.package.py
|
|
:language: python
|
|
:linenos:
|
|
|
|
Again we fill in the details:
|
|
|
|
.. literalinclude:: tutorial/examples/Cmake/1.package.py
|
|
:language: python
|
|
:linenos:
|
|
:emphasize-lines: 9,13,14,18,19,20,21,22,23
|
|
|
|
As mentioned earlier, Spack will use sensible defaults to prevent repeated code
|
|
and to make writing :code:`CMake` package files simpler.
|
|
|
|
In callpath, we want to add options to :code:`CALLPATH_WALKER` as well as add
|
|
compiler flags. We add the following options like so:
|
|
|
|
.. literalinclude:: tutorial/examples/Cmake/2.package.py
|
|
:language: python
|
|
:linenos:
|
|
:emphasize-lines: 26,30,31
|
|
|
|
Now we can control our build options using :code:`cmake_args()`. If defaults are
|
|
sufficient enough for the package, we can leave this method out.
|
|
|
|
:code:`CMakePackage` classes allow for control of other features in the
|
|
build system. For example, you can specify the path to the "out of source"
|
|
build directory and also point to the root of the :code:`CMakeLists.txt` file if it
|
|
is placed in a non-standard location.
|
|
|
|
A good example of a package that has its :code:`CMakeLists.txt` file located at a
|
|
different location is found in :code:`spades`.
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack edit spades
|
|
|
|
.. code-block:: python
|
|
|
|
root_cmakelists_dir = "src"
|
|
|
|
Here :code:`root_cmakelists_dir` will tell Spack where to find the location
|
|
of :code:`CMakeLists.txt`. In this example, it is located a directory level below in
|
|
the :code:`src` directory.
|
|
|
|
Some :code:`CMake` packages also require the :code:`install` phase to be
|
|
overridden. For example, let's take a look at :code:`sniffles`.
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack edit sniffles
|
|
|
|
In the :code:`install()` method, we have to manually install our targets
|
|
so we override the :code:`install()` method to do it for us:
|
|
|
|
.. code-block:: python
|
|
|
|
# the build process doesn't actually install anything, do it by hand
|
|
def install(self, spec, prefix):
|
|
mkdir(prefix.bin)
|
|
src = "bin/sniffles-core-{0}".format(spec.version.dotted)
|
|
binaries = ['sniffles', 'sniffles-debug']
|
|
for b in binaries:
|
|
install(join_path(src, b), join_path(prefix.bin, b))
|
|
|
|
|
|
--------------
|
|
PythonPackage
|
|
--------------
|
|
|
|
Python extensions and modules are built differently from source than most
|
|
applications. Python uses a :code:`setup.py` script to install Python modules.
|
|
The script consists of a call to :code:`setup()` which provides the information
|
|
required to build a module to Distutils. If you're familiar with pip or
|
|
easy_install, setup.py does the same thing.
|
|
|
|
These modules are usually installed using the following line:
|
|
|
|
.. code-block:: console
|
|
|
|
$ python setup.py install
|
|
|
|
There are also a list of commands and phases that you can call. To see the full
|
|
list you can run:
|
|
|
|
.. code-block:: console
|
|
|
|
$ python setup.py --help-commands
|
|
Standard commands:
|
|
build build everything needed to install
|
|
build_py "build" pure Python modules (copy to build directory)
|
|
build_ext build C/C++ extensions (compile/link to build directory)
|
|
build_clib build C/C++ libraries used by Python extensions
|
|
build_scripts "build" scripts (copy and fixup #! line)
|
|
clean (no description available)
|
|
install install everything from build directory
|
|
install_lib install all Python modules (extensions and pure Python)
|
|
install_headers install C/C++ header files
|
|
install_scripts install scripts (Python or otherwise)
|
|
install_data install data files
|
|
sdist create a source distribution (tarball, zip file, etc.)
|
|
register register the distribution with the Python package index
|
|
bdist create a built (binary) distribution
|
|
bdist_dumb create a "dumb" built distribution
|
|
bdist_rpm create an RPM distribution
|
|
bdist_wininst create an executable installer for MS Windows
|
|
upload upload binary package to PyPI
|
|
check perform some checks on the package
|
|
|
|
|
|
We can write package files for Python packages using the :code:`Package` class,
|
|
but the class brings with it a lot of methods that are useless for Python packages.
|
|
Instead, Spack has a :code:`PythonPackage` subclass that allows packagers
|
|
of Python modules to be able to invoke :code:`setup.py` and use :code:`Distutils`,
|
|
which is much more familiar to a typical python user.
|
|
|
|
To see the defaults that Spack has for each a methods, we will take a look
|
|
at the :code:`PythonPackage` class:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack edit --build-system python
|
|
|
|
We see the following:
|
|
|
|
|
|
.. literalinclude:: ../../../lib/spack/spack/build_systems/python.py
|
|
:language: python
|
|
:lines: 19,146-357
|
|
:linenos:
|
|
|
|
Each of these methods have sensible defaults or they can be overridden.
|
|
|
|
We will write a package file for Pandas_:
|
|
|
|
.. _pandas: https://pandas.pydata.org
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack create -f https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz
|
|
==> This looks like a URL for pandas
|
|
==> Warning: Spack was unable to fetch url list due to a certificate verification problem. You can try running spack -k, which will not check SSL certificates. Use this at your own risk.
|
|
==> Found 1 version of pandas:
|
|
|
|
0.19.0 https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz
|
|
|
|
==> How many would you like to checksum? (default is 1, q to abort) 1
|
|
==> Downloading...
|
|
==> Fetching https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz
|
|
######################################################################## 100.0%
|
|
==> Checksummed 1 version of pandas
|
|
==> This package looks like it uses the python build system
|
|
==> Changing package name from pandas to py-pandas
|
|
==> Created template for py-pandas package
|
|
==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/py-pandas/package.py
|
|
|
|
And we are left with the following template:
|
|
|
|
.. literalinclude:: tutorial/examples/PyPackage/0.package.py
|
|
:language: python
|
|
:linenos:
|
|
|
|
As you can see this is not any different than any package template that we have
|
|
written. We have the choice of providing build options or using the sensible
|
|
defaults
|
|
|
|
Luckily for us, there is no need to provide build args.
|
|
|
|
Next we need to find the dependencies of a package. Dependencies are usually
|
|
listed in :code:`setup.py`. You can find the dependencies by searching for
|
|
:code:`install_requires` keyword in that file. Here it is for :code:`Pandas`:
|
|
|
|
.. code-block:: python
|
|
|
|
# ... code
|
|
if sys.version_info[0] >= 3:
|
|
|
|
setuptools_kwargs = {
|
|
'zip_safe': False,
|
|
'install_requires': ['python-dateutil >= 2',
|
|
'pytz >= 2011k',
|
|
'numpy >= %s' % min_numpy_ver],
|
|
'setup_requires': ['numpy >= %s' % min_numpy_ver],
|
|
}
|
|
if not _have_setuptools:
|
|
sys.exit("need setuptools/distribute for Py3k"
|
|
"\n$ pip install distribute")
|
|
|
|
# ... more code
|
|
|
|
You can find a more comprehensive list at the Pandas documentation_.
|
|
|
|
.. _documentation: https://pandas.pydata.org/pandas-docs/stable/install.html
|
|
|
|
|
|
By reading the documentation and :code:`setup.py` we found that :code:`Pandas`
|
|
depends on :code:`python-dateutil`, :code:`pytz`, and :code:`numpy`, :code:`numexpr`,
|
|
and finally :code:`bottleneck`.
|
|
|
|
Here is the completed :code:`Pandas` script:
|
|
|
|
.. literalinclude:: tutorial/examples/PyPackage/1.package.py
|
|
:language: python
|
|
:linenos:
|
|
|
|
It is quite important to declare all the dependencies of a Python package.
|
|
Spack can "activate" Python packages to prevent the user from having to
|
|
load each dependency module explictly. If a dependency is missed, Spack will
|
|
be unable to properly activate the package and it will cause an issue. To
|
|
learn more about extensions go to :ref:`cmd-spack-extensions`.
|
|
|
|
From this example, you can see that building Python modules is made easy
|
|
through the :code:`PythonPackage` class.
|
|
|
|
-------------------
|
|
Other Build Systems
|
|
-------------------
|
|
|
|
Although we won't get in depth with any of the other build systems that Spack
|
|
supports, it is worth mentioning that Spack does provide subclasses
|
|
for the following build systems:
|
|
|
|
1. :code:`IntelPackage`
|
|
2. :code:`SconsPackage`
|
|
3. :code:`WafPackage`
|
|
4. :code:`RPackage`
|
|
5. :code:`PerlPackage`
|
|
6. :code:`QMakePackage`
|
|
|
|
|
|
Each of these classes have their own abstractions to help assist in writing
|
|
package files. For whatever doesn't fit nicely into the other build-systems,
|
|
you can use the :code:`Package` class.
|
|
|
|
Hopefully by now you can see how we aim to make packaging simple and
|
|
robust through these classes. If you want to learn more about these build
|
|
systems, check out :ref:`installation_procedure` in the Packaging Guide.
|