zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add documentation on build systems (#5015)

Browse Source 
Spack provides a number of classes based on commonly-used build systems
that users can extend when writing packages; the classes provide functionality
to perform the actions relevant to the build system (e.g. running "configure" for
an Autotools-based package). This adds documentation for classes supporting the
following build systems:

* Makefile
* Autotools
* CMake
* QMake
* SCons
* Waf

This includes build systems for managing extensions of the following packages:

* Perl
* Python
* R
* Octave

This also adds documentation on implementing packages that use a custom build
system (e.g. Perl/CMake).

Spack also provides extendable classes which aggregate functionality for related
sets of packages, e.g. those using CUDA. Documentation is added for
CudaPackage.
This commit is contained in:
Adam J. Stewart 2018-07-17 13:28:38 -05:00 committed by scheibelp
parent 25062d0bd4
commit 8ce62ba513
17 changed files with 3941 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

83
lib/spack/docs/build_systems.rst Normal file
View File

@ -0,0 +1,83 @@
.. _build-systems:
=============
Build Systems
=============
Spack defines a number of classes which understand how to use common
`build systems <https://en.wikipedia.org/wiki/List_of_build_automation_software>`_
(Makefiles, CMake, etc.). Spack package definitions can inherit these
classes in order to streamline their builds.
This guide provides information specific to each particular build system.
It assumes that you've read the :ref:`packaging-guide` and expands
on these ideas for each distinct build system that Spack supports:
.. toctree::
:maxdepth: 1
:caption: Make-based
build_systems/makefilepackage
.. toctree::
:maxdepth: 1
:caption: Make-incompatible
build_systems/sconspackage
build_systems/wafpackage
.. toctree::
:maxdepth: 1
:caption: Build-script generation
build_systems/autotoolspackage
build_systems/cmakepackage
build_systems/qmakepackage
.. toctree::
:maxdepth: 1
:caption: Language-specific
build_systems/octavepackage
build_systems/perlpackage
build_systems/pythonpackage
build_systems/rpackage
build_systems/rubypackage
.. toctree::
:maxdepth: 1
:caption: Other
build_systems/cudapackage
build_systems/intelpackage
build_systems/custompackage
For reference, the :py:mod:`Build System API docs <spack.build_systems>`
provide a list of build systems and methods/attributes that can be
overridden. If you are curious about the implementation of a particular
build system, you can view the source code by running:
.. code-block:: console
$ spack edit --build-system autotools
This will open up the ``AutotoolsPackage`` definition in your favorite
editor. In addition, if you are working with a less common build system
like QMake, SCons, or Waf, it may be useful to see examples of other
packages. You can quickly find examples by running:
.. code-block:: console
$ cd var/spack/repos/builtin/packages
$ grep -l QMakePackage */package.py
You can then view these packages with ``spack edit``.
This guide is intended to supplement the
:py:mod:`Build System API docs <spack.build_systems>` with examples of
how to override commonly used methods. It also provides rules of thumb
and suggestions for package developers who are unfamiliar with a
particular build system.

840
lib/spack/docs/build_systems/Autoconf-automake-process.svg Normal file
View File

@ -0,0 +1,840 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Generated by graphviz version 2.30.1 (20130303.0813)
-->
<!-- Title: autotools Pages: 1 -->
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
width="515pt"
height="936pt"
viewBox="0.00 0.00 515.00 936.00"
id="svg3335"
version="1.1"
inkscape:version="0.48.4 r9939"
sodipodi:docname="Autoconf-automake-process.svg">
<metadata
id="metadata3645">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
</cc:Work>
</rdf:RDF>
</metadata>
<defs
id="defs3643" />
<sodipodi:namedview
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1"
objecttolerance="10"
gridtolerance="10"
guidetolerance="10"
inkscape:pageopacity="0"
inkscape:pageshadow="2"
inkscape:window-width="1309"
inkscape:window-height="744"
id="namedview3641"
showgrid="false"
inkscape:zoom="0.70163371"
inkscape:cx="271.30388"
inkscape:cy="758.87622"
inkscape:window-x="57"
inkscape:window-y="24"
inkscape:window-maximized="1"
inkscape:current-layer="svg3335" />
<polygon
style="fill:#ffffff;stroke:#ffffff"
id="polygon3340"
points="512,-932 512,5 -4,5 -4,5 -4,-932 "
transform="translate(4,932)" />
<g
class="node"
id="node1"
transform="translate(4,932)">
<title
id="title3343">configure.ac</title>
<polygon
style="fill:#0000f0;stroke:#000000;opacity:1;fill-opacity:0.08627451"
id="polygon3345"
points="114.75,-818 209.25,-818 209.25,-854 209.25,-854 114.75,-854 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3347"
font-size="14.00"
y="-832.29999"
x="162">configure.ac</text>
</g>
<g
class="node"
id="node5"
transform="translate(4,932)">
<title
id="title3350">aclocal</title>
<ellipse
style="fill:none;stroke:#000000"
sodipodi:ry="18"
sodipodi:rx="39.469101"
sodipodi:cy="-762"
sodipodi:cx="66"
d="m 105.4691,-762 c 0,9.94113 -17.670917,18 -39.4691,18 -21.798183,0 -39.469101,-8.05887 -39.469101,-18 0,-9.94113 17.670918,-18 39.469101,-18 21.798183,0 39.4691,8.05887 39.4691,18 z"
id="ellipse3352"
ry="18"
rx="39.469101"
cy="-762"
cx="66" />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3354"
font-size="14.00"
y="-758.29999"
x="66">aclocal</text>
</g>
<g
class="edge"
id="edge3"
transform="translate(4,932)">
<title
id="title3357">configure.ac-&gt;aclocal</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3359"
d="m 139.249,-817.937 c -13.626,10.22 -31.049,23.287 -45.4828,34.112" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3361"
points="89.6641,-780.748 93.6641,-783.748 93.6641,-783.748 93.6641,-783.748 89.6641,-780.748 90.9641,-787.348 85.6641,-777.748 85.6641,-777.748 85.6641,-777.748 96.3642,-780.148 " />
</g>
<g
class="node"
id="node6"
transform="translate(4,932)">
<title
id="title3364">autoconf</title>
<ellipse
style="fill:none;stroke:#000000"
sodipodi:ry="18"
sodipodi:rx="46.219101"
sodipodi:cy="-614"
sodipodi:cx="130"
d="m 176.2191,-614 c 0,9.94113 -20.693,18 -46.2191,18 -25.5261,0 -46.219101,-8.05887 -46.219101,-18 0,-9.94113 20.693001,-18 46.219101,-18 25.5261,0 46.2191,8.05887 46.2191,18 z"
id="ellipse3366"
ry="18"
rx="46.219101"
cy="-614"
cx="130" />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3368"
font-size="14.00"
y="-610.29999"
x="130">autoconf</text>
</g>
<g
class="edge"
id="edge4"
transform="translate(4,932)">
<title
id="title3371">configure.ac-&gt;autoconf</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3373"
d="m 159.489,-817.737 c -5.538,38.073 -18.837,129.506 -25.518,175.436" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3375"
points="133.219,-637.131 133.939,-642.079 133.939,-642.079 133.939,-642.079 133.219,-637.131 129.486,-642.726 132.499,-632.183 132.499,-632.183 132.499,-632.183 138.392,-641.431 " />
</g>
<g
class="node"
id="node7"
transform="translate(4,932)">
<title
id="title3378">autoheader</title>
<ellipse
style="fill:none;stroke:#000000"
sodipodi:ry="18"
sodipodi:rx="57.292702"
sodipodi:cy="-762"
sodipodi:cx="220"
d="m 277.2927,-762 c 0,9.94113 -25.65081,18 -57.2927,18 -31.64189,0 -57.2927,-8.05887 -57.2927,-18 0,-9.94113 25.65081,-18 57.2927,-18 31.64189,0 57.2927,8.05887 57.2927,18 z"
id="ellipse3380"
ry="18"
rx="57.292702"
cy="-762"
cx="220" />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3382"
font-size="14.00"
y="-758.29999"
x="220">autoheader</text>
</g>
<g
class="edge"
id="edge5"
transform="translate(4,932)">
<title
id="title3385">configure.ac-&gt;autoheader</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3387"
d="m 175.745,-817.937 c 7.223,8.967 16.212,20.125 24.197,30.038" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3389"
points="203.217,-783.834 200.08,-787.728 200.08,-787.728 200.08,-787.728 203.217,-783.834 196.576,-784.905 206.353,-779.941 206.353,-779.941 206.353,-779.941 203.585,-790.551 " />
</g>
<g
class="node"
id="node8"
transform="translate(4,932)">
<title
id="title3392">automake</title>
<ellipse
style="fill:none;stroke:#000000"
sodipodi:ry="18"
sodipodi:rx="50.542702"
sodipodi:cy="-614"
sodipodi:cx="309"
d="m 359.5427,-614 c 0,9.94113 -22.62874,18 -50.5427,18 -27.91396,0 -50.5427,-8.05887 -50.5427,-18 0,-9.94113 22.62874,-18 50.5427,-18 27.91396,0 50.5427,8.05887 50.5427,18 z"
id="ellipse3394"
ry="18"
rx="50.542702"
cy="-614"
cx="309" />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3396"
font-size="14.00"
y="-610.29999"
x="309">automake</text>
</g>
<g
class="edge"
id="edge6"
transform="translate(4,932)">
<title
id="title3399">configure.ac-&gt;automake</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3401"
d="m 209.463,-825.823 c 26.664,7.68 58.18,21.469 76.537,45.823 30.292,40.188 30.031,101.971 26.756,137.428" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3403"
points="312.206,-637.438 312.739,-642.41 312.739,-642.41 312.739,-642.41 312.206,-637.438 308.265,-642.889 311.674,-632.467 311.674,-632.467 311.674,-632.467 317.213,-641.93 " />
</g>
<g
class="node"
id="node2"
transform="translate(4,932)">
<title
id="title3406">autoscan</title>
<ellipse
style="fill:none;stroke:#000000"
sodipodi:ry="18"
sodipodi:rx="48.1437"
sodipodi:cy="-910"
sodipodi:cx="162"
d="m 210.1437,-910 c 0,9.94113 -21.55467,18 -48.1437,18 -26.58903,0 -48.1437,-8.05887 -48.1437,-18 0,-9.94113 21.55467,-18 48.1437,-18 26.58903,0 48.1437,8.05887 48.1437,18 z"
id="ellipse3408"
ry="18"
rx="48.1437"
cy="-910"
cx="162" />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3410"
font-size="14.00"
y="-906.29999"
x="162">autoscan</text>
</g>
<g
class="edge"
id="edge1"
transform="translate(4,932)">
<title
id="title3413">autoscan-&gt;configure.ac</title>
<path
style="fill:none;stroke:#000000;stroke-width:2"
inkscape:connector-curvature="0"
id="path3415"
d="m 162,-891.937 c 0,6.716 0,14.661 0,22.392" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3417"
points="156.75,-869.441 167.25,-869.441 167.25,-869.441 162,-854.441 " />
</g>
<g
class="node"
id="node3"
transform="translate(4,932)">
<title
id="title3420">Makefile.am</title>
<polygon
style="fill:#0000f0;stroke:#000000;fill-opacity:0.08627451"
id="polygon3422"
points="333,-670 425,-670 425,-706 425,-706 333,-706 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3424"
font-size="14.00"
y="-684.29999"
x="379">Makefile.am</text>
</g>
<g
class="edge"
id="edge10"
transform="translate(4,932)">
<title
id="title3427">Makefile.am-&gt;automake</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3429"
d="m 362.411,-669.937 c -9.021,9.279 -20.323,20.903 -30.204,31.067" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3431"
points="328.474,-635.03 331.959,-638.615 331.959,-638.615 331.959,-638.615 328.474,-635.03 328.732,-641.752 324.988,-631.445 324.988,-631.445 324.988,-631.445 335.185,-635.478 " />
</g>
<g
class="node"
id="node9"
transform="translate(4,932)">
<title
id="title3434">aclocal.m4</title>
<polygon
style="fill:#0000f0;stroke:#000000;fill-opacity:0.08627451"
id="polygon3436"
points="21.75,-670 106.25,-670 106.25,-706 106.25,-706 21.75,-706 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3438"
font-size="14.00"
y="-684.29999"
x="64">aclocal.m4</text>
</g>
<g
class="edge"
id="edge7"
transform="translate(4,932)">
<title
id="title3441">aclocal-&gt;aclocal.m4</title>
<path
style="fill:none;stroke:#000000;stroke-width:2"
inkscape:connector-curvature="0"
id="path3443"
d="m 65.526,-743.937 c -0.1865,6.716 -0.4072,14.661 -0.622,22.392" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3445"
points="59.6531,-721.581 70.149,-721.289 70.149,-721.289 64.4845,-706.441 " />
</g>
<g
class="node"
id="node12"
transform="translate(4,932)">
<title
id="title3448">configure</title>
<ellipse
style="fill:#0000f0;stroke:#000000;fill-opacity:0.08627451"
sodipodi:ry="72"
sodipodi:rx="72"
sodipodi:cy="-486"
sodipodi:cx="130"
d="m 202,-486 c 0,39.7645 -32.2355,72 -72,72 -39.764502,0 -72,-32.2355 -72,-72 0,-39.7645 32.235498,-72 72,-72 39.7645,0 72,32.2355 72,72 z"
id="ellipse3450"
ry="72"
rx="72"
cy="-486"
cx="130" />
<polyline
style="fill:none;stroke:#000000"
id="polyline3452"
points="177.621,-540 82.3792,-540 " />
<polyline
style="fill:none;stroke:#000000"
id="polyline3454"
points="177.621,-432 82.3792,-432 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3456"
font-size="14.00"
y="-482.29999"
x="130">configure</text>
</g>
<g
class="edge"
id="edge13"
transform="translate(4,932)">
<title
id="title3459">autoconf-&gt;configure</title>
<path
style="fill:none;stroke:#000000;stroke-width:2"
inkscape:connector-curvature="0"
id="path3461"
d="m 130,-595.744 c 0,6.397 0,14.16 0,22.563" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3463"
points="124.75,-573.051 135.25,-573.051 135.25,-573.051 130,-558.051 " />
</g>
<g
class="node"
id="node10"
transform="translate(7.9661017,932)">
<title
id="title3466">config.h.in</title>
<polygon
style="fill:#0000f0;stroke:#000000;fill-opacity:0.08627451"
id="polygon3468"
points="260.25,-670 260.25,-706 260.25,-706 179.75,-706 179.75,-670 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3470"
font-size="14.00"
y="-684.29999"
x="220">config.h.in</text>
</g>
<g
class="edge"
id="edge8"
transform="translate(4,932)">
<title
id="title3473">autoheader-&gt;config.h.in</title>
<path
style="fill:none;stroke:#000000;stroke-width:2"
inkscape:connector-curvature="0"
id="path3475"
d="m 220,-743.937 c 0,6.716 0,14.661 0,22.392" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3477"
points="214.75,-721.441 225.25,-721.441 225.25,-721.441 220,-706.441 " />
</g>
<g
class="node"
id="node11"
transform="translate(4,932)">
<title
id="title3480">Makefile.in</title>
<polygon
style="fill:#0000f0;stroke:#000000;fill-opacity:0.08627451"
id="polygon3482"
points="260.75,-468 343.25,-468 343.25,-504 343.25,-504 260.75,-504 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3484"
font-size="14.00"
y="-482.29999"
x="302">Makefile.in</text>
</g>
<g
class="edge"
id="edge11"
transform="translate(4,932)">
<title
id="title3487">automake-&gt;Makefile.in</title>
<path
style="fill:none;stroke:#000000;stroke-width:2"
inkscape:connector-curvature="0"
id="path3489"
d="m 308.041,-595.744 c -1.089,19.604 -2.89,52.031 -4.243,76.384" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3491"
points="298.537,-519.315 309.021,-518.733 309.021,-518.733 302.947,-504.047 " />
</g>
<g
class="edge"
id="edge12"
transform="translate(4,932)">
<title
id="title3494">aclocal.m4-&gt;autoconf</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3496"
d="m 79.6411,-669.937 c 8.5054,9.279 19.1615,20.903 28.4789,31.067" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3498"
points="111.547,-635.131 108.168,-638.816 108.168,-638.816 108.168,-638.816 111.547,-635.131 104.851,-635.776 114.925,-631.445 114.925,-631.445 114.925,-631.445 111.485,-641.857 " />
</g>
<g
class="edge"
id="edge9"
transform="translate(4,932)">
<title
id="title3501">config.h.in-&gt;automake</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3503"
d="m 241.092,-669.937 c 12.174,9.849 27.617,22.342 40.698,32.925" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3505"
points="285.693,-633.855 281.806,-637 281.806,-637 281.806,-637 285.693,-633.855 278.975,-633.501 289.58,-630.71 289.58,-630.71 289.58,-630.71 284.636,-640.498 " />
</g>
<g
class="node"
id="node15"
transform="translate(4,932)">
<title
id="title3508">config.status</title>
<ellipse
style="fill:#0000f0;stroke:#000000;fill-opacity:0.08627451"
sodipodi:ry="63.088799"
sodipodi:rx="63.066601"
sodipodi:cy="-313"
sodipodi:cx="230"
d="m 293.0666,-313 c 0,34.84298 -28.23588,63.0888 -63.0666,63.0888 -34.83072,0 -63.0666,-28.24582 -63.0666,-63.0888 0,-34.84298 28.23588,-63.0888 63.0666,-63.0888 34.83072,0 63.0666,28.24582 63.0666,63.0888 z"
id="ellipse3510"
ry="63.088799"
rx="63.066601"
cy="-313"
cx="230" />
<polyline
style="fill:none;stroke:#000000"
id="polyline3512"
points="271.668,-360.283 188.332,-360.283 " />
<polyline
style="fill:none;stroke:#000000"
id="polyline3514"
points="271.668,-265.717 188.332,-265.717 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3516"
font-size="14.00"
y="-309.29999"
x="230">config.status</text>
</g>
<g
class="edge"
id="edge14"
transform="translate(4,932)">
<title
id="title3519">config.h.in-&gt;config.status</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3521"
d="m 227.40968,-670.00576 c 1.335,49.807 -1.72568,195.00376 0.65132,283.69476"
sodipodi:nodetypes="cc" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3523"
points="228.202,-381.047 228.068,-386.045 228.068,-386.045 228.068,-386.045 228.202,-381.047 223.57,-385.925 228.336,-376.049 228.336,-376.049 228.336,-376.049 232.567,-386.166 " />
</g>
<g
class="edge"
id="edge15"
transform="translate(4,932)">
<title
id="title3526">Makefile.in-&gt;config.status</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3528"
d="m 294.833,-467.979 c -8.579,20.376 -23.463,55.724 -36.807,87.417" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3530"
points="256.067,-375.909 258.007,-380.517 258.007,-380.517 258.007,-380.517 256.067,-375.909 253.86,-382.264 254.127,-371.301 254.127,-371.301 254.127,-371.301 262.155,-378.771 " />
</g>
<g
class="edge"
id="edge16"
transform="translate(4,932)">
<title
id="title3533">configure-&gt;config.status</title>
<path
style="fill:none;stroke:#000000;stroke-width:2"
inkscape:connector-curvature="0"
id="path3535"
d="m 166.102,-423.265 c 8.055,13.774 16.633,28.443 24.803,42.413" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3537"
points="186.478,-378.022 195.542,-383.323 195.542,-383.323 198.582,-367.724 " />
</g>
<g
class="node"
id="node13"
transform="translate(4,932)">
<title
id="title3540">config.h</title>
<polygon
style="fill:#0000f0;stroke:#000000;fill-opacity:0.08627451"
id="polygon3542"
points="110.75,-176 177.25,-176 177.25,-212 177.25,-212 110.75,-212 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3544"
font-size="14.00"
y="-190.3"
x="144">config.h</text>
</g>
<g
class="node"
id="node18"
transform="translate(4,932)">
<title
id="title3547">make</title>
<ellipse
style="fill:none;stroke:#000000"
sodipodi:ry="18"
sodipodi:rx="33.220901"
sodipodi:cy="-106"
sodipodi:cx="230"
d="m 263.2209,-106 c 0,9.941125 -14.8735,18 -33.2209,18 -18.3474,0 -33.2209,-8.058875 -33.2209,-18 0,-9.94113 14.8735,-18 33.2209,-18 18.3474,0 33.2209,8.05887 33.2209,18 z"
id="ellipse3549"
ry="18"
rx="33.220901"
cy="-106"
cx="230" />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3551"
font-size="14.00"
y="-102.3"
x="230">make</text>
</g>
<g
class="edge"
id="edge21"
transform="translate(4,932)">
<title
id="title3554">config.h-&gt;make</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3556"
d="m 161.403,-175.597 c 13.375,13.375 31.944,31.944 46.39,46.39" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3558"
points="211.386,-125.614 207.85,-129.15 207.85,-129.15 207.85,-129.15 211.386,-125.614 204.668,-125.968 214.922,-122.078 214.922,-122.078 214.922,-122.078 211.032,-132.332 " />
</g>
<g
class="node"
id="node14"
transform="translate(4,932)">
<title
id="title3561">Makefile</title>
<polygon
style="fill:#0000f0;stroke:#000000;fill-opacity:0.08627451"
id="polygon3563"
points="195.75,-176 264.25,-176 264.25,-212 264.25,-212 195.75,-212 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3565"
font-size="14.00"
y="-190.3"
x="230">Makefile</text>
</g>
<g
class="edge"
id="edge22"
transform="translate(4,932)">
<title
id="title3568">Makefile-&gt;make</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3570"
d="m 230,-175.597 c 0,11.851 0,27.78 0,41.305" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3572"
points="230,-129.084 230,-134.084 230,-134.084 230,-134.084 230,-129.084 225.5,-134.084 230,-124.084 230,-124.084 230,-124.084 234.5,-134.084 " />
</g>
<g
class="edge"
id="edge18"
transform="translate(4,932)">
<title
id="title3575">config.status-&gt;config.h</title>
<path
style="fill:none;stroke:#000000;stroke-width:2"
inkscape:connector-curvature="0"
id="path3577"
d="m 193.029,-261.702 c -9.358,12.731 -19.105,25.992 -27.42,37.304" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3579"
points="161.268,-227.357 169.729,-221.138 169.729,-221.138 156.615,-212.162 " />
</g>
<g
class="edge"
id="edge19"
transform="translate(4,932)">
<title
id="title3582">config.status-&gt;Makefile</title>
<path
style="fill:none;stroke:#000000;stroke-width:2"
inkscape:connector-curvature="0"
id="path3584"
d="m 230,-249.732 c 0,7.628 0,15.171 0,22.106" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3586"
points="224.75,-227.368 235.25,-227.368 235.25,-227.368 230,-212.368 " />
</g>
<g
class="node"
id="node20"
transform="translate(4,932)">
<title
id="title3589">input file</title>
<polygon
style="fill:#0000f0;stroke:#000000;fill-opacity:0.08627451"
id="polygon3591"
points="393,-176 461,-176 461,-212 461,-212 393,-212 " />
<text
style="font-size:14px;font-style:italic;text-anchor:start;font-family:URW Palladio L"
id="text3593"
font-size="14.00"
font-style="italic"
y="-191.3"
x="401.5">input file</text>
</g>
<g
class="node"
id="node19"
transform="translate(4,932)">
<title
id="title3596">executable</title>
<polygon
style="fill:#00c800;stroke:#000000;fill-opacity:0.19607843"
id="polygon3598"
points="192.75,0 267.25,0 267.25,-36 267.25,-36 192.75,-36 " />
<text
style="font-size:14px;font-style:italic;text-anchor:start;font-family:URW Palladio L"
id="text3600"
font-size="14.00"
font-style="italic"
y="-15.3"
x="201">executable</text>
</g>
<g
class="edge"
id="edge24"
transform="translate(4,932)">
<title
id="title3603">make-&gt;executable</title>
<path
style="fill:none;stroke:#000000;stroke-width:2"
inkscape:connector-curvature="0"
id="path3605"
d="m 230,-87.5966 c 0,10.4427 0,24.0522 0,36.3931" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3607"
points="224.75,-51.084 235.25,-51.084 235.25,-51.084 230,-36.084 " />
</g>
<g
class="node"
id="node21"
transform="translate(4,932)">
<title
id="title3610">process</title>
<ellipse
style="fill:none;stroke:#000000"
sodipodi:ry="18"
sodipodi:rx="37.070099"
sodipodi:cy="-106"
sodipodi:cx="427"
d="m 464.0701,-106 c 0,9.941125 -16.59685,18 -37.0701,18 -20.47325,0 -37.0701,-8.058875 -37.0701,-18 0,-9.94113 16.59685,-18 37.0701,-18 20.47325,0 37.0701,8.05887 37.0701,18 z"
id="ellipse3612"
ry="18"
rx="37.070099"
cy="-106"
cx="427" />
<text
style="font-size:14px;font-style:italic;text-anchor:start;font-family:URW Palladio L"
id="text3614"
font-size="14.00"
font-style="italic"
y="-103.3"
x="407">process</text>
</g>
<g
class="edge"
id="edge26"
transform="translate(4,932)">
<title
id="title3617">input file-&gt;process</title>
<path
style="fill:none;stroke:#000000;stroke-width:1;stroke-miterlimit:4;stroke-dasharray:12,1;stroke-dashoffset:0"
inkscape:connector-curvature="0"
id="path3619"
d="m 427,-175.597 c 0,11.851 0,27.78 0,41.305" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3621"
points="427,-129.084 427,-134.084 427,-134.084 427,-134.084 427,-129.084 422.5,-134.084 427,-124.084 427,-124.084 427,-124.084 431.5,-134.084 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3623"
font-size="14.00"
y="-146.3"
x="467"> influences  </text>
</g>
<g
class="node"
id="node22"
transform="translate(4,932)">
<title
id="title3626">output file</title>
<polygon
style="fill:#00c800;stroke:#000000;fill-opacity:0.19607843"
id="polygon3628"
points="389.75,0 464.25,0 464.25,-36 464.25,-36 389.75,-36 " />
<text
style="font-size:14px;font-style:italic;text-anchor:start;font-family:URW Palladio L"
id="text3630"
font-size="14.00"
font-style="italic"
y="-15.3"
x="398">output file</text>
</g>
<g
class="edge"
id="edge27"
transform="translate(4,932)">
<title
id="title3633">process-&gt;output file</title>
<path
style="fill:none;stroke:#000000;stroke-width:2"
inkscape:connector-curvature="0"
id="path3635"
d="m 427,-87.5966 c 0,10.4427 0,24.0522 0,36.3931" />
<polygon
style="fill:#000000;stroke:#000000"
id="polygon3637"
points="421.75,-51.084 432.25,-51.084 432.25,-51.084 427,-36.084 " />
<text
style="font-size:14px;text-anchor:middle;font-family:Liberation Sans"
id="text3639"
font-size="14.00"
y="-58.299999"
x="458.5"> creates  </text>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 27 KiB

300
lib/spack/docs/build_systems/autotoolspackage.rst Normal file
View File

@ -0,0 +1,300 @@
.. _autotoolspackage:
----------------
AutotoolsPackage
----------------
Autotools is a GNU build system that provides a build-script generator.
By running the platform-independent ``./configure`` script that comes
with the package, you can generate a platform-dependent Makefile.
^^^^^^
Phases
^^^^^^
The ``AutotoolsPackage`` base class comes with the following phases:
#. ``autoreconf`` - generate the configure script
#. ``configure`` - generate the Makefiles
#. ``build`` - build the package
#. ``install`` - install the package
Most of the time, the ``autoreconf`` phase will do nothing, but if the
package is missing a ``configure`` script, ``autoreconf`` will generate
one for you.
The other phases run:
.. code-block:: console
$ ./configure --prefix=/path/to/installation/prefix
$ make
$ make check # optional
$ make install
$ make installcheck # optional
Of course, you may need to add a few arguments to the ``./configure``
line.
^^^^^^^^^^^^^^^
Important files
^^^^^^^^^^^^^^^
The most important file for an Autotools-based package is the ``configure``
script. This script is automatically generated by Autotools and generates
the appropriate Makefile when run.
.. warning::
Watch out for fake Autotools packages!
Autotools is a very popular build system, and many people are used to the
classic steps to install a package:
.. code-block:: console
$ ./configure
$ make
$ make install
For this reason, some developers will write their own ``configure``
scripts that have nothing to do with Autotools. These packages may
not accept the same flags as other Autotools packages, so it is
better to use the ``Package`` base class and create a
:ref:`custom build system <custompackage>`. You can tell if a package
uses Autotools by running ``./configure --help`` and comparing the output
to other known Autotools packages. You should also look for files like:
* ``configure.ac``
* ``configure.in``
* ``Makefile.am``
Packages that don't use Autotools aren't likely to have these files.
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
Whether or not your package requires Autotools to install depends on
how the source code is distributed. Most of the time, when developers
distribute tarballs, they will already contain the ``configure`` script
necessary for installation. If this is the case, your package does not
require any Autotools dependencies.
However, a basic rule of version control systems is to never commit
code that can be generated. The source code repository itself likely
does not have a ``configure`` script. Developers typically write
(or auto-generate) a ``configure.ac`` script that contains configuration
preferences and a ``Makefile.am`` script that contains build instructions.
Then, ``autoconf`` is used to convert ``configure.ac`` into ``configure``,
while ``automake`` is used to convert ``Makefile.am`` into ``Makefile.in``.
``Makefile.in`` is used by ``configure`` to generate a platform-dependent
``Makefile`` for you. The following diagram provides a high-level overview
of the process:
.. figure:: Autoconf-automake-process.*
:target: https://commons.wikimedia.org/w/index.php?curid=15581407
`GNU autoconf and automake process for generating makefiles <https://commons.wikimedia.org/wiki/File:Autoconf-automake-process.svg>`_
by `Jdthood` under `CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0/deed.en>`_
If a ``configure`` script is not present in your tarball, you will
need to generate one yourself. Luckily, Spack already has an ``autoreconf``
phase to do most of the work for you. By default, the ``autoreconf``
phase runs:
.. code-block:: console
$ libtoolize
$ aclocal
$ autoreconf --install --verbose --force
All you need to do is add a few Autotools dependencies to the package.
Most stable releases will come with a ``configure`` script, but if you
check out a commit from the ``develop`` branch, you would want to add:
.. code-block:: python
depends_on('autoconf', type='build', when='@develop')
depends_on('automake', type='build', when='@develop')
depends_on('libtool', type='build', when='@develop')
depends_on('m4', type='build', when='@develop')
In some cases, developers might need to distribute a patch that modifies
one of the files used to generate ``configure`` or ``Makefile.in``.
In this case, these scripts will need to be regenerated. It is
preferable to regenerate these manually using the patch, and then
create a new patch that directly modifies ``configure``. That way,
Spack can use the secondary patch and additional build system
dependencies aren't necessary.
""""""""""""""""
force_autoreconf
""""""""""""""""
If for whatever reason you really want to add the original patch
and tell Spack to regenerate ``configure``, you can do so using the
following setting:
.. code-block:: python
force_autoreconf = True
This line tells Spack to wipe away the existing ``configure`` script
and generate a new one. If you only need to do this for a single
version, this can be done like so:
.. code-block:: python
@property
def force_autoreconf(self):
return self.version == Version('1.2.3'):
^^^^^^^^^^^^^^^^^^^^^^^
Finding configure flags
^^^^^^^^^^^^^^^^^^^^^^^
Once you have a ``configure`` script present, the next step is to
determine what option flags are available. These flags can be found
by running:
.. code-block:: console
$ ./configure --help
``configure`` will display a list of valid flags separated into
some or all of the following sections:
* Configuration
* Installation directories
* Fine tuning of the installation directories
* Program names
* X features
* System types
* **Optional Features**
* **Optional Packages**
* **Some influential environment variables**
For the most part, you can ignore all but the last 3 sections.
The "Optional Features" sections lists flags that enable/disable
features you may be interested in. The "Optional Packages" section
often lists dependencies and the flags needed to locate them. The
"environment variables" section lists environment variables that the
build system uses to pass flags to the compiler and linker.
^^^^^^^^^^^^^^^^^^^^^^^^^^
Addings flags to configure
^^^^^^^^^^^^^^^^^^^^^^^^^^
For most of the flags you encounter, you will want a variant to
optionally enable/disable them. You can then optionally pass these
flags to the ``configure`` call by overriding the ``configure_args``
function like so:
.. code-block:: python
def configure_args(self):
args = []
if '+mpi' in self.spec:
args.append('--enable-mpi')
else:
args.append('--disable-mpi')
return args
Note that we are explicitly disabling MPI support if it is not
requested. This is important, as many Autotools packages will enable
options by default if the dependencies are found, and disable them
otherwise. We want Spack installations to be as deterministic as possible.
If two users install a package with the same variants, the goal is that
both installations work the same way. See `here <https://www.linux.com/news/best-practices-autotools>`__
and `here <https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Automagic_dependencies>`__
for a rationale as to why these so-called "automagic" dependencies
are a problem.
By default, Autotools installs packages to ``/usr``. We don't want this,
so Spack automatically adds ``--prefix=/path/to/installation/prefix``
to your list of ``configure_args``. You don't need to add this yourself.
^^^^^^^^^^^^^^^^
Helper functions
^^^^^^^^^^^^^^^^
You may have noticed that most of the Autotools flags are of the form
``--enable-foo``, ``--disable-bar``, ``--with-baz=<prefix>``, or
``--without-baz``. Since these flags are so common, Spack provides a
couple of helper functions to make your life easier.
TODO: document ``with_or_without`` and ``enable_or_disable``.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Configure script in a sub-directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Occasionally, developers will hide their source code and ``configure``
script in a subdirectory like ``src``. If this happens, Spack won't
be able to automatically detect the build system properly when running
``spack create``. You will have to manually change the package base
class and tell Spack where the ``configure`` script resides. You can
do this like so:
.. code-block:: python
configure_directory = 'src'
^^^^^^^^^^^^^^^^^^^^^^
Building out of source
^^^^^^^^^^^^^^^^^^^^^^
Some packages like ``gcc`` recommend building their software in a
different directory than the source code to prevent build pollution.
This can be done using the ``build_directory`` variable:
.. code-block:: python
build_directory = 'spack-build'
By default, Spack will build the package in the same directory that
contains the ``configure`` script
^^^^^^^^^^^^^^^^^^^^^^^^^
Build and install targets
^^^^^^^^^^^^^^^^^^^^^^^^^
For most Autotools packages, the usual:
.. code-block:: console
$ configure
$ make
$ make install
is sufficient to install the package. However, if you need to run
make with any other targets, for example, to build an optional
library or build the documentation, you can add these like so:
.. code-block:: python
build_targets = ['all', 'docs']
install_targets = ['install', 'docs']
^^^^^^^
Testing
^^^^^^^
Autotools-based packages typically provide unit testing via the
``check`` and ``installcheck`` targets. If you build your software
with ``spack install --test=root``, Spack will check for the presence
of a ``check`` or ``test`` target in the Makefile and run
``make check`` for you. After installation, it will check for an
``installcheck`` target and run ``make installcheck`` if it finds one.
^^^^^^^^^^^^^^^^^^^^^^
External documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on the Autotools build system, see:
https://www.gnu.org/software/automake/manual/html_node/Autotools-Introduction.html

274
lib/spack/docs/build_systems/cmakepackage.rst Normal file
View File

@ -0,0 +1,274 @@
.. _cmakepackage:
------------
CMakePackage
------------
Like Autotools, CMake is a widely-used build-script generator. Designed
by Kitware, CMake is the most popular build system for new C, C++, and
Fortran projects, and many older projects are switching to it as well.
Unlike Autotools, CMake can generate build scripts for builders other
than Make: Ninja, Visual Studio, etc. It is therefore cross-platform,
whereas Autotools is Unix-only.
^^^^^^
Phases
^^^^^^
The ``CMakePackage`` base class comes with the following phases:
#. ``cmake`` - generate the Makefile
#. ``build`` - build the package
#. ``install`` - install the package
By default, these phases run:
.. code-block:: console
$ mkdir spack-build
$ cd spack-build
$ cmake .. -DCMAKE_INSTALL_PREFIX=/path/to/installation/prefix
$ make
$ make test # optional
$ make install
A few more flags are passed to ``cmake`` by default, including flags
for setting the build type and flags for locating dependencies. Of
course, you may need to add a few arguments yourself.
^^^^^^^^^^^^^^^
Important files
^^^^^^^^^^^^^^^
A CMake-based package can be identified by the presence of a
``CMakeLists.txt`` file. This file defines the build flags that can be
passed to the cmake invocation, as well as linking instructions. If
you are familiar with CMake, it can prove very useful for determining
dependencies and dependency version requirements.
One thing to look for is the ``cmake_minimum_required`` function:
.. code-block:: cmake
cmake_minimum_required(VERSION 2.8.12)
This means that CMake 2.8.12 is the earliest release that will work.
You should specify this in a ``depends_on`` statement.
CMake-based packages may also contain ``CMakeLists.txt`` in subdirectories.
This modularization helps to manage complex builds in a hierarchical
fashion. Sometimes these nested ``CMakeLists.txt`` require additional
dependencies not mentioned in the top-level file.
There's also usually a ``cmake`` or ``CMake`` directory containing
additional macros, find scripts, etc. These may prove useful in
determining dependency version requirements.
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
Every package that uses the CMake build system requires a ``cmake``
dependency. Since this is always the case, the ``CMakePackage`` base
class already contains:
.. code-block:: python
depends_on('cmake', type='build')
If you need to specify a particular version requirement, you can
override this in your package:
.. code-block:: python
depends_on('cmake@2.8.12:', type='build')
^^^^^^^^^^^^^^^^^^^
Finding cmake flags
^^^^^^^^^^^^^^^^^^^
To get a list of valid flags that can be passed to ``cmake``, run the
following command in the directory that contains ``CMakeLists.txt``:
.. code-block:: console
$ cmake . -LAH
CMake will start by checking for compilers and dependencies. Eventually
it will begin to list build options. You'll notice that most of the
build options at the top are prefixed with ``CMAKE_``. You can safely
ignore most of these options as Spack already sets them for you. This
includes flags needed to locate dependencies, RPATH libraries, set the
installation directory, and set the build type.
The rest of the flags are the ones you should consider adding to your
package. They often include flags to enable/disable support for certain
features and locate specific dependencies. One thing you'll notice that
makes CMake different from Autotools is that CMake has an understanding
of build flag hierarchy. That is, certain flags will not display unless
their parent flag has been selected. For example, flags to specify the
``lib`` and ``include`` directories for a package might not appear
unless CMake found the dependency it was looking for. You may need to
manually specify certain flags to explore the full depth of supported
build flags, or check the ``CMakeLists.txt`` yourself.
^^^^^^^^^^^^^^^^^^^^^
Adding flags to cmake
^^^^^^^^^^^^^^^^^^^^^
To add additional flags to the ``cmake`` call, simply override the
``cmake_args`` function:
.. code-block:: python
def cmake_args(self):
args = []
if '+hdf5' in self.spec:
args.append('-DDETECT_HDF5=ON')
else:
args.append('-DDETECT_HDF5=OFF')
return args
^^^^^^^^^^
Generators
^^^^^^^^^^
CMake and Autotools are build-script generation tools; they "generate"
the Makefiles that are used to build a software package. CMake actually
supports multiple generators, not just Makefiles. Another common
generator is Ninja. To switch to the Ninja generator, simply add:
.. code-block:: python
generator = 'Ninja'
``CMakePackage`` defaults to "Unix Makefiles". If you switch to the
Ninja generator, make sure to add:
.. code-block:: python
depends_on('ninja', type='build')
to the package as well. Aside from that, you shouldn't need to do
anything else. Spack will automatically detect that you are using
Ninja and run:
.. code-block:: console
$ cmake .. -G Ninja
$ ninja
$ ninja install
Spack currently only supports "Unix Makefiles" and "Ninja" as valid
generators, but it should be simple to add support for alternative
generators. For more information on CMake generators, see:
https://cmake.org/cmake/help/latest/manual/cmake-generators.7.html
^^^^^^^^^^^^^^^^
CMAKE_BUILD_TYPE
^^^^^^^^^^^^^^^^
Every CMake-based package accepts a ``-DCMAKE_BUILD_TYPE`` flag to
dictate which level of optimization to use. In order to ensure
uniformity across packages, the ``CMakePackage`` base class adds
a variant to control this:
.. code-block:: python
variant('build_type', default='RelWithDebInfo',
description='CMake build type',
values=('Debug', 'Release', 'RelWithDebInfo', 'MinSizeRel'))
However, not every CMake package accepts all four of these options.
Grep the ``CMakeLists.txt`` file to see if the default values are
missing or replaced. For example, the
`dealii <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/dealii/package.py>`_
package overrides the default variant with:
.. code-block:: python
variant('build_type', default='DebugRelease',
description='The build type to build',
values=('Debug', 'Release', 'DebugRelease'))
For more information on ``CMAKE_BUILD_TYPE``, see:
https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
CMakeLists.txt in a sub-directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Occasionally, developers will hide their source code and ``CMakeLists.txt``
in a subdirectory like ``src``. If this happens, Spack won't
be able to automatically detect the build system properly when running
``spack create``. You will have to manually change the package base
class and tell Spack where ``CMakeLists.txt`` resides. You can do this
like so:
.. code-block:: python
root_cmakelists_dir = 'src'
Note that this path is relative to the root of the extracted tarball,
not to the ``build_directory``. It defaults to the current directory.
^^^^^^^^^^^^^^^^^^^^^^
Building out of source
^^^^^^^^^^^^^^^^^^^^^^
By default, Spack builds every ``CMakePackage`` in a ``spack-build``
sub-directory. If, for whatever reason, you would like to build in a
different sub-directory, simply override ``build_directory`` like so:
.. code-block:: python
build_directory = 'my-build'
^^^^^^^^^^^^^^^^^^^^^^^^^
Build and install targets
^^^^^^^^^^^^^^^^^^^^^^^^^
For most CMake packages, the usual:
.. code-block:: console
$ cmake
$ make
$ make install
is sufficient to install the package. However, if you need to run
make with any other targets, for example, to build an optional
library or build the documentation, you can add these like so:
.. code-block:: python
build_targets = ['all', 'docs']
install_targets = ['install', 'docs']
^^^^^^^
Testing
^^^^^^^
CMake-based packages typically provide unit testing via the
``test`` target. If you build your software with ``--test=root``,
Spack will check for the presence of a ``test`` target in the
Makefile and run ``make test`` for you. If you want to run a
different test instead, simply override the ``check`` method.
^^^^^^^^^^^^^^^^^^^^^^
External documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on the CMake build system, see:
https://cmake.org/cmake/help/latest/

38
lib/spack/docs/build_systems/cudapackage.rst Normal file
View File

@ -0,0 +1,38 @@
.. _cudapackage:
-----------
CudaPackage
-----------
Different from other packages, ``CudaPackage`` does not represent a build
system. Instead its goal is to simplify and unify usage of ``CUDA`` in other
packages.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Provided variants and dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``CudaPackage`` provides ``cuda`` variant (default to ``off``) to enable/disable
``CUDA``, and ``cuda_arch`` variant to optionally specify the architecture.
It also declares dependencies on the ``CUDA`` package ``depends_on('cuda@...')``
based on the architecture as well as specifies conflicts for certain compiler versions.
^^^^^
Usage
^^^^^
In order to use it, just add another base class to your package, for example:
.. code-block:: python
class MyPackage(CMakePackage, CudaPackage):
...
def cmake_args(self):
spec = self.spec
if '+cuda' in spec:
options.append('-DWITH_CUDA=ON')
cuda_arch = spec.variants['cuda_arch'].value
if cuda_arch is not None:
options.append('-DCUDA_FLAGS=-arch=sm_{0}'.format(cuda_arch[0]))
else:
options.append('-DWITH_CUDA=OFF')

204
lib/spack/docs/build_systems/custompackage.rst Normal file
View File

@ -0,0 +1,204 @@
.. _custompackage:
--------------------
Custom Build Systems
--------------------
While the build systems listed above should meet your needs for the
vast majority of packages, some packages provide custom build scripts.
This guide is intended for the following use cases:
* Packaging software with its own custom build system
* Adding support for new build systems
If you want to add support for a new build system, a good place to
start is to look at the definitions of other build systems. This guide
focuses mostly on how Spack's build systems work.
In this guide, we will be using the
`perl <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/perl/package.py>`_ and
`cmake <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cmake/package.py>`_
packages as examples. ``perl``'s build system is a hand-written
``Configure`` shell script, while ``cmake`` bootstraps itself during
installation. Both of these packages require custom build systems.
^^^^^^^^^^
Base class
^^^^^^^^^^
If your package does not belong to any of the aforementioned build
systems that Spack already supports, you should inherit from the
``Package`` base class. ``Package`` is a simple base class with a
single phase: ``install``. If your package is simple, you may be able
to simply write an ``install`` method that gets the job done. However,
if your package is more complex and installation involves multiple
steps, you should add separate phases as mentioned in the next section.
If you are creating a new build system base class, you should inherit
from ``PackageBase``. This is the superclass for all build systems in
Spack.
^^^^^^
Phases
^^^^^^
The most important concept in Spack's build system support is the idea
of phases. Each build system defines a set of phases that are necessary
to install the package. They usually follow some sort of "configure",
"build", "install" guideline, but any of those phases may be missing
or combined with another phase.
If you look at the ``perl`` package, you'll see:
.. code-block:: python
phases = ['configure', 'build', 'install']
Similarly, ``cmake`` defines:
.. code-block:: python
phases = ['bootstrap', 'build', 'install']
If we look at the ``cmake`` example, this tells Spack's ``PackageBase``
class to run the ``bootstrap``, ``build``, and ``install`` functions
in that order. It is now up to you to define these methods.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Phase and phase_args functions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If we look at ``perl``, we see that it defines a ``configure`` method:
.. code-block:: python
def configure(self, spec, prefix):
configure = Executable('./Configure')
configure(*self.configure_args())
There is also a corresponding ``configure_args`` function that handles
all of the arguments to pass to ``Configure``, just like in
``AutotoolsPackage``. Comparatively, the ``build`` and ``install``
phases are pretty simple:
.. code-block:: python
def build(self, spec, prefix):
make()
def install(self, spec, prefix):
make('install')
The ``cmake`` package looks very similar, but with a ``bootstrap``
function instead of ``configure``:
.. code-block:: python
def bootstrap(self, spec, prefix):
bootstrap = Executable('./bootstrap')
bootstrap(*self.bootstrap_args())
def build(self, spec, prefix):
make()
def install(self, spec, prefix):
make('install')
Again, there is a ``boostrap_args`` function that determines the
correct bootstrap flags to use.
^^^^^^^^^^^^^^^^^^^^
run_before/run_after
^^^^^^^^^^^^^^^^^^^^
Occasionally, you may want to run extra steps either before or after
a given phase. This applies not just to custom build systems, but to
existing build systems as well. You may need to patch a file that is
generated by configure, or install extra files in addition to what
``make install`` copies to the installation prefix. This is where
``@run_before`` and ``@run_after`` come in.
These Python decorators allow you to write functions that are called
before or after a particular phase. For example, in ``perl``, we see:
.. code-block:: python
@run_after('install')
def install_cpanm(self):
spec = self.spec
if '+cpanm' in spec:
with working_dir(join_path('cpanm', 'cpanm')):
perl = spec['perl'].command
perl('Makefile.PL')
make()
make('install')
This extra step automatically installs ``cpanm`` in addition to the
base Perl installation.
^^^^^^^^^^^^^^^^^^^^^
on_package_attributes
^^^^^^^^^^^^^^^^^^^^^
The ``run_before``/``run_after`` logic discussed above becomes
particularly powerful when combined with the ``@on_package_attributes``
decorator. This decorator allows you to conditionally run certain
functions depending on the attributes of that package. The most
common example is conditional testing. Many unit tests are prone to
failure, even when there is nothing wrong with the installation.
Unfortunately, non-portable unit tests and tests that are
"supposed to fail" are more common than we would like. Instead of
always running unit tests on installation, Spack lets users
conditionally run tests with the ``--test=root`` flag.
If we wanted to define a function that would conditionally run
if and only if this flag is set, we would use the following line:
.. code-block:: python
@on_package_attributes(run_tests=True)
^^^^^^^
Testing
^^^^^^^
Let's put everything together and add unit tests to our package.
In the ``perl`` package, we can see:
.. code-block:: python
@run_after('build')
@on_package_attributes(run_tests=True)
def test(self):
make('test')
As you can guess, this runs ``make test`` *after* building the package,
if and only if testing is requested. Again, this is not specific to
custom build systems, it can be added to existing build systems as well.
Ideally, every package in Spack will have some sort of test to ensure
that it was built correctly. It is up to the package authors to make
sure this happens. If you are adding a package for some software and
the developers list commands to test the installation, please add these
tests to your ``package.py``.
.. warning::
The order of decorators matters. The following ordering:
.. code-block:: python
@run_after('install')
@on_package_attributes(run_tests=True)
works as expected. However, if you reverse the ordering:
.. code-block:: python
@on_package_attributes(run_tests=True)
@run_after('install')
the tests will always be run regardless of whether or not
``--test=root`` is requested. See https://github.com/spack/spack/issues/3833
for more information

13
lib/spack/docs/build_systems/intelpackage.rst Normal file
View File

@ -0,0 +1,13 @@
.. _intelpackage:
------------
IntelPackage
------------
Intel provides many licensed software packages, which all share the
same basic steps for configuring and installing, as well as license
management.
This build system is a work-in-progress. See
https://github.com/spack/spack/pull/4300 and
https://github.com/spack/spack/pull/7469 for more information.

304
lib/spack/docs/build_systems/makefilepackage.rst Normal file
View File

@ -0,0 +1,304 @@
.. _makefilepackage:
---------------
MakefilePackage
---------------
The most primitive build system a package can use is a plain Makefile.
Makefiles are simple to write for small projects, but they usually
require you to edit the Makefile to set platform and compiler-specific
variables.
^^^^^^
Phases
^^^^^^
The ``MakefilePackage`` base class comes with 3 phases:
#. ``edit`` - edit the Makefile
#. ``build`` - build the project
#. ``install`` - install the project
By default, ``edit`` does nothing, but you can override it to replace
hard-coded Makefile variables. The ``build`` and ``install`` phases
run:
.. code-block:: console
$ make
$ make install
^^^^^^^^^^^^^^^
Important files
^^^^^^^^^^^^^^^
The main file that matters for a ``MakefilePackage`` is the Makefile.
This file will be named one of the following ways:
* GNUmakefile (only works with GNU Make)
* Makefile (most common)
* makefile
Some Makefiles also *include* other configuration files. Check for an
``include`` directive in the Makefile.
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
Spack assumes that the operating system will have a valid ``make`` utility
installed already, so you don't need to add a dependency on ``make``.
However, if the package uses a ``GNUmakefile`` or the developers recommend
using GNU Make, you should add a dependency on ``gmake``:
.. code-block:: python
depends_on('gmake', type='build')
^^^^^^^^^^^^^^^^^^^^^^^^^^
Types of Makefile packages
^^^^^^^^^^^^^^^^^^^^^^^^^^
Most of the work involved in packaging software that uses Makefiles
involves overriding or replacing hard-coded variables. Many packages
make the mistake of hard-coding compilers, usually for GCC or Intel.
This is fine if you happen to be using that particular compiler, but
Spack is designed to work with *any* compiler, and you need to ensure
that this is the case.
Depending on how the Makefile is designed, there are 4 common strategies
that can be used to set or override the appropriate variables:
"""""""""""""""""""""
Environment variables
"""""""""""""""""""""
Make has multiple types of
`assignment operators <https://www.gnu.org/software/make/manual/make.html#Setting>`_.
Some Makefiles use ``=`` to assign variables. The only way to override
these variables is to edit the Makefile or override them on the
command-line. However, Makefiles that use ``?=`` for assignment honor
environment variables. Since Spack already sets ``CC``, ``CXX``, ``F77``,
and ``FC``, you won't need to worry about setting these variables. If
there are any other variables you need to set, you can do this in the
``edit`` method:
.. code-block:: python
def edit(self, spec, prefix):
env['PREFIX'] = prefix
env['BLASLIB'] = spec['blas'].libs.ld_flags
`cbench <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cbench/package.py>`_
is a good example of a simple package that does this, while
`esmf <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/esmf/package.py>`_
is a good example of a more complex package.
""""""""""""""""""""""
Command-line arguments
""""""""""""""""""""""
If the Makefile ignores environment variables, the next thing to try
is command-line arguments. You can do this by overriding the
``build_targets`` attribute. If you don't need access to the spec,
you can do this like so:
.. code-block:: python
build_targets = ['CC=cc']
If you do need access to the spec, you can create a property like so:
.. code-block:: python
@property
def build_targets(self):
spec = self.spec
return [
'CC=cc',
'BLASLIB={0}'.format(spec['blas'].libs.ld_flags),
]
`cloverleaf <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cloverleaf/package.py>`_
is a good example of a package that uses this strategy.
"""""""""""""
Edit Makefile
"""""""""""""
Some Makefiles are just plain stubborn and will ignore command-line
variables. The only way to ensure that these packages build correctly
is to directly edit the Makefile. Spack provides a ``FileFilter`` class
and a ``filter_file`` method to help with this. For example:
.. code-block:: python
def edit(self, spec, prefix):
makefile = FileFilter('Makefile')
makefile.filter('CC = gcc', 'CC = cc')
makefile.filter('CXX = g++', 'CC = c++')
`stream <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/stream/package.py>`_
is a good example of a package that involves editing a Makefile to set
the appropriate variables.
"""""""""""
Config file
"""""""""""
More complex packages often involve Makefiles that *include* a
configuration file. These configuration files are primarily composed
of variables relating to the compiler, platform, and the location of
dependencies or names of libraries. Since these config files are
dependent on the compiler and platform, you will often see entire
directories of examples for common compilers and architectures. Use
these examples to help determine what possible values to use.
If the config file is long and only contains one or two variables
that need to be modified, you can use the technique above to edit
the config file. However, if you end up needing to modify most of
the variables, it may be easier to write a new file from scratch.
If each variable is independent of each other, a dictionary works
well for storing variables:
.. code-block:: python
def edit(self, spec, prefix):
config = {
'CC': 'cc',
'MAKE': 'make',
}
if '+blas' in spec:
config['BLAS_LIBS'] = spec['blas'].libs.joined()
with open('make.inc', 'w') as inc:
for key in config:
inc.write('{0} = {1}\n'.format(key, config[key]))
`elk <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/elk/package.py>`_
is a good example of a package that uses a dictionary to store
configuration variables.
If the order of variables is important, it may be easier to store
them in a list:
.. code-block:: python
def edit(self, spec, prefix):
config = [
'INSTALL_DIR = {0}'.format(prefix),
'INCLUDE_DIR = $(INSTALL_DIR)/include',
'LIBRARY_DIR = $(INSTALL_DIR)/lib',
]
with open('make.inc', 'w') as inc:
for var in config:
inc.write('{0}\n'.format(var))
`hpl <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/hpl/package.py>`_
is a good example of a package that uses a list to store
configuration variables.
^^^^^^^^^^^^^^^^^^^^^^^^^^
Variables to watch out for
^^^^^^^^^^^^^^^^^^^^^^^^^^
The following is a list of common variables to watch out for. The first
two sections are
`implicit variables <https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html>`_
defined by Make and will always use the same name, while the rest are
user-defined variables and may vary from package to package.
* **Compilers**
This includes variables such as ``CC``, ``CXX``, ``F77``, ``F90``,
and ``FC``, as well as variables related to MPI compiler wrappers,
like ``MPICC`` and friends.
* **Compiler flags**
This includes variables for specific compilers, like ``CFLAGS``,
``CXXFLAGS``, ``F77FLAGS``, ``F90FLAGS``, ``FCFLAGS``, and ``CPPFLAGS``.
These variables are often hard-coded to contain flags specific to a
certain compiler. If these flags don't work for every compiler,
you may want to consider filtering them.
* **Variables that enable or disable features**
This includes variables like ``MPI``, ``OPENMP``, ``PIC``, and
``DEBUG``. These flags often require you to create a variant
so that you can either build with or without MPI support, for
example. These flags are often compiler-dependent. You should
replace them with the appropriate compiler flags, such as
``self.compiler.openmp_flag`` or ``self.compiler.pic_flag``.
* **Platform flags**
These flags control the type of architecture that the executable
is compiler for. Watch out for variables like ``PLAT`` or ``ARCH``.
* **Dependencies**
Look out for variables that sound like they could be used to
locate dependencies, such as ``JAVA_HOME``, ``JPEG_ROOT``, or
``ZLIBDIR``. Also watch out for variables that control linking,
such as ``LIBS``, ``LDFLAGS``, and ``INCLUDES``. These variables
need to be set to the installation prefix of a dependency, or
to the correct linker flags to link to that dependency.
* **Installation prefix**
If your Makefile has an ``install`` target, it needs some way of
knowing where to install. By default, many packages install to
``/usr`` or ``/usr/local``. Since many Spack users won't have
sudo privileges, it is imperative that each package is installed
to the proper prefix. Look for variables like ``PREFIX`` or
``INSTALL``.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Makefiles in a sub-directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Not every package places their Makefile in the root of the package
tarball. If the Makefile is in a sub-directory like ``src``, you
can tell Spack where to locate it like so:
.. code-block:: python
build_directory = 'src'
^^^^^^^^^^^^^^^^^^^
Manual installation
^^^^^^^^^^^^^^^^^^^
Not every Makefile includes an ``install`` target. If this is the
case, you can override the default ``install`` method to manually
install the package:
.. code-block:: python
def install(self, spec, prefix):
mkdir(prefix.bin)
install('foo', prefix.bin)
install_tree('lib', prefix.lib)
^^^^^^^^^^^^^^^^^^^^^^
External documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on reading and writing Makefiles, see:
https://www.gnu.org/software/make/manual/make.html

47
lib/spack/docs/build_systems/octavepackage.rst Normal file
View File

@ -0,0 +1,47 @@
.. _octavepackage:
-------------
OctavePackage
-------------
Octave has its own build system for installing packages.
^^^^^^
Phases
^^^^^^
The ``OctavePackage`` base class has a single phase:
#. ``install`` - install the package
By default, this phase runs the following command:
.. code-block:: console
$ octave '--eval' 'pkg prefix <prefix>; pkg install <archive_file>'
Beware that uninstallation is not implemented at the moment. After uninstalling
a package via Spack, you also need to manually uninstall it from Octave via
``pkg uninstall <package_name>``.
^^^^^^^^^^^^^^^^^^^^^^^
Finding Octave packages
^^^^^^^^^^^^^^^^^^^^^^^
Most Octave packages are listed at https://octave.sourceforge.io/packages.php.
^^^^^^^^^^^^
Dependencies
^^^^^^^^^^^^
Usually, the homepage of a package will list dependencies, i.e.
``Dependencies: Octave >= 3.6.0 struct >= 1.0.12``. The same information should
be available in the ``DESCRIPTION`` file in the root of each archive.
^^^^^^^^^^^^^^^^^^^^^^
External Documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on the Octave build system, see:
https://octave.org/doc/v4.4.0/Installing-and-Removing-Packages.html

207
lib/spack/docs/build_systems/perlpackage.rst Normal file
View File

@ -0,0 +1,207 @@
.. _perlpackage:
-----------
PerlPackage
-----------
Much like Octave, Perl has its own language-specific
build system.
^^^^^^
Phases
^^^^^^
The ``PerlPackage`` base class comes with 3 phases that can be overridden:
#. ``configure`` - configure the package
#. ``build`` - build the package
#. ``install`` - install the package
Perl packages have 2 common modules used for module installation:
"""""""""""""""""""""""
``ExtUtils::MakeMaker``
"""""""""""""""""""""""
The ``ExtUtils::MakeMaker`` module is just what it sounds like, a module
designed to generate Makefiles. It can be identified by the presence of
a ``Makefile.PL`` file, and has the following installation steps:
.. code-block:: console
$ perl Makefile.PL INSTALL_BASE=/path/to/installation/prefix
$ make
$ make test # optional
$ make install
"""""""""""""""""
``Module::Build``
"""""""""""""""""
The ``Module::Build`` module is a pure-Perl build system, and can be
identified by the presence of a ``Build.PL`` file. It has the following
installation steps:
.. code-block:: console
$ perl Build.PL --install_base /path/to/installation/prefix
$ ./Build
$ ./Build test # optional
$ ./Build install
If both ``Makefile.PL`` and ``Build.PL`` files exist in the package,
Spack will use ``Makefile.PL`` by default. If your package uses a
different module, ``PerlPackage`` will need to be extended to support
it.
``PerlPackage`` automatically detects which build steps to use, so there
shouldn't be much work on the package developer's side to get things
working.
^^^^^^^^^^^^^^^^^^^^^
Finding Perl packages
^^^^^^^^^^^^^^^^^^^^^
Most Perl modules are hosted on CPAN - The Comprehensive Perl Archive
Network. If you need to find a package for ``XML::Parser``, for example,
you should search for "CPAN XML::Parser".
Some CPAN pages are versioned. Check for a link to the
"Latest Release" to make sure you have the latest version.
^^^^^^^^^^^^
Package name
^^^^^^^^^^^^
When you use ``spack create`` to create a new Perl package, Spack will
automatically prepend ``perl-`` to the front of the package name. This
helps to keep Perl modules separate from other packages. The same
naming scheme is used for other language extensions, like Python and R.
^^^^^^^^^^^
Description
^^^^^^^^^^^
Most CPAN pages have a short description under "NAME" and a longer
description under "DESCRIPTION". Use whichever you think is more
useful while still being succinct.
^^^^^^^^
Homepage
^^^^^^^^
In the top-right corner of the CPAN page, you'll find a "permalink"
for the package. This should be used instead of the current URL, as
it doesn't contain the version number and will always link to the
latest release.
^^^
URL
^^^
If you haven't found it already, the download URL is on the right
side of the page below the permalink. Search for "Download".
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
Every ``PerlPackage`` obviously depends on Perl at build and run-time,
so ``PerlPackage`` contains:
.. code-block:: python
extends('perl')
depends_on('perl', type=('build', 'run'))
If your package requires a specific version of Perl, you should
specify this.
Although newer versions of Perl include ``ExtUtils::MakeMaker`` and
``Module::Build`` as "core" modules, you may want to add dependencies
on ``perl-extutils-makemaker`` and ``perl-module-build`` anyway. Many
people add Perl as an external package, and we want the build to work
properly. If your package uses ``Makefile.PL`` to build, add:
.. code-block:: python
depends_on('perl-extutils-makemaker', type='build')
If your package uses ``Build.PL`` to build, add:
.. code-block:: python
depends_on('perl-module-build', type='build')
^^^^^^^^^^^^^^^^^
Perl dependencies
^^^^^^^^^^^^^^^^^
Below the download URL, you will find a "Dependencies" link, which
takes you to a page listing all of the dependencies of the package.
Packages listed as "Core module" don't need to be added as dependencies,
but all direct dependencies should be added. Don't add dependencies of
dependencies. These should be added as dependencies to the dependency,
not to your package.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Passing arguments to configure
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Packages that have non-Perl dependencies often use command-line
variables to specify their installation directory. You can pass
arguments to ``Makefile.PL`` or ``Build.PL`` by overriding
``configure_args`` like so:
.. code-block:: python
def configure_args(self):
expat = self.spec['expat'].prefix
return [
'EXPATLIBPATH={0}'.format(expat.lib),
'EXPATINCPATH={0}'.format(expat.include),
]
^^^^^^^^^^^^^^^^^^^^^
Alternatives to Spack
^^^^^^^^^^^^^^^^^^^^^
If you need to maintain a stack of Perl modules for a user and don't
want to add all of them to Spack, a good alternative is ``cpanm``.
If Perl is already installed on your system, it should come with a
``cpan`` executable. To install ``cpanm``, run the following command:
.. code-block:: console
$ cpan App::cpanminus
Now, you can install any Perl module you want by running:
.. code-block:: console
$ cpanm Module::Name
Obviously, these commands can only be run if you have root privileges.
Furthermore, ``cpanm`` is not capable of installing non-Perl dependencies.
If you need to install to your home directory or need to install a module
with non-Perl dependencies, Spack is a better option.
^^^^^^^^^^^^^^^^^^^^^^
External documentation
^^^^^^^^^^^^^^^^^^^^^^
You can find more information on installing Perl modules from source
at: http://www.perlmonks.org/?node_id=128077
More generic Perl module installation instructions can be found at:
http://www.cpan.org/modules/INSTALL.html

742
lib/spack/docs/build_systems/pythonpackage.rst Normal file
View File

@ -0,0 +1,742 @@
.. _pythonpackage:
-------------
PythonPackage
-------------
Python packages and modules have their own special build system.
^^^^^^
Phases
^^^^^^
The ``PythonPackage`` base class provides the following phases that
can be overridden:
* ``build``
* ``build_py``
* ``build_ext``
* ``build_clib``
* ``build_scripts``
* ``clean``
* ``install``
* ``install_lib``
* ``install_headers``
* ``install_scripts``
* ``install_data``
* ``sdist``
* ``register``
* ``bdist``
* ``bdist_dumb``
* ``bdist_rpm``
* ``bdist_wininst``
* ``upload``
* ``check``
These are all standard ``setup.py`` commands and can be found by running:
.. code-block:: console
$ python setup.py --help-commands
By default, only the ``build`` and ``install`` phases are run:
#. ``build`` - build everything needed to install
#. ``install`` - install everything from build directory
If for whatever reason you need to run more phases, simply modify your
``phases`` list like so:
.. code-block:: python
phases = ['build_ext', 'install', 'bdist']
Each phase provides a function ``<phase>`` that runs:
.. code-block:: console
$ python -s setup.py --no-user-cfg <phase>
Each phase also has a ``<phase_args>`` function that can pass arguments to
this call. All of these functions are empty except for the ``install_args``
function, which passes ``--prefix=/path/to/installation/prefix``. There is
also some additional logic specific to setuptools and eggs.
If you need to run a phase that is not a standard ``setup.py`` command,
you'll need to define a function for it like so:
.. code-block:: python
phases = ['configure', 'build', 'install']
def configure(self, spec, prefix):
self.setup_py('configure')
^^^^^^^^^^^^^^^
Important files
^^^^^^^^^^^^^^^
Python packages can be identified by the presence of a ``setup.py`` file.
This file is used by package managers like ``pip`` to determine a
package's dependencies and the version of dependencies required, so if
the ``setup.py`` file is not accurate, the package will not build properly.
For this reason, the ``setup.py`` file should be fairly reliable. If the
documentation and ``setup.py`` disagree on something, the ``setup.py``
file should be considered to be the truth. As dependencies are added or
removed, the documentation is much more likely to become outdated than
the ``setup.py``.
^^^^^^^^^^^^^^^^^^^^^^^
Finding Python packages
^^^^^^^^^^^^^^^^^^^^^^^
The vast majority of Python packages are hosted on PyPI - The Python
Package Index. ``pip`` only supports packages hosted on PyPI, making
it the only option for developers who want a simple installation.
Search for "PyPI <package-name>" to find the download page. Note that
some pages are versioned, and the first result may not be the newest
version. Click on the "Latest Version" button to the top right to see
if a newer version is available. The download page is usually at:
https://pypi.org/project/<package-name>
^^^^^^^^^^^
Description
^^^^^^^^^^^
The top of the PyPI downloads page contains a description of the
package. The first line is usually a short description, while there
may be a several line "Project Description" that follows. Choose whichever
is more useful. You can also get these descriptions on the command-line
using:
.. code-block:: console
$ python setup.py --description
$ python setup.py --long-description
^^^^^^^^
Homepage
^^^^^^^^
Package developers use ``setup.py`` to upload new versions to PyPI.
The ``setup`` method often passes metadata like ``homepage`` to PyPI.
This metadata is displayed on the left side of the download page.
Search for the text "Homepage" under "Project links" to find it. You
should use this page instead of the PyPI page if they differ. You can
also get the homepage on the command-line by running:
.. code-block:: console
$ python setup.py --url
^^^
URL
^^^
You may have noticed that Spack allows you to add multiple versions of
the same package without adding multiple versions of the download URL.
It does this by guessing what the version string in the URL is and
replacing this with the requested version. Obviously, if Spack cannot
guess the version correctly, or if non-version-related things change
in the URL, Spack cannot substitute the version properly.
Once upon a time, PyPI offered nice, simple download URLs like:
https://pypi.python.org/packages/source/n/numpy/numpy-1.13.1.zip
As you can see, the version is 1.13.1. It probably isn't hard to guess
what URL to use to download version 1.12.0, and Spack was perfectly
capable of performing this calculation.
However, PyPI switched to a new download URL format:
https://pypi.python.org/packages/c0/3a/40967d9f5675fbb097ffec170f59c2ba19fc96373e73ad47c2cae9a30aed/numpy-1.13.1.zip#md5=2c3c0f4edf720c3a7b525dacc825b9ae
and more recently:
https://files.pythonhosted.org/packages/b0/2b/497c2bb7c660b2606d4a96e2035e92554429e139c6c71cdff67af66b58d2/numpy-1.14.3.zip
As you can imagine, it is impossible for Spack to guess what URL to
use to download version 1.12.0 given this URL. There is a solution,
however. PyPI offers a new hidden interface for downloading
Python packages that does not include a hash in the URL:
https://pypi.io/packages/source/n/numpy/numpy-1.13.1.zip
This URL redirects to the files.pythonhosted.org URL. The general syntax for
this pypi.io URL is:
https://pypi.io/packages/source/<first-letter-of-name>/<name>/<name>-<version>.<extension>
Please use the pypi.io URL instead of the pypi.python.org URL. If both
``.tar.gz`` and ``.zip`` versions are available, ``.tar.gz`` is preferred.
If some releases offer both ``.tar.gz`` and ``.zip`` versions, but some
only offer ``.zip`` versions, use ``.zip``.
"""""""""""""""
PyPI vs. GitHub
"""""""""""""""
Many packages are hosted on PyPI, but are developed on GitHub and other
version control systems. The tarball can be downloaded from either
location, but PyPI is preferred for the following reasons:
#. PyPI contains the bare minimum of files to install the package.
You may notice that the tarball you download from PyPI does not
have the same checksum as the tarball you download from GitHub.
When a developer uploads a new release to PyPI, it doesn't contain
every file in the repository, only the files necessary to install
the package. PyPI tarballs are therefore smaller.
#. PyPI is the official source for package managers like ``pip``.
Let's be honest, ``pip`` is much more popular than Spack. If the
GitHub tarball contains a file not present in the PyPI tarball that
causes a bug, the developers may not realize this for quite some
time. If the bug was in a file contained in the PyPI tarball, users
would notice the bug much more quickly.
#. GitHub release may be a beta version.
When a developer releases a new version of a package on GitHub,
it may not be intended for most users. Until that release also
makes its way to PyPI, it should be assumed that the release is
not yet ready for general use.
#. The checksum for a GitHub release may change.
Unfortunately, some developers have a habit of patching releases
without incrementing the version number. This results in a change
in tarball checksum. Package managers like Spack that use checksums
to verify the integrity of a download tarball grind to a halt when
the checksum for a known version changes. Most of the time, the
change is intentional, and contains a needed bug fix. However,
sometimes the change indicates a download source that has been
compromised, and a tarball that contains a virus. If this happens,
you must contact the developers to determine which is the case.
PyPI is nice because it makes it physically impossible to
re-release the same version of a package with a different checksum.
There are some reasons to prefer downloading from GitHub:
#. The GitHub tarball may contain unit tests
As previously mentioned, the PyPI tarball contains the bare minimum
of files to install the package. Unless explicitly specified by the
developers, it will not contain development files like unit tests.
If you desire to run the unit tests during installation, you should
use the GitHub tarball instead.
#. Spack does not yet support ``spack versions`` and ``spack checksum``
with PyPI URLs
These commands work just fine with GitHub URLs. This is a minor
annoyance, not a reason to prefer GitHub over PyPI.
If you really want to run these unit tests, no one will stop you from
submitting a PR for a new package that downloads from GitHub.
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
There are a few dependencies common to the ``PythonPackage`` build system.
""""""
Python
""""""
Obviously, every ``PythonPackage`` needs Python at build-time to run
``python setup.py build && python setup.py install``. Python is also
needed at run-time if you want to import the module. Due to backwards
incompatible changes between Python 2 and 3, it is very important to
specify which versions of Python are supported. If the documentation
mentions that Python 3 is required, this can be specified as:
.. code-block:: python
depends_on('python@3:', type=('build', 'run')
If Python 2 is required, this would look like:
.. code-block:: python
depends_on('python@:2', type=('build', 'run')
If Python 2.7 is the only version that works, you can use:
.. code-block:: python
depends_on('python@2.7:2.8', type=('build', 'run')
The documentation may not always specify supported Python versions.
Another place to check is in the ``setup.py`` file. Look for a line
containing ``python_requires``. An example from
`py-numpy <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-numpy/package.py>`_
looks like:
.. code-block:: python
python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*'
More commonly, you will find a version check at the top of the file:
.. code-block:: python
if sys.version_info[:2] < (2, 7) or (3, 0) <= sys.version_info[:2] < (3, 4):
raise RuntimeError("Python version 2.7 or >= 3.4 required.")
This can be converted to Spack's spec notation like so:
.. code-block:: python
depends_on('python@2.7:2.8,3.4:', type=('build', 'run'))
""""""""""
setuptools
""""""""""
Originally, the Python language had a single build system called
distutils, which is built into Python. Distutils provided a common
framework for package authors to describe their project and how it
should be built. However, distutils was not without limitations.
Most notably, there was no way to list a project's dependencies
with distutils. Along came setuptools, a non-builtin build system
designed to overcome the limitations of distutils. Both projects
use a similar API, making the transition easy while adding much
needed functionality. Today, setuptools is used in around 75% of
the Python packages in Spack.
Since setuptools isn't built-in to Python, you need to add it as a
dependency. To determine whether or not a package uses setuptools,
search the file for an import statement like:
.. code-block:: python
import setuptools
or:
.. code-block:: python
from setuptools import setup
Some packages are designed to work with both setuptools and distutils,
so you may find something like:
.. code-block:: python
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
This uses setuptools if available, and falls back to distutils if not.
In this case, you would still want to add a setuptools dependency, as
it offers us more control over the installation.
Unless specified otherwise, setuptools is usually a build-only dependency.
That is, it is needed to install the software, but is not needed at
run-time. This can be specified as:
.. code-block:: python
depends_on('py-setuptools', type='build')
""""""
cython
""""""
Compared to compiled languages, interpreted languages like Python can
be quite a bit slower. To work around this, some Python developers
rewrite computationally demanding sections of code in C, a process
referred to as "cythonizing". In order to build these package, you
need to add a build dependency on cython:
.. code-block:: python
depends_on('py-cython', type='build')
Look for references to "cython" in the ``setup.py`` to determine
whether or not this is necessary. Cython may be optional, but
even then you should list it as a required dependency. Spack is
designed to compile software, and is meant for HPC facilities
where speed is crucial. There is no reason why someone would not
want an optimized version of a library instead of the pure-Python
version.
^^^^^^^^^^^^^^^^^^^
Python dependencies
^^^^^^^^^^^^^^^^^^^
When you install a package with ``pip``, it reads the ``setup.py``
file in order to determine the dependencies of the package.
If the dependencies are not yet installed, ``pip`` downloads them
and installs them for you. This may sound convenient, but Spack
cannot rely on this behavior for two reasons:
#. Spack needs to be able to install packages on air-gapped networks.
If there is no internet connection, ``pip`` can't download the
package dependencies. By explicitly listing every dependency in
the ``package.py``, Spack knows what to download ahead of time.
#. Duplicate installations of the same dependency may occur.
Spack supports *activation* of Python extensions, which involves
symlinking the package installation prefix to the Python installation
prefix. If your package is missing a dependency, that dependency
will be installed to the installation directory of the same package.
If you try to activate the package + dependency, it may cause a
problem if that package has already been activated.
For these reasons, you must always explicitly list all dependencies.
Although the documentation may list the package's dependencies,
often the developers assume people will use ``pip`` and won't have to
worry about it. Always check the ``setup.py`` to find the true
dependencies.
If the package relies on ``distutils``, it may not explicitly list its
dependencies. Check for statements like:
.. code-block:: python
try:
import numpy
except ImportError:
raise ImportError("numpy must be installed prior to installation")
Obviously, this means that ``py-numpy`` is a dependency.
If the package uses ``setuptools``, check for the following clues:
* ``install_requires``
These packages are required for installation.
* ``extra_requires``
These packages are optional dependencies that enable additional
functionality. You should add a variant that optionally adds these
dependencies.
* ``test_requires``
These are packages that are required to run the unit tests for the
package. These dependencies can be specified using the
``type='test'`` dependency type.
In the root directory of the package, you may notice a
``requirements.txt`` file. It may look like this file contains a list
of all of the package's dependencies. Don't be fooled. This file is
used by tools like Travis to install the pre-requisites for the
package... and a whole bunch of other things. It often contains
dependencies only needed for unit tests, like:
* mock
* nose
* pytest
It can also contain dependencies for building the documentation, like
sphinx. If you can't find any information about the package's
dependencies, you can take a look in ``requirements.txt``, but be sure
not to add test or documentation dependencies.
""""""""""
setuptools
""""""""""
Setuptools is a bit of a special case. If a package requires setuptools
at run-time, how do they express this? They could add it to
``install_requires``, but setuptools is imported long before this and
needed to read this line. And since you can't install the package
without setuptools, the developers assume that setuptools will already
be there, so they never mention when it is required. We don't want to
add run-time dependencies if they aren't needed, so you need to
determine whether or not setuptools is needed. Grep the installation
directory for any files containing a reference to ``setuptools`` or
``pkg_resources``. Both modules come from ``py-setuptools``.
``pkg_resources`` is particularly common in scripts in ``prefix/bin``.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Passing arguments to setup.py
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The default build and install phases should be sufficient to install
most packages. However, you may want to pass additional flags to
either phase.
You can view the available options for a particular phase with:
.. code-block:: console
$ python setup.py <phase> --help
Each phase provides a ``<phase_args>`` function that can be used to
pass arguments to that phase. For example,
`py-numpy <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-numpy/package.py>`_
adds:
.. code-block:: python
def build_args(self, spec, prefix):
args = []
# From NumPy 1.10.0 on it's possible to do a parallel build.
if self.version >= Version('1.10.0'):
# But Parallel build in Python 3.5+ is broken. See:
# https://github.com/spack/spack/issues/7927
# https://github.com/scipy/scipy/issues/7112
if spec['python'].version < Version('3.5'):
args = ['-j', str(make_jobs)]
return args
^^^^^^^
Testing
^^^^^^^
``PythonPackage`` provides a couple of options for testing packages.
""""""""""""
Import tests
""""""""""""
Just because a package successfully built does not mean that it built
correctly. The most reliable test of whether or not the package was
correctly installed is to attempt to import all of the modules that
get installed. To get a list of modules, run the following command
in the source directory:
.. code-block:: console
$ python
>>> import setuptools
>>> setuptools.find_packages()
['numpy', 'numpy._build_utils', 'numpy.compat', 'numpy.core', 'numpy.distutils', 'numpy.doc', 'numpy.f2py', 'numpy.fft', 'numpy.lib', 'numpy.linalg', 'numpy.ma', 'numpy.matrixlib', 'numpy.polynomial', 'numpy.random', 'numpy.testing', 'numpy.core.code_generators', 'numpy.distutils.command', 'numpy.distutils.fcompiler']
Large, complex packages like ``numpy`` will return a long list of
packages, while other packages like ``six`` will return an empty list.
``py-six`` installs a single ``six.py`` file. In Python packaging lingo,
a "package" is a directory containing files like:
.. code-block:: none
foo/__init__.py
foo/bar.py
foo/baz.py
whereas a "module" is a single Python file. Since ``find_packages``
only returns packages, you'll have to determine the correct module
names yourself. You can now add these packages and modules to the
package like so:
.. code-block:: python
import_modules = ['six']
When you run ``spack install --test=root py-six``, Spack will attempt
to import the ``six`` module after installation.
These tests most often catch missing dependencies and non-RPATHed
libraries. Make sure not to add modules/packages containing the word
"test", as these likely won't end up in installation directory.
""""""""""
Unit tests
""""""""""
The package you want to install may come with additional unit tests.
By default, Spack runs:
.. code-block:: console
$ python setup.py test
if it detects that the ``setup.py`` file supports a ``test`` phase.
You can add additional build-time or install-time tests by overriding
``test`` and ``installtest``, respectively. For example, ``py-numpy``
adds:
.. code-block:: python
def install_test(self):
with working_dir('..'):
python('-c', 'import numpy; numpy.test("full", verbose=2)')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Setup file in a sub-directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In order to be compatible with package managers like ``pip``, the package
is required to place its ``setup.py`` in the root of the tarball. However,
not every Python package cares about ``pip`` or PyPI. If you are installing
a package that is not hosted on PyPI, you may find that it places its
``setup.py`` in a sub-directory. To handle this, add the directory containing
``setup.py`` to the package like so:
.. code-block:: python
build_directory = 'source'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Alternate names for setup.py
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As previously mentioned, packages need to call their setup script ``setup.py``
in order to be compatible with package managers like ``pip``. However, some
packages like
`py-meep <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-meep/package.py>`_ and
`py-adios <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-adios/package.py>`_
come with multiple setup scripts, one for a serial build and another for a
parallel build. You can override the default name to use like so:
.. code-block:: python
def setup_file(self):
return 'setup-mpi.py' if '+mpi' in self.spec else 'setup.py'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PythonPackage vs. packages that use Python
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are many packages that make use of Python, but packages that depend
on Python are not necessarily ``PythonPackages``.
"""""""""""""""""""""""
Choosing a build system
"""""""""""""""""""""""
First of all, you need to select a build system. ``spack create`` usually
does this for you, but if for whatever reason you need to do this manually,
choose ``PythonPackage`` if and only if the package contains a ``setup.py``
file.
"""""""""""""""""""""""
Choosing a package name
"""""""""""""""""""""""
Selecting the appropriate package name is a little more complicated
than choosing the build system. By default, ``spack create`` will
prepend ``py-`` to the beginning of the package name if it detects
that the package uses the ``PythonPackage`` build system. However, there
are occasionally packages that use ``PythonPackage`` that shouldn't
start with ``py-``. For example:
* busco
* easybuild
* httpie
* mercurial
* scons
* snakemake
The thing these packages have in common is that they are command-line
tools that just so happen to be written in Python. Someone who wants
to install ``mercurial`` with Spack isn't going to realize that it is
written in Python, and they certainly aren't going to assume the package
is called ``py-mercurial``. For this reason, we manually renamed the
package to ``mercurial``.
Likewise, there are occasionally packages that don't use the
``PythonPackage`` build system but should still be prepended with ``py-``.
For example:
* py-genders
* py-py2cairo
* py-pygobject
* py-pygtk
* py-pyqt
* py-pyserial
* py-sip
* py-xpyb
These packages are primarily used as Python libraries, not as
command-line tools. You may see C/C++ packages that have optional
Python language-bindings, such as:
* antlr
* cantera
* conduit
* pagmo
* vtk
Don't prepend these kind of packages with ``py-``. When in doubt,
think about how this package will be used. Is it primarily a Python
library that will be imported in other Python scripts? Or is it a
command-line tool, or C/C++/Fortran program with optional Python
modules? The former should be prepended with ``py-``, while the
latter should not.
""""""""""""""""""""""
extends vs. depends_on
""""""""""""""""""""""
This is very similar to the naming dilemma above, with a slight twist.
As mentioned in the :ref:`Packaging Guide <packaging_extensions>`,
``extends`` and ``depends_on`` are very similar, but ``extends`` adds
the ability to *activate* the package. Activation involves symlinking
everything in the installation prefix of the package to the installation
prefix of Python. This allows the user to import a Python module without
having to add that module to ``PYTHONPATH``.
When deciding between ``extends`` and ``depends_on``, the best rule of
thumb is to check the installation prefix. If Python libraries are
installed to ``prefix/lib/python2.7/site-packages`` (where 2.7 is the
MAJOR.MINOR version of Python you used to install the package), then
you should use ``extends``. If Python libraries are installed elsewhere
or the only files that get installed reside in ``prefix/bin``, then
don't use ``extends``, as symlinking the package wouldn't be useful.
^^^^^^^^^^^^^^^^^^^^^
Alternatives to Spack
^^^^^^^^^^^^^^^^^^^^^
PyPI has hundreds of thousands of packages that are not yet in Spack,
and ``pip`` may be a perfectly valid alternative to using Spack. The
main advantage of Spack over ``pip`` is its ability to compile
non-Python dependencies. It can also build cythonized versions of a
package or link to an optimized BLAS/LAPACK library like MKL,
resulting in calculations that run orders of magnitude faster.
Spack does not offer a significant advantage to other python-management
systems for installing and using tools like flake8 and sphinx.
But if you need packages with non-Python dependencies like
numpy and scipy, Spack will be very valuable to you.
Anaconda is another great alternative to Spack, and comes with its own
``conda`` package manager. Like Spack, Anaconda is capable of compiling
non-Python dependencies. Anaconda contains many Python packages that
are not yet in Spack, and Spack contains many Python packages that are
not yet in Anaconda. The main advantage of Spack over Anaconda is its
ability to choose a specific compiler and BLAS/LAPACK or MPI library.
Spack also has better platform support for supercomputers. On the
other hand, Anaconda offers Windows support.
^^^^^^^^^^^^^^^^^^^^^^
External documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on Python packaging, see:
https://packaging.python.org/

111
lib/spack/docs/build_systems/qmakepackage.rst Normal file
View File

@ -0,0 +1,111 @@
.. _qmakepackage:
------------
QMakePackage
------------
Much like Autotools and CMake, QMake is a build-script generator
designed by the developers of Qt. In its simplest form, Spack's
``QMakePackage`` runs the following steps:
.. code-block:: console
$ qmake
$ make
$ make check # optional
$ make install
QMake does not appear to have a standardized way of specifying
the installation directory, so you may have to set environment
variables or edit ``*.pro`` files to get things working properly.
^^^^^^
Phases
^^^^^^
The ``QMakePackage`` base class comes with the following phases:
#. ``qmake`` - generate Makefiles
#. ``build`` - build the project
#. ``install`` - install the project
By default, these phases run:
.. code-block:: console
$ qmake
$ make
$ make install
Any of these phases can be overridden in your package as necessary.
There is also a ``check`` method that looks for a ``check`` target
in the Makefile. If a ``check`` target exists and the user runs:
.. code-block:: console
$ spack install --test=root <qmake-package>
Spack will run ``make check`` after the build phase.
^^^^^^^^^^^^^^^
Important files
^^^^^^^^^^^^^^^
Packages that use the QMake build system can be identified by the
presence of a ``<project-name>.pro`` file. This file declares things
like build instructions and dependencies.
One thing to look for is the ``minQtVersion`` function:
.. code-block:: none
minQtVersion(5, 6, 0)
This means that Qt 5.6.0 is the earliest release that will work.
You should specify this in a ``depends_on`` statement.
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
At the bare minimum, packages that use the QMake build system need a
``qt`` dependency. Since this is always the case, the ``QMakePackage``
base class already contains:
.. code-block:: python
depends_on('qt', type='build')
If you want to specify a particular version requirement, or need to
link to the ``qt`` libraries, you can override this in your package:
.. code-block:: python
depends_on('qt@5.6.0:')
^^^^^^^^^^^^^^^^^^^^^^^^^^
Passing arguments to qmake
^^^^^^^^^^^^^^^^^^^^^^^^^^
If you need to pass any arguments to the ``qmake`` call, you can
override the ``qmake_args`` method like so:
.. code-block:: python
def qmake_args(self):
return ['-recursive']
This method can be used to pass flags as well as variables.
^^^^^^^^^^^^^^^^^^^^^^
External documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on the QMake build system, see:
http://doc.qt.io/qt-5/qmake-manual.html

341
lib/spack/docs/build_systems/rpackage.rst Normal file
View File

@ -0,0 +1,341 @@
.. _rpackage:
--------
RPackage
--------
Like Python, R has its own built-in build system.
The R build system is remarkably uniform and well-tested.
This makes it one of the easiest build systems to create
new Spack packages for.
^^^^^^
Phases
^^^^^^
The ``RPackage`` base class has a single phase:
#. ``install`` - install the package
By default, this phase runs the following command:
.. code-block:: console
$ R CMD INSTALL --library=/path/to/installation/prefix/rlib/R/library .
^^^^^^^^^^^^^^^^^^
Finding R packages
^^^^^^^^^^^^^^^^^^
The vast majority of R packages are hosted on CRAN - The Comprehensive
R Archive Network. If you are looking for a particular R package, search
for "CRAN <package-name>" and you should quickly find what you want.
If it isn't on CRAN, try Bioconductor, another common R repository.
For the purposes of this tutorial, we will be walking through
`r-caret <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/r-caret/package.py>`_
as an example. If you search for "CRAN caret", you will quickly find what
you are looking for at https://cran.r-project.org/web/packages/caret/index.html.
If you search for "Package source", you will find the download URL for
the latest release. Use this URL with ``spack create`` to create a new
package.
^^^^^^^^^^^^
Package name
^^^^^^^^^^^^
The first thing you'll notice is that Spack prepends ``r-`` to the front
of the package name. This is how Spack separates R package extensions
from the rest of the packages in Spack. Without this, we would end up
with package name collisions more frequently than we would like. For
instance, there are already packages for both:
* ``ape`` and ``r-ape``
* ``curl`` and ``r-curl``
* ``gmp`` and ``r-gmp``
* ``jpeg`` and ``r-jpeg``
* ``openssl`` and ``r-openssl``
* ``uuid`` and ``r-uuid``
* ``xts`` and ``r-xts``
Many popular programs written in C/C++ are later ported to R as a
separate project.
^^^^^^^^^^^
Description
^^^^^^^^^^^
The first thing you'll need to add to your new package is a description.
The top of the homepage for ``caret`` lists the following description:
caret: Classification and Regression Training
Misc functions for training and plotting classification and regression models.
You can either use the short description (first line), long description
(second line), or both depending on what you feel is most appropriate.
^^^^^^^^
Homepage
^^^^^^^^
If you look at the bottom of the page, you'll see:
Linking:
Please use the canonical form https://CRAN.R-project.org/package=caret to link to this page.
Please uphold the wishes of the CRAN admins and use
https://CRAN.R-project.org/package=caret as the homepage instead of
https://cran.r-project.org/web/packages/caret/index.html. The latter may
change without notice.
^^^
URL
^^^
As previously mentioned, the download URL for the latest release can be
found by searching "Package source" on the homepage.
^^^^^^^^
List URL
^^^^^^^^
CRAN maintains a single webpage containing the latest release of every
single package: https://cran.r-project.org/src/contrib/
Of course, as soon as a new release comes out, the version you were using
in your package is no longer available at that URL. It is moved to an
archive directory. If you search for "Old sources", you will find:
https://cran.r-project.org/src/contrib/Archive/caret
If you only specify the URL for the latest release, your package will
no longer be able to fetch that version as soon as a new release comes
out. To get around this, add the archive directory as a ``list_url``.
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
As an extension of the R ecosystem, your package will obviously depend
on R to build and run. Normally, we would use ``depends_on`` to express
this, but for R packages, we use ``extends``. ``extends`` is similar to
``depends_on``, but adds an additional feature: the ability to "activate"
the package by symlinking it to the R installation directory. Since
every R package needs this, the ``RPackage`` base class contains:
.. code-block:: python
extends('r')
depends_on('r', type=('build', 'run'))
Take a close look at the homepage for ``caret``. If you look at the
"Depends" section, you'll notice that ``caret`` depends on "R (≥ 2.10)".
You should add this to your package like so:
.. code-block:: python
depends_on('r@2.10:', type=('build', 'run'))
^^^^^^^^^^^^^^
R dependencies
^^^^^^^^^^^^^^
R packages are often small and follow the classic Unix philosophy
of doing one thing well. They are modular and usually depend on
several other packages. You may find a single package with over a
hundred dependencies. Luckily, CRAN packages are well-documented
and list all of their dependencies in the following sections:
* Depends
* Imports
* LinkingTo
As far as Spack is concerned, all 3 of these dependency types
correspond to ``type=('build', 'run')``, so you don't have to worry
about them. If you are curious what they mean,
https://github.com/spack/spack/issues/2951 has a pretty good summary:
``Depends`` is required and will cause those R packages to be *attached*,
that is, their APIs are exposed to the user. ``Imports`` *loads* packages
so that *the package* importing these packages can access their APIs,
while *not* being exposed to the user. When a user calls ``library(foo)``
s/he *attaches* package ``foo`` and all of the packages under ``Depends``.
Any function in one of these package can be called directly as ``bar()``.
If there are conflicts, user can also specify ``pkgA::bar()`` and
``pkgB::bar()`` to distinguish between them. Historically, there was only
``Depends`` and ``Suggests``, hence the confusing names. Today, maybe
``Depends`` would have been named ``Attaches``.
The ``LinkingTo`` is not perfect and there was recently an extensive
discussion about API/ABI among other things on the R-devel mailing
list among very skilled R developers:
* https://stat.ethz.ch/pipermail/r-devel/2016-December/073505.html
* https://stat.ethz.ch/pipermail/r-devel/2017-January/073647.html
Some packages also have a fourth section:
* Suggests
These are optional, rarely-used dependencies that a user might find
useful. You should **NOT** add these dependencies to your package.
R packages already have enough dependencies as it is, and adding
optional dependencies can really slow down the concretization
process. They can also introduce circular dependencies.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Core, recommended, and non-core packages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you look at "Depends", "Imports", and "LinkingTo", you will notice
3 different types of packages:
"""""""""""""
Core packages
"""""""""""""
If you look at the ``caret`` homepage, you'll notice a few dependencies
that don't have a link to the package, like ``methods``, ``stats``, and
``utils``. These packages are part of the core R distribution and are
tied to the R version installed. You can basically consider these to be
"R itself". These are so essential to R so it would not make sense that
they could be updated via CRAN. If so, you would basically get a different
version of R. Thus, they're updated when R is updated.
You can find a list of these core libraries at:
https://github.com/wch/r-source/tree/trunk/src/library
""""""""""""""""""""
Recommended packages
""""""""""""""""""""
When you install R, there is an option called ``--with-recommended-packages``.
This flag causes the R installation to include a few "Recommended" packages
(legacy term). They are for historical reasons quite tied to the core R
distribution, developed by the R core team or people closely related to it.
The R core distribution "knows" about these package, but they are indeed
distributed via CRAN. Because they're distributed via CRAN, they can also be
updated between R version releases.
Spack explicitly adds the ``--without-recommended-packages`` flag to prevent
the installation of these packages. Due to the way Spack handles package
activation (symlinking packages to the R installation directory),
pre-existing recommended packages will cause conflicts for already-existing
files. We could either not include these recommended packages in Spack and
require them to be installed through ``--with-recommended-packages``, or
we could not install them with R and let users choose the version of the
package they want to install. We chose the latter.
Since these packages are so commonly distributed with the R system, many
developers may assume these packages exist and fail to list them as
dependencies. Watch out for this.
You can find a list of these recommended packages at:
https://github.com/wch/r-source/blob/trunk/share/make/vars.mk
"""""""""""""""""
Non-core packages
"""""""""""""""""
These are packages that are neither "core" nor "recommended". There are more
than 10,000 of these packages hosted on CRAN alone.
For each of these package types, if you see that a specific version is
required, for example, "lattice (≥ 0.20)", please add this information to
the dependency:
.. code-block:: python
depends_on('r-lattice@0.20:', type=('build', 'run'))
^^^^^^^^^^^^^^^^^^
Non-R dependencies
^^^^^^^^^^^^^^^^^^
Some packages depend on non-R libraries for linking. Check out the
`r-stringi <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/r-stringi/package.py>`_
package for an example: https://CRAN.R-project.org/package=stringi.
If you search for the text "SystemRequirements", you will see:
ICU4C (>= 52, optional)
This is how non-R dependencies are listed. Make sure to add these
dependencies. The default dependency type should suffice.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Passing arguments to the installation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some R packages provide additional flags that can be passed to
``R CMD INSTALL``, often to locate non-R dependencies.
`r-rmpi <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/r-rmpi/package.py>`_
is an example of this, and flags for linking to an MPI library. To pass
these to the installation command, you can override ``configure_args``
like so:
.. code-block:: python
def configure_args(self, spec, prefix):
mpi_name = spec['mpi'].name
# The type of MPI. Supported values are:
# OPENMPI, LAM, MPICH, MPICH2, or CRAY
if mpi_name == 'openmpi':
Rmpi_type = 'OPENMPI'
elif mpi_name == 'mpich':
Rmpi_type = 'MPICH2'
else:
raise InstallError('Unsupported MPI type')
return [
'--with-Rmpi-type={0}'.format(Rmpi_type),
'--with-mpi={0}'.format(spec['mpi'].prefix),
]
There is a similar ``configure_vars`` function that can be overridden
to pass variables to the build.
^^^^^^^^^^^^^^^^^^^^^
Alternatives to Spack
^^^^^^^^^^^^^^^^^^^^^
CRAN hosts over 10,000 R packages, most of which are not in Spack. Many
users may not need the advanced features of Spack, and may prefer to
install R packages the normal way:
.. code-block:: console
$ R
> install.packages("ggplot2")
R will search CRAN for the ``ggplot2`` package and install all necessary
dependencies for you. If you want to update all installed R packages to
the latest release, you can use:
.. code-block:: console
> update.packages(ask = FALSE)
This works great for users who have internet access, but those on an
air-gapped cluster will find it easier to let Spack build a download
mirror and install these packages for you.
Where Spack really shines is its ability to install non-R dependencies
and link to them properly, something the R installation mechanism
cannot handle.
^^^^^^^^^^^^^^^^^^^^^^
External documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on installing R packages, see:
https://stat.ethz.ch/R-manual/R-devel/library/utils/html/INSTALL.html

11
lib/spack/docs/build_systems/rubypackage.rst Normal file
View File

@ -0,0 +1,11 @@
.. _rubypackage:
-----------
RubyPackage
-----------
Like Perl, Python, and R, Ruby has its own build system for
installing Ruby gems.
This build system is a work-in-progress. See
https://github.com/spack/spack/pull/3127 for more information.

301
lib/spack/docs/build_systems/sconspackage.rst Normal file
View File

@ -0,0 +1,301 @@
.. _sconspackage:
------------
SConsPackage
------------
SCons is a general-purpose build system that does not rely on
Makefiles to build software. SCons is written in Python, and handles
all building and linking itself.
As far as build systems go, SCons is very non-uniform. It provides a
common framework for developers to write build scripts, but the build
scripts themselves can vary drastically. Some developers add subcommands
like:
.. code-block:: console
$ scons clean
$ scons build
$ scons test
$ scons install
Others don't add any subcommands. Some have configuration options that
can be specified through variables on the command line. Others don't.
^^^^^^
Phases
^^^^^^
As previously mentioned, SCons allows developers to add subcommands like
``build`` and ``install``, but by default, installation usually looks like:
.. code-block:: console
$ scons
$ scons install
To facilitate this, the ``SConsPackage`` base class provides the
following phases:
#. ``build`` - build the package
#. ``install`` - install the package
Package developers often add unit tests that can be invoked with
``scons test`` or ``scons check``. Spack provides a ``test`` method
to handle this. Since we don't know which one the package developer
chose, the ``test`` method does nothing by default, but can be easily
overridden like so:
.. code-block:: python
def test(self):
scons('check')
^^^^^^^^^^^^^^^
Important files
^^^^^^^^^^^^^^^
SCons packages can be identified by their ``SConstruct`` files. These
files handle everything from setting up subcommands and command-line
options to linking and compiling.
One thing to look for is the ``EnsureSConsVersion`` function:
.. code-block:: none
EnsureSConsVersion(2, 3, 0)
This means that SCons 2.3.0 is the earliest release that will work.
You should specify this in a ``depends_on`` statement.
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
At the bare minimum, packages that use the SCons build system need a
``scons`` dependency. Since this is always the case, the ``SConsPackage``
base class already contains:
.. code-block:: python
depends_on('scons', type='build')
If you want to specify a particular version requirement, you can override
this in your package:
.. code-block:: python
depends_on('scons@2.3.0:', type='build')
^^^^^^^^^^^^^^^^^^^^^^^^^
Finding available options
^^^^^^^^^^^^^^^^^^^^^^^^^
The first place to start when looking for a list of valid options to
build a package is ``scons --help``. Some packages like
`kahip <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/kahip/package.py>`_
don't bother overwriting the default SCons help message, so this isn't
very useful, but other packages like
`serf <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/serf/package.py>`_
print a list of valid command-line variables:
.. code-block:: console
$ scons --help
scons: Reading SConscript files ...
Checking for GNU-compatible C compiler...yes
scons: done reading SConscript files.
PREFIX: Directory to install under ( /path/to/PREFIX )
default: /usr/local
actual: /usr/local
LIBDIR: Directory to install architecture dependent libraries under ( /path/to/LIBDIR )
default: $PREFIX/lib
actual: /usr/local/lib
APR: Path to apr-1-config, or to APR's install area ( /path/to/APR )
default: /usr
actual: /usr
APU: Path to apu-1-config, or to APR's install area ( /path/to/APU )
default: /usr
actual: /usr
OPENSSL: Path to OpenSSL's install area ( /path/to/OPENSSL )
default: /usr
actual: /usr
ZLIB: Path to zlib's install area ( /path/to/ZLIB )
default: /usr
actual: /usr
GSSAPI: Path to GSSAPI's install area ( /path/to/GSSAPI )
default: None
actual: None
DEBUG: Enable debugging info and strict compile warnings (yes|no)
default: False
actual: False
APR_STATIC: Enable using a static compiled APR (yes|no)
default: False
actual: False
CC: Command name or path of the C compiler
default: None
actual: gcc
CFLAGS: Extra flags for the C compiler (space-separated)
default: None
actual:
LIBS: Extra libraries passed to the linker, e.g. "-l<library1> -l<library2>" (space separated)
default: None
actual: None
LINKFLAGS: Extra flags for the linker (space-separated)
default: None
actual:
CPPFLAGS: Extra flags for the C preprocessor (space separated)
default: None
actual: None
Use scons -H for help about command-line options.
More advanced packages like
`cantera <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cantera/package.py>`_
use ``scons --help`` to print a list of subcommands:
.. code-block:: console
$ scons --help
scons: Reading SConscript files ...
SCons build script for Cantera
Basic usage:
'scons help' - print a description of user-specifiable options.
'scons build' - Compile Cantera and the language interfaces using
default options.
'scons clean' - Delete files created while building Cantera.
'[sudo] scons install' - Install Cantera.
'[sudo] scons uninstall' - Uninstall Cantera.
'scons test' - Run all tests which did not previously pass or for which the
results may have changed.
'scons test-reset' - Reset the passing status of all tests.
'scons test-clean' - Delete files created while running the tests.
'scons test-help' - List available tests.
'scons test-NAME' - Run the test named "NAME".
'scons <command> dump' - Dump the state of the SCons environment to the
screen instead of doing <command>, e.g.
'scons build dump'. For debugging purposes.
'scons samples' - Compile the C++ and Fortran samples.
'scons msi' - Build a Windows installer (.msi) for Cantera.
'scons sphinx' - Build the Sphinx documentation
'scons doxygen' - Build the Doxygen documentation
You'll notice that cantera provides a ``scons help`` subcommand. Running
``scons help`` prints a list of valid command-line variables.
^^^^^^^^^^^^^^^^^^^^^^^^^^
Passing arguments to scons
^^^^^^^^^^^^^^^^^^^^^^^^^^
Now that you know what arguments the project accepts, you can add them to
the package build phase. This is done by overriding ``build_args`` like so:
.. code-block:: python
def build_args(self, spec, prefix):
args = [
'PREFIX={0}'.format(prefix),
'ZLIB={0}'.format(spec['zlib'].prefix),
]
if '+debug' in spec:
args.append('DEBUG=yes')
else:
args.append('DEBUG=no')
return args
``SConsPackage`` also provides an ``install_args`` function that you can
override to pass additional arguments to ``scons install``.
^^^^^^^^^^^^^^^^^
Compiler wrappers
^^^^^^^^^^^^^^^^^
By default, SCons builds all packages in a separate execution environment,
and doesn't pass any environment variables from the user environment.
Even changes to ``PATH`` are not propagated unless the package developer
does so.
This is particularly troublesome for Spack's compiler wrappers, which depend
on environment variables to manage dependencies and linking flags. In many
cases, SCons packages are not compatible with Spack's compiler wrappers,
and linking must be done manually.
First of all, check the list of valid options for anything relating to
environment variables. For example, cantera has the following option:
.. code-block:: none
* env_vars: [ string ]
Environment variables to propagate through to SCons. Either the
string "all" or a comma separated list of variable names, e.g.
'LD_LIBRARY_PATH,HOME'.
- default: 'LD_LIBRARY_PATH,PYTHONPATH'
In the case of cantera, using ``env_vars=all`` allows us to use
Spack's compiler wrappers. If you don't see an option related to
environment variables, try using Spack's compiler wrappers by passing
``spack_cc``, ``spack_cxx``, and ``spack_fc`` via the ``CC``, ``CXX``,
and ``FC`` arguments, respectively. If you pass them to the build and
you see an error message like:
.. code-block:: none
Spack compiler must be run from Spack! Input 'SPACK_PREFIX' is missing.
you'll know that the package isn't compatible with Spack's compiler
wrappers. In this case, you'll have to use the path to the actual
compilers, which are stored in ``self.compiler.cc`` and friends.
Note that this may involve passing additional flags to the build to
locate dependencies, a task normally done by the compiler wrappers.
serf is an example of a package with this limitation.
^^^^^^^^^^^^^^^^^^^^^^
External documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on the SCons build system, see:
http://scons.org/documentation.html

124
lib/spack/docs/build_systems/wafpackage.rst Normal file
View File

@ -0,0 +1,124 @@
.. _wafpackage:
----------
WafPackage
----------
Like SCons, Waf is a general-purpose build system that does not rely
on Makefiles to build software.
^^^^^^
Phases
^^^^^^
The ``WafPackage`` base class comes with the following phases:
#. ``configure`` - configure the project
#. ``build`` - build the project
#. ``install`` - install the project
By default, these phases run:
.. code-block:: console
$ python waf configure --prefix=/path/to/installation/prefix
$ python waf build
$ python waf install
Each of these are standard Waf commands and can be found by running:
.. code-block:: console
$ python waf --help
Each phase provides a ``<phase>`` function that runs:
.. code-block:: console
$ python waf -j<jobs> <phase>
where ``<jobs>`` is the number of parallel jobs to build with. Each phase
also has a ``<phase_args>`` function that can pass arguments to this call.
All of these functions are empty except for the ``configure_args``
function, which passes ``--prefix=/path/to/installation/prefix``.
^^^^^^^
Testing
^^^^^^^
``WafPackage`` also provides ``test`` and ``installtest`` methods,
which are run after the ``build`` and ``install`` phases, respectively.
By default, these phases do nothing, but you can override them to
run package-specific unit tests. For example, the
`py-py2cairo <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-py2cairo/package.py>`_
package uses:
.. code-block:: python
def installtest(self):
with working_dir('test'):
pytest = which('py.test')
pytest()
^^^^^^^^^^^^^^^
Important files
^^^^^^^^^^^^^^^
Each Waf package comes with a custom ``waf`` build script, written in
Python. This script contains instructions to build the project.
The package also comes with a ``wscript`` file. This file is used to
override the default ``configure``, ``build``, and ``install`` phases
to customize the Waf project. It also allows developers to override
the default ``./waf --help`` message. Check this file to find useful
information about dependencies and the minimum versions that are
supported.
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
``WafPackage`` does not require ``waf`` to build. ``waf`` is only
needed to create the ``./waf`` script. Since ``./waf`` is a Python
script, Python is needed to build the project. ``WafPackage`` adds
the following dependency automatically:
.. code-block:: python
depends_on('python@2.5:', type='build')
Waf only supports Python 2.5 and up.
^^^^^^^^^^^^^^^^^^^^^^^^
Passing arguments to waf
^^^^^^^^^^^^^^^^^^^^^^^^
As previously mentioned, each phase comes with a ``<phase_args>``
function that can be used to pass arguments to that particular
phase. For example, if you need to pass arguments to the build
phase, you can use:
.. code-block:: python
def build_args(self, spec, prefix):
args = []
if self.run_tests:
args.append('--test')
return args
A list of valid options can be found by running ``./waf --help``.
^^^^^^^^^^^^^^^^^^^^^^
External documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on the Waf build system, see:
https://waf.io/book/

1
lib/spack/docs/index.rst
View File

@ -73,6 +73,7 @@ or refer to the full manual below.
contribution_guide
packaging_guide
build_systems
developer_guide
docker_for_developers
Spack API Docs <spack>