README.rst 33.2 KB
Newer Older
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
1 2 3 4 5 6
=====================
 slapos.recipe.build
=====================

.. contents::

Julien Muchembled's avatar
Julien Muchembled committed
7 8 9 10 11
The default recipe can be used to execute ad-hoc Python code at
init/install/update phases. `install` must create the path pointed to by
`location` (default is ${buildout:parts-directory}/${:_buildout_section_name_})
and any other file system change is not tracked by buildout. `install` defaults
to `update`, in which case `location` is ignored.
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
12

Julien Muchembled's avatar
Julien Muchembled committed
13
Example that installs software::
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
14 15 16 17 18 19 20 21 22

  [buildout]
  parts =
    script

  [script]
  recipe = slapos.recipe.build
  slapos_promise =
    directory:include
Julien Muchembled's avatar
Julien Muchembled committed
23 24 25 26
    file:share/man/man1/foo.1
    statlib:lib/libfoo.a
    statlib:lib/libfoo.la
    dynlib:bin/foo linked:libbar.so.1,libc.so.6,libfoo.so.1 rpath:${bar:location}/lib,!/lib
27 28
  url = http://host/path/foo.tar.gz
  md5sum = ...
Julien Muchembled's avatar
Julien Muchembled committed
29
  install =
30
    extract_dir = self.extract(self.download())
Julien Muchembled's avatar
Julien Muchembled committed
31 32 33 34
    self.copyTree(guessworkdir(extract_dir), location)
    ${:update}
  update =
    ...
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
35

36 37 38 39 40
Using the init option::

  [section-one]
  recipe = slapos.recipe.build
  init =
Julien Muchembled's avatar
Julien Muchembled committed
41 42
    import platform
    options['foo'] = platform.uname()[4]
43 44 45 46

  [section-two]
  bar = ${section-one:foo}

47
In case of error, a proper traceback is displayed and nothing is installed::
48

49
  >>> write(sample_buildout, 'buildout.cfg', """
50
  ... [buildout]
51
  ... parts = script
52
  ...
53
  ... [script]
54 55 56
  ... recipe = slapos.recipe.build
  ... install =
  ...   import os
57 58
  ...   os.mkdir(location)
  ...   print(1 / 0.) # this is an error !
59 60 61
  ... """)

  >>> print(system(buildout))
62 63 64 65 66 67 68
  Installing script.
  While:
    Installing script.
  <BLANKLINE>
  An internal error occurred due to a bug in either zc.buildout or in a
  recipe being used:
  Traceback (most recent call last):
69
  ...
70 71 72
    File "script", line 3, in <module>
      print(1 / 0.) # this is an error !
  ZeroDivisionError: float division by zero
73 74

  >>> ls(sample_buildout, 'parts')
75
  <BLANKLINE>
76

77 78
option: environment
-------------------
79

80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96
Customizing environment variables can be easier with the this option.
Values are expanded with Python %-dict formatting, using ``os.environ``. The
resulting environ dict is computed on first access of ``self.environ``.
Environment variables can be either inlined::

  >>> base = """
  ... [buildout]
  ... parts = script
  ...
  ... [script]
  ... recipe = slapos.recipe.build
  ... update =
  ...   import os
  ...   os.environ["FOO"] = "1"
  ...   print("%(FOO)s %(BAR)s" % self.environ)
  ...   os.environ["FOO"] = "2"
  ...   print("%(FOO)s %(BAR)s" % self.environ)
97
  ... """
98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137
  >>> write(sample_buildout, 'buildout.cfg', base + """
  ... environment =
  ...   BAR=%(FOO)s:%%
  ... """)
  >>> print(system(buildout))
  Installing script.
  script: [ENV] BAR = 1:%
  1 1:%
  1 1:%

or put inside a separate section::

  >>> write(sample_buildout, 'buildout.cfg', base + """
  ... environment = env
  ... [env]
  ... BAR=%(FOO)s:%%
  ... """)
  >>> print(system(buildout))
  Uninstalling script.
  Installing script.
  script: [ENV] BAR = 1:%
  1 1:%
  1 1:%

This option works the same way in other recipes that support it, in which case
the resulting environ dict is computed at install/update.

option: shared
--------------

Boolean (``false`` by default, or ``true``), this option specifies that the
part can be installed in a shared mode. This is enabled if paths are listed in
the ``shared-part-list`` option of the ``[buildout]`` section: the location of
the part is ``<one of shared-part-list>/<part name>/<hash of options>`` and
it contains a signature file ``.buildout-shared.json``.

`install` option is required::

  >>> del MD5SUM[:]
  >>> base = """
138
  ... [buildout]
139 140 141 142
  ... parts = script
  ... shared-part-list =
  ...   ${:directory}/shared1
  ...   ${:directory}/shared2
143
  ...
144
  ... [script]
145
  ... recipe = slapos.recipe.build
146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161
  ... shared = true
  ... """
  >>> write(sample_buildout, 'buildout.cfg', base + """
  ... init = pass
  ... """)
  >>> print(system(buildout))
  script: shared at .../shared2/script/<MD5SUM:0>
  While:
    Installing.
    Getting section script.
    Initializing section script.
  Error: When shared=true, option 'install' must be set

`update` option is incompatible::

  >>> base += """
162 163
  ... install =
  ...   import os
164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224
  ...   os.makedirs(os.path.join(location, 'foo'))
  ...   print("directory created")
  ... """
  >>> write(sample_buildout, 'buildout.cfg', base)
  >>> print(system(buildout + ' script:update=pass'))
  script: shared at .../shared2/script/<MD5SUM:1>
  While:
    Installing.
    Getting section script.
    Initializing section script.
  Error: When shared=true, option 'update' can't be set

A shared part is installed in the last folder that is listed by
``shared-part-list``::

  >>> print(system(buildout))
  script: shared at .../shared2/script/<MD5SUM:2>
  Uninstalling script.
  Installing script.
  directory created
  >>> shared = 'shared2/script/' + MD5SUM[2]
  >>> ls(shared)
  -  .buildout-shared.json
  l  .buildout-shared.signature
  d  foo

``.buildout-shared.signature`` is only there for backward compatibility.

Uninstalling the part leaves the shared part available::

  >>> print(system(buildout + ' buildout:parts='))
  Uninstalling script.
  Unused options for buildout: 'shared-part-list'.
  >>> ls(shared)
  -  .buildout-shared.json
  l  .buildout-shared.signature
  d  foo

And reinstalling is instantaneous::

  >>> print(system(buildout))
  script: shared at .../shared2/script/<MD5SUM:2>
  Installing script.
  script: shared part is already installed

Setting `location` option is incompatible::

  >>> write(sample_buildout, 'buildout.cfg', base + """
  ... init =
  ...   import os
  ...   options['location'] = os.path.join(
  ...     self.buildout['buildout']['parts-directory'], 'foo')
  ... """)
  >>> print(system(buildout))
  script: shared at .../shared2/script/<MD5SUM:3>
  While:
    Installing.
    Getting section script.
    Initializing section script.
  Error: When shared=true, option 'location' can't be set

225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267
option: location
----------------

If empty or unset, the value is initialized automatically according to rules
defined above (`shared` option), and before the `init` code is executed.
This way, it's possible to initialize other values of the section depending
on the actual value of location.

If not shared, the value can be customized in `init`. This is actually the
only way to empty location, which is useful if there's nothing file/directory
to track but you need to distinguish install from update::

  >>> write(sample_buildout, 'buildout.cfg', """
  ... [buildout]
  ... parts = script
  ...
  ... [script]
  ... recipe = slapos.recipe.build
  ... init =
  ...   options['location'] = ''
  ... install =
  ...   print("install")
  ... update =
  ...   print("update")
  ... """)

  >>> print(system(buildout))
  Uninstalling script.
  Installing script.
  install
  >>> print(system(buildout))
  Updating script.
  update
  >>> cat('.installed.cfg')
  [buildout]
  ...
  [script]
  __buildout_installed__ =
  __buildout_signature__ = ...

If install & update run the same code, `install` can be unset (or empty)
and you can ignore `location`.

268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292

=============================
 slapos.recipe.build:download
=============================

Simplest usage is to only specify a URL::

  >>> base = """
  ... [buildout]
  ... parts = download
  ...
  ... [download]
  ... recipe = slapos.recipe.build:download
  ... url = https://lab.nexedi.com/nexedi/slapos.recipe.build/raw/master/MANIFEST.in
  ... """
  >>> write(sample_buildout, 'buildout.cfg', base)
  >>> print(system(buildout))
  Uninstalling script.
  Installing download.
  Downloading ...
  >>> ls('parts/download')
  -  download

The file is downloaded to ``parts/<section_name>/<section_name>``.

293 294 295 296
Because the destination file may be hardlinked (e.g. download from cache
or from local file), it shall not be modified in-place without first making
sure that ``st_nlink`` is 1.

297 298 299 300 301 302 303
option: filename
----------------

In the part folder, the filename can be customized::

  >>> write(sample_buildout, 'buildout.cfg', base + """
  ... filename = somefile
304
  ... """)
305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324
  >>> print(system(buildout))
  Uninstalling download.
  Installing download.
  Downloading ...
  >>> ls('parts/download')
  -  somefile

When an MD5 checksum is not given, updating the part downloads the file again::

  >>> remove('parts/download/somefile')
  >>> print(system(buildout))
  Updating download.
  Downloading ...
  >>> ls('parts/download')
  -  somefile

option: destination
-------------------

Rather than having a file inside a part folder, a full path can be given::
325

326 327 328
  >>> write(sample_buildout, 'buildout.cfg', base + """
  ... destination = ${buildout:parts-directory}/somepath
  ... """)
329
  >>> print(system(buildout))
330 331 332 333 334 335 336 337 338 339 340 341 342
  Uninstalling download.
  Installing download.
  Downloading ...
  >>> ls('parts')
  -  somepath

option: target
--------------

In any case, path to download file is exposed by the ``target`` option::

  >>> cat('.installed.cfg')
  [buildout]
343
  ...
344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382
  [download]
  __buildout_installed__ = .../parts/somepath
  __buildout_signature__ = ...
  destination = .../parts/somepath
  recipe = slapos.recipe.build:download
  target = .../parts/somepath
  url = ...

option: md5sum
--------------

An MD5 checksum can be specified to check the contents::

  >>> base += """
  ... md5sum = b90c12a875df544907bc84d9c7930653
  ... """
  >>> write(sample_buildout, 'buildout.cfg', base)
  >>> print(system(buildout))
  Uninstalling download.
  Installing download.
  Downloading ...
  >>> ls('parts/download')
  -  download

In such case, updating the part does nothing::

  >>> remove('parts/download/download')
  >>> print(system(buildout))
  Updating download.
  >>> ls('parts/download')

In case of checksum mismatch::

  >>> print(system(buildout
  ... + ' download:md5sum=00000000000000000000000000000000'
  ... ))
  Uninstalling download.
  Installing download.
  Downloading ...
383
  While:
384 385 386 387
    Installing download.
  Error: MD5 checksum mismatch downloading '...'
  >>> ls('parts')

388 389 390 391 392 393
option: offline
---------------

Boolean option that can be specified to override `${buildout:offline}`.


394 395 396 397 398 399 400 401 402 403 404
option: alternate-url
---------------------

Alternate URL. If supported by Buildout, it is used as fallback if the main
URL (`url` option) fails at HTTP level.

Useful when a version of a resource can only be downloaded with a temporary
URL as long as it's the last version, and this version is then moved to a
permanent place when a newer version is released: `url` shall be the final URL
and `alternate-url` the temporary one.

405 406 407 408 409 410 411 412 413 414 415 416 417
option: shared
--------------

Works like the default recipe. Constraints on options are:

- ``md5sum`` option is required
- ``destination`` option is incompatible

Example::

  >>> del MD5SUM[4:] # drop added values since previous shared test
  >>> write(sample_buildout, 'buildout.cfg', base + """
  ... shared = true
418
  ...
419 420 421 422 423 424 425 426 427 428 429 430 431
  ... [buildout]
  ... shared-part-list =
  ...   ${:directory}/shared
  ... """)
  >>> print(system(buildout))
  download: shared at .../shared/download/<MD5SUM:4>
  Installing download.
  Downloading ...
  >>> shared = 'shared/download/' + MD5SUM[4]
  >>> ls(shared)
  -  .buildout-shared.json
  l  .buildout-shared.signature
  -  download
432 433


434 435 436
=======================================
 slapos.recipe.build:download-unpacked
=======================================
437

438 439 440
Downloads and extracts an archive. In addition to format that setuptools is
able to extract, XZ & lzip compression are also supported if ``xzcat`` &
``lunzip`` executables are available.
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
441

442 443
By default, the archive is extracted to ``parts/<section_name>`` and a single
directory at the root of the archive is stripped::
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
444

445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464
  >>> URL = "https://lab.nexedi.com/nexedi/slapos.recipe.build/-/archive/master/slapos.recipe.build-master.tar.gz?path=slapos/recipe/build"
  >>> base = """
  ... [buildout]
  ... download-cache = download-cache
  ... parts = download
  ...
  ... [download]
  ... recipe = slapos.recipe.build:download-unpacked
  ... url = %s
  ... """ % URL
  >>> write(sample_buildout, 'buildout.cfg', base)
  >>> print(system(buildout))
  Creating directory '.../download-cache'.
  Uninstalling download.
  Installing download.
  Downloading ...
  >>> ls('parts/download')
  d  slapos

The download cache will avoid to download the same tarball several times.
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
465

466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488
option: destination
-------------------

Similar to ``download`` recipe::

  >>> write(sample_buildout, 'buildout.cfg', base + """
  ... destination = ${buildout:parts-directory}/somepath
  ... """)
  >>> print(system(buildout))
  Uninstalling download.
  Installing download.
  >>> ls('parts/somepath')
  d  slapos

option: target
--------------

Like for ``download`` recipe, the installation path of the part is exposed by
the ``target`` option::

  >>> cat('.installed.cfg')
  [buildout]
  ...
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
489
  [download]
490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517
  __buildout_installed__ = .../parts/somepath
  __buildout_signature__ = ...
  destination = .../parts/somepath
  recipe = slapos.recipe.build:download-unpacked
  target = .../parts/somepath
  url = ...

option: strip-top-level-dir
---------------------------

Stripping can be enforced::

  >>> print(system(buildout + ' download:strip-top-level-dir=true'))
  Uninstalling download.
  Installing download.
  >>> ls('parts/somepath')
  d  slapos

Or disabled::

  >>> print(system(buildout + ' download:strip-top-level-dir=false'))
  Uninstalling download.
  Installing download.
  >>> ls('parts/somepath')
  d  slapos.recipe.build-master-slapos-recipe-build

option: md5sum
--------------
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
518

519 520
An MD5 checksum can be specified to check the downloaded file, like for the
``download`` recipe. However, if unset, updating the part does nothing.
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
521

522 523 524 525 526
option: alternate-url
---------------------

See the ``download`` recipe.

527 528
option: environment
-------------------
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
529

530 531
Like for the default recipe, environment variables can be customized, here
for ``xzcat`` & ``lunzip`` subprocesses (e.g. PATH).
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
532

533 534
option: shared
--------------
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
535

536 537
Works like the default recipe. The only constraint on options is that
the ``destination`` option is incompatible.
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
538

539
Example::
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
540

541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556
  >>> del MD5SUM[5:] # drop added values since previous shared test
  >>> write(sample_buildout, 'buildout.cfg', """
  ... [buildout]
  ... download-cache = download-cache
  ... parts = download
  ... shared-part-list = ${:directory}/shared
  ...
  ... [download]
  ... recipe = slapos.recipe.build:download-unpacked
  ... url = %s
  ... shared = true
  ... """ % URL)
  >>> print(system(buildout))
  download: shared at .../shared/download/<MD5SUM:5>
  Uninstalling download.
  Installing download.
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
557 558 559 560 561


==============================
 slapos.recipe.build:gitclone
==============================
Antoine Catton's avatar
Antoine Catton committed
562

563
Checkout a git repository and its submodules by default.
564 565
Supports slapos.libnetworkcache if present, and if boolean 'use-cache' option
is true.
Antoine Catton's avatar
Antoine Catton committed
566 567

Examples
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
568
--------
Antoine Catton's avatar
Antoine Catton committed
569

570 571
Those examples use slapos.recipe.build repository as an example.

Antoine Catton's avatar
Antoine Catton committed
572
Simple clone
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
573
~~~~~~~~~~~~
Antoine Catton's avatar
Antoine Catton committed
574

575
Only `repository` parameter is required. For each buildout run,
Antoine Catton's avatar
Antoine Catton committed
576 577
the recipe will pick up the latest commit on the remote master branch::

578 579 580 581 582 583 584
  >>> write(sample_buildout, 'buildout.cfg',
  ... """
  ... [buildout]
  ... parts = git-clone
  ...
  ... [git-clone]
  ... recipe = slapos.recipe.build:gitclone
585
  ... repository = https://lab.nexedi.com/nexedi/slapos.recipe.build.git
586
  ... use-cache = true
587
  ... """)
Antoine Catton's avatar
Antoine Catton committed
588 589

This will clone the git repository in `parts/git-clone` directory.
590 591
Then let's run the buildout::

Bryton Lacquement's avatar
Bryton Lacquement committed
592
  >>> print(system(buildout))
593
  Uninstalling download.
594
  Installing git-clone.
595
  Cloning into '/sample-buildout/parts/git-clone'...
596 597 598 599 600

Let's take a look at the buildout parts directory now::

  >>> ls(sample_buildout, 'parts')
  d git-clone
Antoine Catton's avatar
Antoine Catton committed
601

602 603
When updating, it will do a "git fetch; git reset @{upstream}"::

Bryton Lacquement's avatar
Bryton Lacquement committed
604
  >>> print(system(buildout))
605 606 607 608
  Updating git-clone.
  Fetching origin
  HEAD is now at ...

Antoine Catton's avatar
Antoine Catton committed
609
Specific branch
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
610
~~~~~~~~~~~~~~~
Antoine Catton's avatar
Antoine Catton committed
611 612 613 614

You can specify a specific branch using `branch` option. For each
run it will take the latest commit on this remote branch::

615 616 617 618 619 620 621
  >>> write(sample_buildout, 'buildout.cfg',
  ... """
  ... [buildout]
  ... parts = git-clone
  ...
  ... [git-clone]
  ... recipe = slapos.recipe.build:gitclone
622 623
  ... repository = https://lab.nexedi.com/nexedi/slapos.recipe.build.git
  ... branch = build_remove_downloaded_files
624
  ... """)
Antoine Catton's avatar
Antoine Catton committed
625

626 627
Then let's run the buildout::

Bryton Lacquement's avatar
Bryton Lacquement committed
628
  >>> print(system(buildout))
629
  Uninstalling git-clone.
630
  Running uninstall recipe.
631
  Installing git-clone.
632
  Cloning into '/sample-buildout/parts/git-clone'...
633 634 635 636 637 638 639

Let's take a look at the buildout parts directory now::

  >>> ls(sample_buildout, 'parts')
  d git-clone

And let's see that current branch is "build"::
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
640

641 642
  >>> import subprocess
  >>> cd('parts', 'git-clone')
Bryton Lacquement's avatar
Bryton Lacquement committed
643
  >>> print(subprocess.check_output(['git', 'branch'], universal_newlines=True))
644
  * build_remove_downloaded_files
Antoine Catton's avatar
Antoine Catton committed
645

646 647 648
When updating, it will do a "git fetch; git reset build"::

  >>> cd(sample_buildout)
Bryton Lacquement's avatar
Bryton Lacquement committed
649
  >>> print(system(buildout))
650 651 652 653
  Updating git-clone.
  Fetching origin
  HEAD is now at ...

Antoine Catton's avatar
Antoine Catton committed
654
Specific revision
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
655
~~~~~~~~~~~~~~~~~
Antoine Catton's avatar
Antoine Catton committed
656 657

You can specify a specific commit hash or tag using `revision` option.
658
This option has priority over the "branch" option::
Antoine Catton's avatar
Antoine Catton committed
659

660 661 662 663 664 665 666 667
  >>> cd(sample_buildout)
  >>> write(sample_buildout, 'buildout.cfg',
  ... """
  ... [buildout]
  ... parts = git-clone
  ...
  ... [git-clone]
  ... recipe = slapos.recipe.build:gitclone
668
  ... repository = https://lab.nexedi.com/nexedi/slapos.recipe.build.git
669 670
  ... revision = 2566127
  ... """)
Antoine Catton's avatar
Antoine Catton committed
671

672 673
Then let's run the buildout::

Bryton Lacquement's avatar
Bryton Lacquement committed
674
  >>> print(system(buildout))
675
  Uninstalling git-clone.
676
  Running uninstall recipe.
677
  Installing git-clone.
678
  Cloning into '/sample-buildout/parts/git-clone'...
679
  HEAD is now at 2566127 ...
680 681 682 683 684 685

Let's take a look at the buildout parts directory now::

  >>> ls(sample_buildout, 'parts')
  d git-clone

686
And let's see that current revision is "2566127"::
687 688 689

  >>> import subprocess
  >>> cd(sample_buildout, 'parts', 'git-clone')
Bryton Lacquement's avatar
Bryton Lacquement committed
690
  >>> print(subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD'], universal_newlines=True))
691
  2566127
Antoine Catton's avatar
Antoine Catton committed
692

693
When updating, it shouldn't do anything as revision is mentioned::
694 695

  >>> cd(sample_buildout)
Bryton Lacquement's avatar
Bryton Lacquement committed
696
  >>> print(system(buildout))
697
  Updating git-clone.
698

699
Empty revision/branch
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
700
~~~~~~~~~~~~~~~~~~~~~
701 702 703 704 705 706 707 708 709 710 711 712 713

Specifying an empty revision or an empty branch will make buildout
ignore those values as if it was not present at all (allowing to easily
extend an existing section specifying a branch)::

  >>> cd(sample_buildout)
  >>> write(sample_buildout, 'buildout.cfg',
  ... """
  ... [buildout]
  ... parts = git-clone
  ...
  ... [git-clone-with-branch]
  ... recipe = slapos.recipe.build:gitclone
714
  ... repository = https://lab.nexedi.com/nexedi/slapos.recipe.build.git
715 716 717 718 719 720 721 722
  ... revision = 2566127
  ...
  ... [git-clone]
  ... <= git-clone-with-branch
  ... revision =
  ... branch = master
  ... """)

Bryton Lacquement's avatar
Bryton Lacquement committed
723
  >>> print(system(buildout))
724 725 726 727 728 729
  Uninstalling git-clone.
  Running uninstall recipe.
  Installing git-clone.
  Cloning into '/sample-buildout/parts/git-clone'...

  >>> cd(sample_buildout, 'parts', 'git-clone')
Bryton Lacquement's avatar
Bryton Lacquement committed
730
  >>> print(system('git branch'))
731
  * master
732

733
Revision/branch priority
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
734
~~~~~~~~~~~~~~~~~~~~~~~~
735 736 737 738 739 740 741 742 743 744 745 746

If both revision and branch parameters are set, revision parameters is used
and branch parameter is ignored::

  >>> cd(sample_buildout)
  >>> write(sample_buildout, 'buildout.cfg',
  ... """
  ... [buildout]
  ... parts = git-clone
  ...
  ... [git-clone]
  ... recipe = slapos.recipe.build:gitclone
747
  ... repository = https://lab.nexedi.com/nexedi/slapos.recipe.build.git
748 749 750 751
  ... branch = mybranch
  ... revision = 2566127
  ... """)

Bryton Lacquement's avatar
Bryton Lacquement committed
752
  >>> print(system(buildout))
753 754 755 756 757 758 759 760
  Uninstalling git-clone.
  Running uninstall recipe.
  Installing git-clone.
  Warning: "branch" parameter with value "mybranch" is ignored. Checking out to revision 2566127...
  Cloning into '/sample-buildout/parts/git-clone'...
  HEAD is now at 2566127 ...

  >>> cd(sample_buildout, 'parts', 'git-clone')
Bryton Lacquement's avatar
Bryton Lacquement committed
761
  >>> print(system('git branch'))
762 763
  * master

764
Setup a "develop" repository
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
765
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
766

767
If you need to setup a repository that will be manually altered over time for
768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786
development purposes, you need to make sure buildout will NOT alter it and NOT
erase your local modifications by specifying the "develop" flag::

  [buildout]
  parts = git-clone

  [git-clone]
  recipe = slapos.recipe.build:gitclone
  repository = https://example.net/example.git/
  develop = true

  >>> cd(sample_buildout)
  >>> write(sample_buildout, 'buildout.cfg',
  ... """
  ... [buildout]
  ... parts = git-clone
  ...
  ... [git-clone]
  ... recipe = slapos.recipe.build:gitclone
787
  ... repository = https://lab.nexedi.com/nexedi/slapos.recipe.build.git
788 789
  ... develop = true
  ... """)
Antoine Catton's avatar
Antoine Catton committed
790

Bryton Lacquement's avatar
Bryton Lacquement committed
791
  >>> print(system(buildout))
792 793 794 795 796 797 798 799 800
  Uninstalling git-clone.
  Running uninstall recipe.
  Installing git-clone.
  Cloning into '/sample-buildout/parts/git-clone'...

Buildout will then keep local modifications, instead of resetting the
repository::

  >>> cd(sample_buildout, 'parts', 'git-clone')
Bryton Lacquement's avatar
Bryton Lacquement committed
801
  >>> print(system('echo foo > setup.py'))
802 803

  >>> cd(sample_buildout)
Bryton Lacquement's avatar
Bryton Lacquement committed
804
  >>> print(system(buildout))
805 806 807
  Updating git-clone.

  >>> cd(sample_buildout, 'parts', 'git-clone')
Bryton Lacquement's avatar
Bryton Lacquement committed
808
  >>> print(system('cat setup.py'))
809 810
  foo

811 812 813
Then, when update occurs, nothing is done::

  >>> cd(sample_buildout, 'parts', 'git-clone')
Bryton Lacquement's avatar
Bryton Lacquement committed
814
  >>> print(system('echo kept > local_change'))
815

Bryton Lacquement's avatar
Bryton Lacquement committed
816
  >>> print(system('git remote add broken http://git.erp5.org/repos/nowhere'))
817 818 819
  ...

  >>> cd(sample_buildout)
Bryton Lacquement's avatar
Bryton Lacquement committed
820
  >>> print(system(buildout))
821
  Updating git-clone.
822

823
  >>> cd(sample_buildout, 'parts', 'git-clone')
Bryton Lacquement's avatar
Bryton Lacquement committed
824
  >>> print(system('cat local_change'))
825 826
  kept

827 828 829 830 831 832 833 834 835 836
In case of uninstall, buildout will keep the repository directory::

  >>> cd(sample_buildout)
  >>> write(sample_buildout, 'buildout.cfg',
  ... """
  ... [buildout]
  ... parts = git-clone
  ...
  ... [git-clone]
  ... recipe = slapos.recipe.build:gitclone
837
  ... repository = https://lab.nexedi.com/nexedi/slapos.recipe.build.git
838 839 840 841 842
  ... develop = true
  ... # Triggers uninstall/install because of section signature change
  ... foo = bar
  ... """)

Bryton Lacquement's avatar
Bryton Lacquement committed
843
  >>> print(system(buildout))
844 845
  Uninstalling git-clone.
  Running uninstall recipe.
846
  You have uncommitted changes in /sample-buildout/parts/git-clone. This folder will be left as is.
847 848
  Installing git-clone.
  destination directory already exists.
Rafael Monnerat's avatar
Rafael Monnerat committed
849 850
  ...
  <BLANKLINE>
Antoine Catton's avatar
Antoine Catton committed
851

852 853
Specific git binary
~~~~~~~~~~~~~~~~~~~
854

855 856 857
The default git command is `git`, if for a any reason you don't
have git in your path, you can specify git binary path with `git-command`
option.
858

859
Ignore SSL certificate
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
860
~~~~~~~~~~~~~~~~~~~~~~
861 862 863 864 865 866 867 868 869 870 871 872 873 874

By default, when remote server use SSL protocol git checks if the SSL
certificate of the remote server is valid before executing commands.
You can force git to ignore this check using `ignore-ssl-certificate`
boolean option::

  [buildout]
  parts = git-clone

  [git-clone]
  recipe = slapos.recipe.build:gitclone
  repository = https://example.net/example.git/
  ignore-ssl-certificate = true

875 876 877 878 879 880 881 882 883 884 885 886 887 888 889
Ignore cloning submodules
~~~~~~~~~~~~~~~~~~~~~~~~~

By default, cloning the repository will clone its submodules also. You can force
git to ignore cloning submodules by defining `ignore-cloning-submodules` boolean
option to 'true'::

  [buildout]
  parts = git-clone

  [git-clone]
  recipe = slapos.recipe.build:gitclone
  repository = https://lab.nexedi.com/tiwariayush/test_erp5
  ignore-cloning-submodules = true

890 891 892
Other options
~~~~~~~~~~~~~

893 894 895
depth
    Clone with ``--depth=<depth>`` option. See ``git-clone`` command.

896 897 898 899 900 901 902 903 904
shared
    Clone with ``--shared`` option if true. See ``git-clone`` command.

sparse-checkout
    The value of the `sparse-checkout` option is written to the
    ``$GITDIR/info/sparse-checkout`` file, which is used to populate the working
    directory sparsely. See the `SPARSE CHECKOUT` section of ``git-read-tree``
    command. This feature is disabled if the value is empty or unset.

Antoine Catton's avatar
Antoine Catton committed
905
Full example
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
906
~~~~~~~~~~~~
Antoine Catton's avatar
Antoine Catton committed
907 908 909 910 911 912 913 914 915 916 917 918

::

  [buildout]
  parts = git-clone

  [git-binary]
  recipe = hexagonit.recipe.cmmi
  url = http://git-core.googlecode.com/files/git-1.7.12.tar.gz

  [git-clone]
  recipe = slapos.recipe.build:gitclone
919
  repository = http://example.net/example.git/
Antoine Catton's avatar
Antoine Catton committed
920
  git-command = ${git-binary:location}/bin/git
921
  revision = 0123456789abcdef
Antoine Catton's avatar
Antoine Catton committed
922

923

Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
924 925 926
=========================
 slapos.recipe.build:npm
=========================
927 928 929 930

Downloads and installs node.js packages using Node Package Manager (NPM).

Examples
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
931
--------
932 933

Basic example
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
934
~~~~~~~~~~~~~
935 936 937 938 939 940 941 942 943 944 945

Here is example to install one or several modules::

  [buildout]
  parts = node-package

  [node-package]
  recipe = slapos.recipe.build:npm
  modules =
    colors
    express
946

947 948 949 950 951 952
  # Optional argument specifying perl buildout part, if existing.
  # If specified, recipe will use the perl installed by buildout.
  # If not specified, will take the globally available perl executable.
  node = node-0.6

Specific version
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
953
~~~~~~~~~~~~~~~~
Cédric de Saint Martin's avatar
Cédric de Saint Martin committed
954
::
955 956 957 958 959

  [buildout]
  parts = node-package

  [node-package]
Kazuhiko Shiozaki's avatar
Kazuhiko Shiozaki committed
960
  recipe = slapos.recipe.build:npm
961
  modules =
962
    express@1.0.2
963
  node = node-0.6
964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980

==========================
 slapos.recipe.build:vm.*
==========================

This is a set of recipes to build Virtual Machine images and execute commands
inside them. They rely on QEMU and OpenSSH: executables are found via the PATH
environment variable. They do nothing on update.

Common options
--------------

location
    Folder where the recipe stores any produced file.
    Default: ${buildout:parts-directory}/<section_name>

environment
981
    Extra environment to spawn executables. See the default recipe.
982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030

mem
    Python expression evaluating to an integer that specifies the
    RAM size in MB for the VM.

smp
    Number of CPUs for the VM. Default: 1

Example
~~~~~~~

::

  [vm-run-environment]
  PATH = ${openssh:location}/bin:${qemu:location}/bin:%(PATH)s

  [vm-run-base]
  recipe = slapos.recipe.build:vm.run
  environment = vm-run-environment
  mem = 256 * (${:smp} + 1)
  smp = 4

slapos.recipe.build:vm.install-debian
-------------------------------------

Install Debian from an ISO image. Additional required binaries:

- ``7z`` (from 7zip), to extract kernel/initrd from the ISO;
- ``file``, which is used to test that the VM image is bootable.

Currently, it only produces `raw` images, in `discard` mode (see ``-drive``
QEMU option): combined the use of ``discard`` mount option, this minimizes
the used space on disk.

Options
~~~~~~~

location
    Produced files: ``<dist>.img`` (1 for each token of `dists`), ``passwd``
    and optionally ``ssh.key``

arch
    QEMU architecture (the recipe runs the ``qemu-system-<arch>`` executable).
    It is also used to select the ISO in the sections refered by `dists`.
    Default to host architecture.

dists
    List of VMs to build: each token refers to a buildout section name that
    describes the ISOs to use. See `ISO sections`_ below.
1031
    Tokens can't contain `'.'` characters.
1032 1033 1034 1035 1036 1037

size
    Size of the VM image. This must be an integer, optionally followed by a
    IEC or SI suffix.

mem
1038
    Default: 384
1039

1040
[<dist>/]preseed.<preseed>
1041 1042
    Set the <preseed> value for the installation. The recipe has many default
    preseed values: you can see the list in the ``InstallDebianRecipe.preseed``
1043 1044 1045 1046
    class attribute (file ``slapos/recipe/vm.py``). Aliases are recognized
    (but the recipe includes a mapping that may be out-of-date.).
    Any value except ``passwd/*`` can optionally be prefixed so that they only
    apply for a particular VM.
1047

1048
[<dist>/]debconf.<owner>
1049 1050
    List of debconf value for <owner> (usually a package name),
    each line with 2 whitespace-separated parts: <key> <value>.
1051
    Like for preseed.* values, they can be specific to <dist>.
1052 1053 1054 1055

late-command
    Shell commands to execute at the end of the installation. They are run
    inside the target system. This is a reliable alternative to the
1056 1057
    ``preseed.preseed/late_command`` option. The ``DIST`` shell variable is
    set to the VM being built.
1058 1059

packages
1060
    Extra packages to install.
1061
    Like for `late-command`, do not use ``preseed.pkgsel/include``.
1062 1063 1064
    If you want to install packages only for some specific <dist>, you can do
    it in ``late-command``, by testing ``$DIST`` and using
    ``apt-get install -y``.
1065

1066 1067 1068 1069 1070 1071 1072 1073
vm.run
    Boolean value that is `true` by default, to configure the VM for use with
    the `slapos.recipe.build:vm.run`_ recipe:

    - make sure that the `ssh` and `sudo` packages are installed
    - an SSH key is automatically created with ``ssh-keygen``, and it can be
      used to connect as `root`

1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087
ISO sections
~~~~~~~~~~~~

<arch>.iso
    Name of the section that provides the ISO image, for example by downloading
    it. This section must define 2 options: `location` is the folder
    containing the ISO, and `filename` is the file name of the ISO.

<arch>.kernel
    Path to kernel image inside the ISO.

<arch>.initrd
    Path to initrd image inside the ISO.

1088 1089 1090
User setup
~~~~~~~~~~

1091 1092
By default, there's no normal user created. Another rule is that a random
password is automatically generated if there is no password specified.
1093

1094
You have nothing to do if you only plan to use the VM with `vm.run`.
1095 1096 1097 1098 1099 1100

For more information about the ``passwd/*`` preseed values, you can look at
the ``user-setup-udeb`` package at
https://anonscm.debian.org/cgit/d-i/user-setup.git/tree/
and in particular the ``user-setup-ask`` and ``user-setup-apply`` scripts.

1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123
Example
~~~~~~~

::

  [vm-install-environment]
  # vm-run-environment refers to the section in common options
  PATH = ${file:location}/bin:${p7zip:location}/bin:${vm-run-environment:PATH}

  [vm-debian]
  recipe = slapos.recipe.build:vm.install-debian
  environment = vm-install-environment
  dists = debian-jessie debian-stretch
  size = 2Gi
  late-command =
  # rdnssd causes too much trouble with QEMU 2.7, because the latter acts as
  # a DNS proxy on both IPv4 and IPv6 without translating queries to what the
  # host supports.
    dpkg -P rdnssd
  debconf.debconf =
    debconf/frontend noninteractive
    debconf/priority critical
  # minimal size
1124
  preseed.apt-setup/enable-source-repositories = false
1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147
  preseed.recommends = false
  preseed.tasks =

  [debian-jessie]
  x86_64.iso = debian-amd64-netinst.iso
  x86_64.kernel = install.amd/vmlinuz
  x86_64.initrd = install.amd/initrd.gz

  [debian-stretch]
  <= debian-jessie
  x86_64.iso = debian-amd64-testing-netinst.iso

  [debian-amd64-netinst.iso]
  ...

slapos.recipe.build:vm.run
--------------------------

Execute shell commands inside a VM, in snapshot mode (the VM image is not
modified).

``${buildout:directory}`` is always mounted as `/mnt/buildout` inside the VM.

1148 1149 1150 1151 1152 1153
Mount points use the 9p file-system. Make sure that:

- QEMU is built with --enable-virtfs;
- the VM runs a kernel that is recent enough (Debian Squeeze kernel 2.6.32 is
  known to fail, and you'd have to use the one from squeeze-backports).

1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168
Options
~~~~~~~

location
    Folder where to store any produce file. Inside the guest, it is pointed to
    by the PARTDIR environment variable. It is also used as temporary storage
    for changes to the VM image.

vm
    Folder containing the VM images and the `ssh.key`` file. See the `location`
    option of the `vm.install-*` recipes.

dist
    VM image to use inside the `vm` folder.

1169 1170 1171
drives
    Extra drives. Each line is passed with -drive

1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184
commands
    List of <command> options, each one being a shell script to execute via
    SSH. They are processed in sequence. This is usually only required if you
    want to reboot the VM. Default: command

mount.<name>
    Extra mount point. The value is a host folder that is mounted as
    ``/mnt/<name>``.

stop-ssh
    Tell `reboot` function how to stop SSH (see Helpers_).
    Default: systemctl stop ssh

1185
user
1186 1187 1188 1189 1190 1191 1192 1193 1194
    Execute commands with this user. The value can be ``root``. By default,
    it is empty and it means that:

    - a ``slapos`` user is created with the same uid/gid than the user using
      this recipe on the host, which can help accessing mount points;
    - sudo must be installed and the created user is allowed to become root
      without password.

    In any case, SSH connects as root.
1195

1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233
wait-ssh
    Time to wait for (re)boot. The recipe fails if it can't connect to the SSH
    server after this number of seconds. Default: 60

Helpers
~~~~~~~

Before commands are executed, all `mount.<name>` are mounted
and a few helpers are set to make scripting easier.

set -e
    This is done before anything else, to make buildout abort if any untested
    command fails.

reboot
    Function to safely reboot the guest. The next command in `commands` will be
    executed once the SSH server is back.

map <host_path>
    Function to map a folder inside ``${buildout:directory}``.

PARTDIR
    Folder where to store any produced file. Inside the guest, it actually
    maps to `location` on the host. This is useful because you can't write
    ``PARTDIR=`map ${:location}``` if you don't explicitly set `location`.

Example
~~~~~~~

::

  [vm-run-base]
  # extends above example in common options
  vm = ${vm-debian:location}
  dist = debian-jessie

  [vm-debian]
  # extends above example in vm.install-debian
1234
  packages += build-essential devscripts equivs git
1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245

  [userhosts-repository]
  recipe = slapos.recipe.build:gitclone
  repository = https://lab.nexedi.com/nexedi/userhosts.git
  # we don't need a working directory on the host
  sparse-checkout = /.gitignore

  [build-userhosts-map]
  <= vm-run-base
  repository = `map ${userhosts-repository:location}`
  command =
1246 1247 1248
    git clone -s ${:repository} userhosts
    cd userhosts
    mk-build-deps -irs sudo -t 'apt-get -y'
1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269
    dpkg-buildpackage -uc -b -jauto
    cd ..
    mv *.changes *.deb $PARTDIR

  # Alternate way, which is required if [userhosts-repository] is extended
  # in such way that the repository is outside ${buildout:directory}.
  [build-userhosts-mount]
  <= build-userhosts-map
  mount.userhosts = ${userhosts-repository:location}
  repository = /mnt/userhosts

  [test-reboot]
  <= vm-run-base
  commands = hello world
  hello =
    uptime -s
    echo Hello ...
    reboot
  world =
    uptime -s
    echo ... world!