Commit f8f127cf authored by Thomas Gambier's avatar Thomas Gambier 🚴🏼

Build slapos-node using either python3 or python2

See merge request nexedi/slapos.package!161
parents 575b660e 9bc01524
Pipeline #22595 failed with stage
in 0 seconds
#!/usr/bin/python -S
#!/usr/bin/python3 -S
import glob, os, shutil, sys, tarfile, tempfile, zipfile
dist = "download-cache/dist"
tmp = tempfile.mkdtemp()
......
......@@ -19,7 +19,7 @@ Requirements
------------
- A computer
- Internet connection
- gcc, g++, make, patch, wget, python, chrpath, osc, devscripts
- gcc, g++, make, patch, wget, python3, chrpath, osc, devscripts
- An account on OBS
Documentation
......@@ -31,20 +31,20 @@ Documentation
Directory Organisation
----------------------
- prepare_slapos_build.sh : This is the script you have to run to prepare the packet.
You can choose the version you want to build in it.
- Makefile : makefile containing all the commands to prepare the packet.
- release_configuration.sh : script defining the version of slapos-node
package. Version is automaticallmy detected but you can decide to override
it.
- ./home:VIFIBnexedi:branches:home:VIFIBnexedi/SlapOS-Node : This is the directory for OBS.
It is the one used to generate unreleased package and testing it.
If you have/want to use another directory, update its path in prepare_slapos_build.sh
If you have/want to use another directory, update its path in release_configuration.sh
- templates/ : Contain templates to build the package.
- debian/ : contain configuration files for .deb packaging
- Makefile.in : Makefile to build slapos
- offline.sh.in : Prepare network-cache build building once slapos
- slapos.dsc.in : File needed by obs to build .deb package
- slapos.spec.in : configuration file for rpm packaging
- debian/ : contain configuration files for .deb packaging
- slapos-node/ : Template to build package
- Makefile : Makefile to build slapos and add files for package
- template/ : template files for package (cron.d, service)
- slapos/ : to build slapos. Contain buildout.cfg.in
......@@ -59,15 +59,15 @@ You need in particular an OBS directory (for example, the vifib test directory):
$ cd home:VIFIBnexedi:branches:home:VIFIBnexedi/SlapOS-Node
$ osc up
SlapOS is build using buildout and need an internet connection to fetch all elements.
On obs package are build on virtual machines without internet access.
prepare_slapos_build.sh will compile slapos on your computer and build a network-cache thanks to offline.sh script.
SlapOS is built using buildout and need an internet connection to fetch all elements.
On OBS, packages are built on virtual machines without internet access.
Makefile will compile slapos on your computer and build a network-cache thanks to prepare_download_cache.sh script.
At the end it remove slapos installation and only keep cache.
It makes a tarball out of it to be used by obs.
It then fetch packaging configuration from templates directory. Everything is then commit to obs thanks to osc.
It makes a tarball out of it to be used by OBS.
It then fetch packaging configuration from templates directory. Everything is then commit to OBS thanks to osc.
So to build package just run:
# bash prepare_slapos_build.sh
# make all
About cron file
---------------
......
Bridge configuration
====================
Before using slapformat commands, you must set up a bridge network interface
named ``slapbr0''. On that bridge, you must set up at least an IPv6 global
address, even if it is only a localhost one.
On Debian systems, this is commonly done in ``/etc/network/interfaces'' file.
Note that if you declare any interface as part of the bridge (when you want
your instances to be available not only on localhost), you should remove them
from the configuration file.
Here is an example of the bridge on eth0 device with static IPv4 address and
private IPv6 address:
auto slapbr0
iface slapbr0 inet static
address ADDRESS
netmask NETMASK
gateway GATEWAY
bridge_ports eth0
iface slapbr0 inet6 static
address fd00::1
netmask 64
You can have a look at bridge-utils-interfaces(5) and interfaces(5) manpages
for further details.
-- Arnaud Fontaine <arnaud.fontaine@nexedi.com>, Fri, 16 Nov 2011 10:24:51 +0900
......@@ -4,7 +4,7 @@ Section: net
Priority: optional
Build-Depends: debhelper,
chrpath,
python
python3 (>=3.7) | python
Homepage: https://slapos.nexedi.com
Package: slapos-node
......
#!/bin/bash
set -e
source release_configuration.sh
##########
......@@ -9,7 +11,7 @@ source release_configuration.sh
TARGET_DIRECTORY=/opt/slapos
BUILD_ROOT_DIRECTORY="$CURRENT_DIRECTORY/$SLAPOS_DIRECTORY/slapos/build"
BUILD_DIRECTORY=$BUILD_ROOT_DIRECTORY$TARGET_DIRECTORY
BUILDOUT_VERSION="2.5.2+slapos013"
BUILDOUT_VERSION="2.7.1+slapos019"
rm -rf $BUILD_ROOT_DIRECTORY
......@@ -17,63 +19,77 @@ echo "Preparing source tarball (recipe version: $RECIPE_VERSION)"
echo " Build Directory: $BUILD_DIRECTORY "
echo " Buildroot Directory: $BUILD_ROOT_DIRECTORY "
mkdir -p $BUILD_DIRECTORY/{eggs,extends-cache,download-cache,download-cache/dist,tmp-networkcached/eggs}
########################################################
# build the package once keeping every source in cache #
########################################################
set -e
mkdir -p $BUILD_DIRECTORY/{eggs,extends-cache,download-cache/dist}
cd $BUILD_DIRECTORY
# 1) boostrap with old version
echo "bootsrapping buildout"
sed "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../slapos.buildout.cfg.in > buildout.cfg
wget https://lab.nexedi.com/nexedi/slapos.buildout/raw/master/bootstrap/bootstrap.py
python3 -S bootstrap.py \
--setuptools-version 40.8.0 \
--setuptools-to-dir eggs
# 2) get newest version of zc.buildout and setuptools
# note that we can't directly do setuptools + zc.buildout +
# slapos.libnetworkcache because buildout would be relaunched in the middle
# without the "-S" option to python
echo "downloading good version of setuptools and zc.buildout"
sed -i '1s/$/ -S/' bin/buildout
sed -i "/def _satisfied(/s/\(\bsource=\)None/\11/" eggs/zc.buildout-*/zc/buildout/easy_install.py # no wheel
bin/buildout buildout:newest=true -v
ls download-cache/dist/*.whl && { echo "There shouldn't be any wheel in download-cache" ; exit 1 ; }
# 3) compile very simple buildout with networkcache
echo "Preparing networkcached zc.buildout"
NETWORKCACHED_DIRECTORY=$BUILD_DIRECTORY/tmp-networkcached
sed "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../networkcached.cfg.in > $NETWORKCACHED_DIRECTORY/buildout.cfg
cd $NETWORKCACHED_DIRECTORY
# Download bootstrap file
wget https://bootstrap.pypa.io/bootstrap-buildout.py -O bootstrap.py
(python2.7 -S bootstrap.py \
--buildout-version $BUILDOUT_VERSION \
--setuptools-version 44.1.1 \
--setuptools-to-dir eggs \
-f http://www.nexedi.org/static/packages/source/slapos.buildout/ && \
./bin/buildout -v)
# build locally everything with gcc
sed "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../buildout_with_gcc.cfg.in > $BUILD_DIRECTORY/buildout.cfg
# Build first time to get download-cache and extends-cache ready
cd $BUILD_DIRECTORY
sed "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../networkcached.cfg.in > buildout.cfg
sed -i '1s/$/ -S/' bin/buildout
bin/buildout buildout:newest=true -v
echo "$BUILD_ROOT_DIRECTORY" > $CURRENT_DIRECTORY/$SLAPOS_DIRECTORY/slapos/original_directory
# 4) build locally everything with gcc to get download-cache and extends-cache ready
echo "Launch the big buildout to compile everything"
sed "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../buildout_with_gcc.cfg.in > buildout.cfg
($NETWORKCACHED_DIRECTORY/bin/buildout bootstrap --buildout-version $BUILDOUT_VERSION \
--setuptools-to-dir eggs \
-f http://www.nexedi.org/static/packages/source/slapos.buildout/ && \
./bin/buildout -v) || (./bin/buildout -v || (echo "Failed to run buildout, exiting." && exit 1))
bin/buildout buildout:newest=true -v
###################################################
# remove all files from build keeping only caches #
###################################################
# remove all files from build keeping only caches
echo "Deleting unecessary files to reduce source tarball size"
# TODO: Figure out why there is no write permission even for
# the owner
chmod -R u+w .
cp -R eggs/slapos.rebootstrap* $BUILD_ROOT_DIRECTORY/..
rm -fv .installed.cfg environment.*
rm -rfv ./{downloads,parts,eggs,develop-eggs,bin,rebootstrap}
# Removing empty directories
find . -type d -empty -prune -exec rmdir '{}' ';'
mkdir -p $BUILD_DIRECTORY/eggs
$NETWORKCACHED_DIRECTORY/bin/buildout bootstrap --buildout-version $BUILDOUT_VERSION \
--setuptools-to-dir eggs \
-f http://www.nexedi.org/static/packages/source/ \
-f http://www.nexedi.org/static/packages/source/slapos.buildout/
cp -R $BUILD_ROOT_DIRECTORY/../slapos.rebootstrap* eggs
# Removing Python byte-compiled files (as it will be done upon
# package installation) and static libraries
find . -regextype posix-extended -type f \
-iregex '.*/*\.(py[co]|[l]?a|exe|bat)$$' -exec rm -fv '{}' ';'
#TODO remove git files
##################################
# prepare compilation inside OBS #
##################################
# we need the very first bootstrap script
cp $CURRENT_DIRECTORY/../re6st/bootstrap $BUILD_DIRECTORY
# we need the original directory to do a sed inside OBS
# TODO remove this and properly use extends-cache instead
echo "$BUILD_ROOT_DIRECTORY" > $CURRENT_DIRECTORY/$SLAPOS_DIRECTORY/slapos/original_directory
# in OBS build, don't force gcc build
sed "s/\%RECIPE_VERSION\%/$RECIPE_VERSION/g;s|\%PATCHES_DIRECTORY\%|$PATCHES_DIRECTORY|g;s|\%TARGET_DIRECTORY\%|$TARGET_DIRECTORY|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_ROOT_DIRECTORY|g;s|\%BUILD_DIRECTORY\%|$BUILD_DIRECTORY|g" $BUILD_ROOT_DIRECTORY/../buildout_without_gcc.cfg.in > $BUILD_DIRECTORY/buildout.cfg
......@@ -2,15 +2,13 @@
# -*- makefile -*-
build:
cd slapos; make build
make -C slapos build SLAPOS_BOOTSTRAP_SYSTEM_PYTHON=$(shell which python2.7 2> /dev/null || echo python3)
all: build
install: all
cd slapos; make install
make -C slapos install
mkdir -p $(DESTDIR)/usr/bin/
mkdir -p $(DESTDIR)/usr/sbin/
sed -i "s/\/opt\/slapos\/rebootstrap\/parts/\/opt\/slapos\/parts/g" $(DESTDIR)/opt/slapos/bin/*
cp $(DESTDIR)/opt/slapos/bin/slapos $(DESTDIR)/usr/bin/
mkdir -p $(DESTDIR)/etc/opt/slapos/
mkdir -p $(DESTDIR)/etc/firewalld/
......
......@@ -4,6 +4,8 @@ destdir = %BUILD_ROOT_DIRECTORY%
builddir = %BUILD_DIRECTORY%
extends =
%BUILD_ROOT_DIRECTORY%/../slapos_repository/component/slapos/obs.cfg
extends-cache = extends-cache
download-cache = download-cache
[gcc]
# force usage of gcc from slapos to have it in the cache
......
......@@ -4,4 +4,5 @@ destdir = %BUILD_ROOT_DIRECTORY%
builddir = %BUILD_DIRECTORY%
extends =
%BUILD_ROOT_DIRECTORY%/../slapos_repository/component/slapos/obs.cfg
download-cache = download-cache
......@@ -2,14 +2,25 @@
rootdir = %TARGET_DIRECTORY%
destdir = %BUILD_ROOT_DIRECTORY%
builddir = %BUILD_DIRECTORY%
extends =
%BUILD_ROOT_DIRECTORY%/../slapos_repository/component/slapos/obs.cfg
extensions =
extends-cache = extends-cache
download-cache = download-cache
parts =
networkcached
# Add location for modified non-official slapos.buildout
find-links +=
http://www.nexedi.org/static/packages/source/
http://www.nexedi.org/static/packages/source/slapos.buildout/
[networkcached]
recipe = zc.recipe.egg
eggs =
slapos.libnetworkcache
zc.buildout
[versions]
setuptools = 44.1.1
zc.buildout = 2.7.1+slapos019
slapos.libnetworkcache = 0.25
zc.recipe.egg = 2.0.3+slapos003
[buildout]
rootdir = %TARGET_DIRECTORY%
destdir = %BUILD_ROOT_DIRECTORY%
builddir = %BUILD_DIRECTORY%
extensions =
extends-cache = extends-cache
download-cache = download-cache
parts =
zc.buildout
# Add location for modified non-official slapos.buildout
find-links +=
http://www.nexedi.org/static/packages/source/
http://www.nexedi.org/static/packages/source/slapos.buildout/
[zc.buildout]
recipe = zc.recipe.egg
eggs =
zc.buildout
[versions]
setuptools = 44.1.1
zc.buildout = 2.7.1+slapos019
zc.recipe.egg = 2.0.3+slapos003
[slapos]
software_root = /opt/slapgrid
instance_root = /srv/slapgrid
master_url = http://127.0.0.1:5000
computer_id = abeautifulbutuniquename
pidfile = /var/opt/slapos/run/slapgrid.pid
logfile = /var/opt/slapos/log/slapgrid.log
supervisord-socket = /var/opt/slapos/run/slapgrid-supervisord.sock
#key_file = /etc/opt/slapos/ssl/computer.key
#cert_file = /etc/opt/slapos/ssl/computer.crt
#certificate_repository_path = /var/opt/slapos/lib/certificates
[slapformat]
computer_xml = /var/opt/slapos/lib/computer.xml
log_file = /var/opt/slapos/log/slapformat.log
partition_amount = 10
ipv4_local_network = 10.0.0.0/16
bridge_name = slapbr0
tap_base_name = slaptap
partition_base_name = slappart
user_base_name = slapuser
[slapproxy]
host = 127.0.0.1
port = 5000
database_uri = /var/opt/slapos/lib/proxy.db
......@@ -4,7 +4,6 @@
VERSION = %VERSION%
RECIPE_VERSION = %RECIPE_VERSION%
PY = $(PYTHON)
INSTALL_DIRECTORY = $(DESTDIR)/opt/slapos
TARGET_DIRECTORY = /opt/slapos
BUILD_ROOT_DIRECTORY := $(shell pwd)/build
......@@ -22,9 +21,13 @@ build-stamp:
xargs sed -i 's#$(ORIGINAL_DIRECTORY)#$(BUILD_ROOT_DIRECTORY)#g' || \
echo "No path to fix."
@echo "Bootstrapping buildout"
cd $(BUILD_DIRECTORY) && \
$(SLAPOS_BOOTSTRAP_SYSTEM_PYTHON) ./bootstrap
@echo 'Preparing source tarball (recipe version: $(RECIPE_VERSION))'
cd $(BUILD_DIRECTORY) && \
$(PY) ./bin/buildout -v
$(SLAPOS_BOOTSTRAP_SYSTEM_PYTHON) ./bin/buildout -v
@touch build-stamp
......
# Maintainer: Rafael Monnerat <rafael@nexedi.com>
# Maintainer: Thomas Gambier <thomas.gambier@nexedi.com>
pkgname=slapos-node
pkgver=%VERSION%+%RECIPE_VERSION%+%RELEASE%
pkgdesc="Slapos node"
......@@ -6,15 +6,15 @@ pkgrel=1
arch=('x86_64' 'i686')
url="http://www.slapos.org"
license=('GPL')
depends=(python2 python2-distribute iptables iproute2)
makedepends=(patch wget chrpath inetutils)
depends=(iptables iproute2)
makedepends=(patch wget chrpath inetutils python3)
install='slapos-node.install'
source=(${pkgname}_${pkgver}.tar.gz)
md5sums=(%SOURCEMD5%)
build() {
cd "${srcdir}/${pkgname}_${pkgver}"
make PYTHON=python2
make
}
package() {
......
......@@ -3,10 +3,10 @@ Source: slapos-node
Binary: slapos-node
Architecture: any
Version: %VERSION%+%RECIPE_VERSION%+%RELEASE%-1
Maintainer: Rafael Monnerat <rafael@nexedi.com>
Homepage: http://community.slapos.org
Maintainer: Thomas Gambier <thomas.gambier@nexedi.com>
Homepage: http://slapos.nexedi.com
Standards-Version: 3.9.2
Build-Depends: debhelper (>= 4.1.16), chrpath, po-debconf, wget, python, python-dev, libssl-dev
Build-Depends: debhelper (>= 4.1.16), chrpath, po-debconf, wget, python3 (>= 3.7) | python, python3-dev (>= 3.7) | python-dev, libssl-dev
Files:
1c2803e4693c74042985d753c9c3ba53 73736102 slapos-node_%VERSION%+%RECIPE_VERSION%+%RELEASE%.orig.tar.gz
1c2803e4693c74042985d753c9c3ba53 6102 slapos-node_%VERSION%+%RECIPE_VERSION%+%RELEASE%-1.diff.tar.gz
......@@ -44,11 +44,22 @@ Group: Application/Network
Source0: slapos-node_%{unmangled_version}+%{slapversion}+%{release_number}.tar.gz
URL: http://community.slapos.org/
Vendor: Vifib
Packager: Rafael Monnerat <rafael@nexedi.com>
BuildRequires: gcc-c++, make, patch, wget, python2, python-devel, chrpath, openssl-devel
Requires: bridge-utils, python2, gcc-c++, make, patch, wget, lm_sensors
Packager: Thomas Gambier <thomas.gambier@nexedi.com>
BuildRequires: gcc-c++
BuildRequires: make
BuildRequires: patch
BuildRequires: wget
BuildRequires: chrpath
BuildRequires: openssl-devel
%if 0%{?centos_version} <= 700 || 0%{?rhel_version} <= 700 || 0%{?suse_version} < 1300 || 0%{?sle_version} < 130000
BuildRequires: python
BuildRequires: python-devel
%else
BuildRequires: python3
BuildRequires: python3-devel
%endif
Requires: gcc-c++, make, patch, wget, lm_sensors
AutoReqProv: no
......@@ -77,7 +88,7 @@ zcat $RPM_SOURCE_DIR/slapos-node_%{unmangled_version}+%{slapversion}+%{release_n
%build
cd $RPM_BUILD_DIR/slapos-node_%{unmangled_version}+%{slapversion}+%{release_number}
make PYTHON=python2
make
%install
cd $RPM_BUILD_DIR/slapos-node_%{unmangled_version}+%{slapversion}+%{release_number}/
......@@ -89,14 +100,6 @@ cp $RPM_BUILD_DIR/slapos-node_%{unmangled_version}+%{slapversion}+%{release_numb
mkdir -p $RPM_BUILD_ROOT/etc/systemd/system/
mkdir -p $RPM_BUILD_ROOT/opt/slapos/log
rm -rf $RPM_BUILD_ROOT/opt/slapos/eggs/setuptools-*.egg/setuptools/tests/test_easy_install.py
rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/test_httpservers.py
rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/badsyntax_nocaret.py
rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/badsyntax_future*.py
rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/bad_*.py
rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/lib2to3/tests/data/py3_test_grammar.py
rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/test_grammar.py
rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/test_compile.py
rm -rf $RPM_BUILD_ROOT/opt/slapos/rebootstrap/parts/python2.7/lib/python2.7/test/_mock_backport.py
%files
/opt/slapos
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment