Commit e697eb57 authored by Ophélie Gagnard's avatar Ophélie Gagnard

obs/generic: Initial commit.

This provides a tool to help package a software from buildout using OBS.
parent 62600746
all: no_obs obs
clean: clean_no_obs clean_obs
no_obs: build_tree template_stage bootstrap_buildout
clean_no_obs: clean_build_tree clean_template_stage clean_bootstrap_buildout
build_tree:
build-scripts/build_tree.sh
clean_build_tree:
build-scripts/clean_build_tree.sh
template_stage:
build-scripts/template_stage.sh
clean_template_stage:
build-scripts/clean_template_stage.sh
bootstrap_buildout:
build-scripts/bootstrap_buildout.sh
clean_bootstrap_buildout:
build-scripts/clean_bootstrap_buildout.sh
obs:
build-scripts/obs.sh
clean_obs:
build-scripts/clean_obs.sh
.PHONY: all clean no_obs clean_no_obs build_tree clean_build_tree template_stage clean_template_stage bootstrap_buildout clean_bootstrap_buildout obs clean_obs
# README
**WARNING**: This project is a **work in progress** (WIP)!
## What is it
### In short
Here we turn a *software release* (SR) or a *buildout.cfg* into a *Linux package* thanks to *OBS*! It is not that simple though and you will have to read a bit on how it works before you can get your package.
### Workflow (short version)
1. Set you OBS repository.
2. Adapt *build-scripts/configure_release.sh*
3. Run ```make```
### How?
#### 1. OBS Repository
Sorry but first you will have to read this
>https://en.opensuse.org/openSUSE:Build_Service_Tutorial
and follow the "command-line" instructions. So you will have to install this (the *osc* command)
>https://en.opensuse.org/openSUSE:OSC
Advice: to install *osc*, if you are working at Nexedi or whatever affiliated company, you should probably go to
>https://github.com/openSUSE/osc
and follow the installation instructions. They are slightly broken... You may have to install some dependencies:
```
sudo apt install git pip
pip install distutils.core
```
and then to execute the instructions modified as follows:
```
python3 setup.py build
sudo python3 setup.py install
ln -s `pwd`/osc-wrapper.py /usr/bin/osc
# add "3" at the end of the first line (python -> python3)
sed -i "1 s/$/3/" osc-wrapper.py
```
You should checkout (```osc co <home-project> <package>```) in the root directory of this project. So after that, along with *Makefile* and *README.md*, you should have a directory *home-project* with a directory *package* inside.
Don't cheat! If you don't create them with ```osc``` it won't work...
### 2. Adapt *build-scripts/configure_release.sh*
[TODO: describe the needed adaptations]
### 3. Run ```make```
Simply run ```make```. More explanations at section [TODO: section]
Good luck!
Some advices though: [TODO: give some tips to get the logs, etc.]
[TODO: Reorganize the rest]
### Tarballs
The *\<tarball\>* directory to be archived and sent to OBS is to be prepared in tarballs/*\<tarball\>*.
### Templates
The *templates/* directory contains the templates used during first local run of buildout and to prepare the templates to be sent to OBS.
### Distribution specifics
The *distribution-specifics/* directory contains the files needed to build a package in a given distribution. Currently only Debian is supported and the files are in the minimal possible stage. Do not rely solely on them for a serious build.
### Packaging with OBS
Currently, only Debian is supported.
The OBS directory is prepared with obs.sh and the temporary files created for it are cleaned with clean_obs.sh. The scripts assume you already checked out your OBS project.
#### Debian
The *debian/* directory provides only the bare minimum for a package to be built. For more support you will have to add it yourself.
### Complete process
#### Workflow preparation workflow
``` # block of code
make
```
or
```
./build_tree.sh
./tempate_stage.sh
./bootstrap_buildout.sh
./obs.sh
```
#### Clean workflow
```
make clean
```
or
```
./clean_build_tree.sh
./clean_template_stage.sh
./clean_bootstram_buildout.sh
./clean_obs.sh
```
------------------
TMP
------------------
This package aims at minimizing efforts when turning a simple set of buildout.cfg files into something OBS-compatible in order to build a package. It provides a set of scripts and some additional files to ease the process.
The main goal is to prepare a tarball usable with the make/make install workflow. Then it is sent to OBS along with some *.spec* files. *.spec* files are used to actually package the tarball and are often specific to the distribution one wants to build a package for.
#!/bin/bash
set -e
source build-scripts/configuration_information.sh
cd $INITIAL_DIR
# Download the bootstrap script
mkdir -p $RUN_BUILDOUT_DIR
cd $RUN_BUILDOUT_DIR
wget https://bootstrap.pypa.io/bootstrap-buildout.py
cd $INITIAL_DIR
# Create a build/bin/buildout (bootstraping) and run it (actual compilation).
# Note: it creates a lot of things in build/eggs/ and uses software_release/ at some point
mkdir -p $RUN_BUILDOUT_DIR
cd $RUN_BUILDOUT_DIR
# should be with gcc here and without in OBS
cp $TEMPLATE_DIR/tmp/buildout_without_gcc.cfg buildout.cfg
(python2.7 -S bootstrap-buildout.py --buildout-version 2.7.1+slapos016 --setuptools-version 44.1.1 --setuptools-to-dir eggs -f http://www.nexedi.org/static/packages/source/slapos.buildout/ && ./bin/buildout -v)
#cp $TEMPLATE_DIR/tmp/buildout_with_gcc.cfg buildout.cfg
#!/bin/bash
set -e
source build-scripts/configuration_information.sh
cd $INITIAL_DIR
mkdir -p $TARBALL_DIR
# Re-downloading the software release to pack
#cp -r /home/test/other-projects/model/1wip_simplify-fluentbit_my-slapos .
cp -r /home/test/other-projects/new_slapos.package/sid.fluentbit.my-slapos $TARBALL_DIR
#mv 1wip_simplify-fluentbit_my-slapos software_release
mv $TARBALL_DIR/sid.fluentbit.my-slapos $TARBALL_DIR/software_release
cd $INITIAL_DIR
# Peparing the build directories
mkdir -p $RUN_BUILDOUT_DIR/{eggs,extends-cache,download-cache/dist}
#!/bin/bash
set -e
source build-scripts/configuration_information.sh
cd $INITIAL_DIR
# Remove the bootstrap script and associated files
rm -f $RUN_BUILDOUT_DIR/{bootstrap.py,bootstrap-buildout.py,buildout.cfg}
cd $INITIAL_DIR
# Clean the material created by the bootstrap script
rm -rf $RUN_BUILDOUT_DIR/{bin/,egg/}
#!/bin/bash
set -e
source build-scripts/configuration_information.sh
cd $INITIAL_DIR
rm -rf $TARBALL_DIR
exit # TODO: separate the tree building from the software release retrieving
cd $INITIAL_DIR
# Clean the software release to pack
rm -rf $TARBALL_DIR/software_release
cd $INITIAL_DIR
# Clean the build tree
rm -rf $RUN_BUILDOUT_DIR/{eggs,extends-cache,download-cache}
#!/bin/bash
set -e
source build-scripts/configuration_information.sh
TMP_DIR=$INITIAL_DIR/tmp/
TMP_DIR=`realpath -m $TMP_DIR`
rm -rf $TMP_DIR
#!/bin/bash
set -e
source build-scripts/configuration_information.sh
cd $INITIAL_DIR
# Clean temporary directory for templates
rm -rf $TEMPLATE_DIR/tmp/
####################################################
# Use on script as source release_configuration.sh
####################################################
INITIAL_DIR="$(pwd)"/
# RELEASE INFORMATION
# Modify the following variables accordingly
VERSION=1
#RECIPE_VERSION=1
RELEASE=1
SOFTWARE_NAME=mca
COMPOUND_VERSION=${VERSION}-${RELEASE}
VERSION_NAME=${SOFTWARE_NAME}_${COMPOUND_VERSION}
ARCHIVE_EXT=.tar.gz
# INSTALL INFORMATION
# TARGET_DIR is only used in the templates via sed regexps
TARGET_DIR=/opt/$VERSION_NAME
# OBS AND DISTRIBUTIONS INFORMATION
OBS_DIR=$INITIAL_DIR/home:oph.nxd/$SOFTWARE_NAME/
DIST_DIR=$INITIAL_DIR/distribution-specifics/$SOFTWARE_NAME/
TARBALL_DIR=$INITIAL_DIR/tarballs/$VERSION_NAME
# BUILD INFORMATION
# Modify the following variables accordingly
#SR_PATH=$TARBALL_DIR/software_release/component/$SOFTWARE_NAME/buildout.cfg
SR_PATH=$TARBALL_DIR/software_release/software/$SOFTWARE_NAME/software.cfg
TEMPLATE_DIR=$INITIAL_DIR/templates
# BUILD_DIR was formerly BUILD_ROOT_DIRECTORY
BUILD_DIR=$TARBALL_DIR/build/
# RUN_BUILDOUT_DIR is where the parts/ directory will be ; formerly BUILD_DIRECTORY
RUN_BUILDOUT_DIR=$BUILD_DIR/$TARGET_DIR
# Path normalization
INITIAL_DIR=`realpath -m $INITIAL_DIR`
TARGET_DIR=`realpath -m $TARGET_DIR`
OBS_DIR=`realpath -m $OBS_DIR`
DIST_DIR=`realpath -m $DIST_DIR`
TARBALL_DIR=`realpath -m $TARBALL_DIR`
SR_PATH=`realpath -m $SR_PATH`
TEMPLATE_DIR=`realpath -m $TEMPLATE_DIR`
BUILD_DIR=`realpath -m $BUILD_DIR`
RUN_BUILDOUT_DIR=`realpath -m $RUN_BUILDOUT_DIR`
# Regular expressions for templates
# versions (not used at the moment)
VERSION_REGEX="s|\%RECIPE_VERSION\%|$RECIPE_VERSION|g;s|\%VERSION\%|$VERSION|g;s|\%RELEASE\%|$RELEASE|g;s|\%COMPOUND_VERSION\%|$COMPOUND_VERSION|g"
# Note: %PATCHES_DIRECTORY% not supported yet
# directories (supporting new macros)
DIR_REGEX="s|\%TARGET_DIR\%|$TARGET_DIR|g;s|\%BUILD_DIR\%|$BUILD_DIR|g;s|\%RUN_BUILDOUT_DIR\%|$RUN_BUILDOUT_DIR|g"
PATH_REGEX="s|\%SR_PATH\%|$SR_PATH|g"
# directories (supporting legacy macros, not used anymore)
#OLD_DIR_REGEX="s|\%TARGET_DIRECTORY\%|$TARGET_DIR|g;s|\%BUILD_ROOT_DIRECTORY\%|$BUILD_DIR|g;s|\%BUILD_DIRECTORY\%|$RUN_BUILDOUT_DIR|g"
# concatenate all regex using ; (quoted, not to end the command)
ALL_REGEX=$VERSION_REGEX";"$DIR_REGEX";"$PATH_REGEX #";"$OLD_DIR_REGEX
#!/bin/bash
set -e
source build-scripts/configuration_information.sh
# temporary directory for the files needed by OBS
TMP_DIR=$INITIAL_DIR/tmp/
TMP_DIR=`realpath -m $TMP_DIR`
echo TMP_DIR = $TMP_DIR
# copy the Makefile at the root of the tarball
cp $TEMPLATE_DIR/tmp/Makefile $TARBALL_DIR/Makefile
# save the local TARBALL_DIR to replace it with the TARBALL_DIR of OBS' VM
echo $TARBALL_DIR > $TARBALL_DIR/cache_creation_build_directory
# clean the parts directory
rm -rf $RUN_BUILDOUT_DIR/{.installed.cfg,parts/}
## prepare the files for OBS
mkdir -p $TMP_DIR
# -C option allows to give tar an absolute path without archiving the directory from / (i.e. home/user/[...])
tar czf $TMP_DIR/${VERSION_NAME}${ARCHIVE_EXT} -C $INITIAL_DIR/tarballs/ $VERSION_NAME/
tar czf $TMP_DIR/debian.tar.gz -C $DIST_DIR/ debian/
cp $DIST_DIR/*.dsc $TMP_DIR/
# move the files for OBS
cp $TMP_DIR/${VERSION_NAME}${ARCHIVE_EXT} $OBS_DIR
cp $TMP_DIR/debian.tar.gz $OBS_DIR
cp $TMP_DIR/*.dsc $OBS_DIR
cd $OBS_DIR
osc commit
#!/bin/bash
set -e
source build-scripts/configuration_information.sh
cd $INITIAL_DIR
# Prepare the templates in $TEMPLATE_DIR/tmp/
mkdir -p $TEMPLATE_DIR/tmp/
sed $ALL_REGEX $TEMPLATE_DIR/buildout_with_gcc.cfg.in > $TEMPLATE_DIR/tmp/buildout_with_gcc.cfg
sed $ALL_REGEX $TEMPLATE_DIR/buildout_without_gcc.cfg.in > $TEMPLATE_DIR/tmp/buildout_without_gcc.cfg
sed $ALL_REGEX $TEMPLATE_DIR/Makefile.in > $TEMPLATE_DIR/tmp/Makefile
bash
python2.7
wget
# required to checkout with osc (command-line for OBS)
python3-m2crypto
fluent-bit (1-1) UNRELEASED; urgency=medium
* Initial release. (Closes: #XXXXXX)
-- test <test@debian> Tue, 25 Jan 2022 18:27:17 +0100
Source: fluent-bit
Maintainer: Ophelie Gagnard <ophelie.gagnard@nexedi.com>
Section: net
Priority: optional
Build-Depends: debhelper,
chrpath,
python
Homepage: https://slapos.nexedi.com
Package: fluent-bit
Architecture: any
Depends: ${misc:Depends},
${shlibs:Depends},
ucf,
uml-utilities,
# slapgrid-sr needed by most recipes
libc6-dev | libc-dev,
gcc | c-compiler,
g++ | c++-compiler,
make,
patch,
# slapos node collect
lm-sensors,
# devperm manager plugin wants to use lsblk
util-linux
Conflicts:
Description: fluent-bit packaging test
/opt/fluent-bit_1-1/bin/
/opt/fluent-bit_1-1/lib/
/opt/fluent-bit_1-1/etc/
#!/usr/bin/make -f
#export DH_VERBOSE=1
PACKAGE = $(shell dh_listpackages)
TMP = $(CURDIR)/debian/$(PACKAGE)
%:
dh $@
override_dh_installdebconf:
override_dh_makeshlibs:
dh_makeshlibs -n
override_dh_shlibdeps:
dh_shlibdeps -- -x$(PACKAGE)
rm -f $(TMP)/DEBIAN/shlibs
Format: 3.0 (native)
Source: fluent-bit
Architecture: any
Version: 1
Build-Depends: debhelper (>= 4.1.16), chrpath, po-debconf, wget, python (>= 2.7), python-dev, libssl-dev
lunzip (1-1) UNRELEASED; urgency=medium
* Initial release. (Closes: #XXXXXX)
-- test <test@debian> Tue, 25 Jan 2022 18:27:17 +0100
Source: lunzip
Maintainer: Ophelie Gagnard <ophelie.gagnard@nexedi.com>
Section: net
Priority: optional
Build-Depends: debhelper,
chrpath,
python
Homepage: https://slapos.nexedi.com
Package: lunzip
Architecture: any
Depends: ${misc:Depends},
${shlibs:Depends},
ucf,
uml-utilities,
# slapgrid-sr needed by most recipes
libc6-dev | libc-dev,
gcc | c-compiler,
g++ | c++-compiler,
make,
patch,
# slapos node collect
lm-sensors,
# devperm manager plugin wants to use lsblk
util-linux
Conflicts:
Description: lunzip packaging test
/opt/lunzip_1-1/bin/
/opt/lunzip_1-1/lib/
/opt/lunzip_1-1/etc/
#!/usr/bin/make -f
#export DH_VERBOSE=1
PACKAGE = $(shell dh_listpackages)
TMP = $(CURDIR)/debian/$(PACKAGE)
%:
dh $@
override_dh_installdebconf:
override_dh_makeshlibs:
dh_makeshlibs -n
override_dh_shlibdeps:
dh_shlibdeps -- -x$(PACKAGE)
rm -f $(TMP)/DEBIAN/shlibs
Format: 3.0 (native)
Source: lunzip
Architecture: any
Version: 1
Build-Depends: debhelper (>= 4.1.16), chrpath, po-debconf, wget, python (>= 2.7), python-dev, libssl-dev
mca (1-1) UNRELEASED; urgency=medium
* Initial release. (Closes: #XXXXXX)
-- test <test@debian> Tue, 25 Jan 2022 18:27:17 +0100
Source: mca
Maintainer: Ophelie Gagnard <ophelie.gagnard@nexedi.com>
Section: net
Priority: optional
Build-Depends: debhelper,
chrpath,
python
Package: mca
Architecture: any
Depends: ${misc:Depends},
${shlibs:Depends},
ucf,
uml-utilities,
# slapgrid-sr needed by most recipes
libc6-dev | libc-dev,
gcc | c-compiler,
g++ | c++-compiler,
make,
patch,
# devperm manager plugin wants to use lsblk
util-linux
Conflicts:
Description: mca packaging test
/opt/mca_1-1/bin/
/opt/mca_1-1/lib/
/opt/mca_1-1/etc/
#!/usr/bin/make -f
#export DH_VERBOSE=1
PACKAGE = $(shell dh_listpackages)
TMP = $(CURDIR)/debian/$(PACKAGE)
%:
dh $@
override_dh_installdebconf:
override_dh_makeshlibs:
dh_makeshlibs -n
override_dh_shlibdeps:
dh_shlibdeps -- -x$(PACKAGE)
rm -f $(TMP)/DEBIAN/shlibs
Format: 3.0 (native)
Source: mca
Architecture: any
Version: 1
Build-Depends: debhelper (>= 4.1.16), chrpath, po-debconf, wget, python (>= 2.7)
PY = $(PYTHON)
#INSTALL_DIR = $(DESTDIR)/opt/test
#TARGET_DIR = /opt/slapos
# This templates are replaced by build-scripts/template_stage.sh
# according to the values and regular expressions defined in
# build-scripts/configuration_information.sh
# The variable TARGET_DIR is pretty useless atm
TARGET_DIR = %TARGET_DIR%
TARBALL_DIR = $(shell pwd)
BUILD_DIR = $(TARBALL_DIR)/build
INSTALL_DIR = $(DESTDIR)$(TARGET_DIR)
RUN_BUILDOUT_DIR = $(BUILD_DIR)$(TARGET_DIR)
#PATCHES_DIR := $(shell pwd)/patches
# get the path of the BUILD_DIR of the first build (performed to prepare the cache for OBS)
OLD_TARBALL_DIR := $(shell cat cache_creation_build_directory)
#Use to get path of buildout correct
#ORIGINAL_DIRECTORY := $(shell cat ./original_directory)
all: build
build: build-stamp
build-stamp:
@echo "Fixing buildout path to $(TARBALL_DIR) rather than $(OLD_TARBALL_DIR) for buildout"
################################################################################
# grep -rIl '$(OLD_DIRECTORY)' $(BUILD_DIR) 2> /dev/null | \
# xargs sed -i 's#$(OLD_DIRECTORY)#$(BUILD_DIR)#g' || \
# echo "No path to fix."
################################################################################
ls -l $(RUN_BUILDOUT_DIR) # DEBUG
ls -l $(RUN_BUILDOUT_DIR)/bin/ # DEBUG
cd $(RUN_BUILDOUT_DIR); sed -i 's#$(OLD_TARBALL_DIR)#$(TARBALL_DIR)#g' buildout.cfg bin/*
echo RUN_BUILDOUT_DIR = $(RUN_BUILDOUT_DIR) \; PY = $(PY)
cd $(RUN_BUILDOUT_DIR) && \
$(PY) ./bin/buildout -v
@touch build-stamp
clean:
# TODO: implement a proper cleaning
#rm -rf $(BUILD_DIR)
rm -f *-stamp
install: all
#cd slapos; make install
mkdir -p $(INSTALL_DIR)/bin
mkdir -p $(INSTALL_DIR)/sbin/
mkdir -p $(INSTALL_DIR)/etc/
mkdir -p $(INSTALL_DIR)/lib/
mkdir -p $(INSTALL_DIR)/share/
#cp `find $(RUN_BUILDOUT_DIR) | grep -e "^$(RUN_BUILDOUT_DIR)/[^/][^/]*/bin/*` $(DESTDIR)/usr/bin/
-cp -r $(RUN_BUILDOUT_DIR)/parts/*/bin/* $(INSTALL_DIR)/bin/
-cp -r $(RUN_BUILDOUT_DIR)/parts/*/sbin/* $(INSTALL_DIR)/sbin/
-cp -r $(RUN_BUILDOUT_DIR)/parts/*/etc/* $(INSTALL_DIR)/etc/
-cp -r $(RUN_BUILDOUT_DIR)/parts/*/lib/* $(INSTALL_DIR)/lib/
-cp -r $(RUN_BUILDOUT_DIR)/parts/*/share/* $(INSTALL_DIR)/share/
ls -Rl $(INSTALL_DIR)/bin/ # DEBUG
.PHONY: build all clean install
[buildout]
rootdir = %TARGET_DIR%
destdir = %BUILD_DIR%
builddir = %RUN_BUILDOUT_DIR%
extends =
%SR_PATH%
# %BUILD_DIR%/../software_release/software/fluent-bit/software.cfg
# %TARBALL_DIR%/software_release/software/fluent-bit/software.cfg
# %TARBALL_DIR%/software_release/component/fluent-bit/buildout.cfg
# %BUILD_DIR%/../software_release/component/fluent-bit/buildout.cfg
versions = versions
extends-cache = extends-cache
download-cache = download-cache
[gcc]
# force usage of gcc from slapos to have it in the cache
max_version = 0
[versions]
setuptools = 44.1.1
# Use SlapOS patched zc.buildout
zc.buildout = 2.7.1+slapos016
# Use SlapOS patched zc.recipe.egg (zc.recipe.egg 2.x is for Buildout 2)
zc.recipe.egg = 2.0.3+slapos003
[buildout]
rootdir = %TARGET_DIR%
destdir = %BUILD_DIR%
builddir = %RUN_BUILDOUT_DIR%
extends =
%SR_PATH%
# %BUILD_DIR%/../software_release/software/fluent-bit/software.cfg
# %TARBALL_DIR%/software_release/software/fluent-bit/software.cfg
# %TARBALL_DIR%/software_release/component/fluent-bit/buildout.cfg
# %BUILD_DIR%/../software_release/component/fluent-bit/buildout.cfg
versions = versions
extends-cache = extends-cache
download-cache = download-cache
[versions]
setuptools = 44.1.1
# Use SlapOS patched zc.buildout
zc.buildout = 2.7.1+slapos016
# Use SlapOS patched zc.recipe.egg (zc.recipe.egg 2.x is for Buildout 2)
zc.recipe.egg = 2.0.3+slapos003
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment