Commit 291678d6 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'ci-services-docs' into 'master'

Introduce CI documentation for services and languages

This introduces guide for PHP, a bunch of services (databases) and cleans up some docker references.

With this there are bunch of example projects in https://gitlab.com/groups/gitlab-examples.

/cc @JobV @axil 

See merge request !1917
parents ee0ab46d 1e7156ed
...@@ -24,9 +24,21 @@ ...@@ -24,9 +24,21 @@
- [Using Docker Images](ci/docker/using_docker_images.md) - [Using Docker Images](ci/docker/using_docker_images.md)
- [Using Docker Build](ci/docker/using_docker_build.md) - [Using Docker Build](ci/docker/using_docker_build.md)
- [Using Variables](ci/variables/README.md) - [Using Variables](ci/variables/README.md)
- [Using SSH keys](ci/ssh_keys/README.md)
- [User permissions](ci/permissions/README.md) - [User permissions](ci/permissions/README.md)
- [API](ci/api/README.md) - [API](ci/api/README.md)
### CI Languages
+ [Testing PHP](ci/languages/php.md)
### CI Services
+ [Using MySQL](ci/services/mysql.md)
+ [Using PostgreSQL](ci/services/postgres.md)
+ [Using Redis](ci/services/redis.md)
+ [Using Other Services](ci/docker/using_docker_images.md#how-to-use-other-images-as-services)
### CI Examples ### CI Examples
- [Test and deploy Ruby applications to Heroku](ci/examples/test-and-deploy-ruby-application-to-heroku.md) - [Test and deploy Ruby applications to Heroku](ci/examples/test-and-deploy-ruby-application-to-heroku.md)
......
...@@ -9,6 +9,18 @@ ...@@ -9,6 +9,18 @@
+ [Using Docker Images](docker/using_docker_images.md) + [Using Docker Images](docker/using_docker_images.md)
+ [Using Docker Build](docker/using_docker_build.md) + [Using Docker Build](docker/using_docker_build.md)
+ [Using Variables](variables/README.md) + [Using Variables](variables/README.md)
+ [Using SSH keys](ssh_keys/README.md)
### Languages
+ [Testing PHP](languages/php.md)
### Services
+ [Using MySQL](services/mysql.md)
+ [Using PostgreSQL](services/postgres.md)
+ [Using Redis](services/redis.md)
+ [Using Other Services](docker/using_docker_images.md#how-to-use-other-images-as-services)
### Examples ### Examples
......
This diff is collapsed.
### Languages
This is a list of languages you can test with GitLab CI. Each section has
comprehensive documentation and comes with a test repository hosted on
GitLab.com
+ [Testing PHP](php.md)
# Testing PHP projects
This guide covers basic building instructions for PHP projects.
There are covered two cases: testing using the Docker executor and testing
using the Shell executor.
## Test PHP projects using the Docker executor
While it is possible to test PHP apps on any system, this would require manual
configuration from the developer. To overcome this we will be using the
official [PHP docker image][php-hub] that can be found in Docker Hub.
This will allow us to test PHP projects against different versions of PHP.
However, not everything is plug 'n' play, you still need to onfigure some
things manually.
As with every build, you need to create a valid `.gitlab-ci.yml` describing the
build environment.
Let's first specify the PHP image that will be used for the build process
(you can read more about what an image means in the Runner's lingo reading
about [Using Docker images](../docker/using_docker_images.md#what-is-image)).
Start by adding the image to your `.gitlab-ci.yml`:
```yaml
image: php:5.6
```
The official images are great, but they lack a few useful tools for testing.
We need to first prepare the build environment. A way to overcome this is to
create a script which installs all prerequisites prior the actual testing is
done.
Let's create a `ci/docker_install.sh` file in the root directory of our
repository with the following content:
```bash
#!/bin/bash
# We need to install dependencies only for Docker
[[ ! -e /.dockerinit ]] && exit 0
set -xe
# Install git (the php image doesn't have it) which is required by composer
apt-get update -yqq
apt-get install git -yqq
# Install phpunit, the tool that we will use for testing
curl -o /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar
chmod +x /usr/local/bin/phpunit
# Install mysql driver
# Here you can install any other extension that you need
docker-php-ext-install pdo_mysql
```
You might wonder what `docker-php-ext-install` is. In short, it is a script
provided by the official php docker image that you can use to easilly install
extensions. For more information read the the documentation at
<https://hub.docker.com/_/php/>.
Now that we created the script that contains all prerequisites for our build
environment, let's add it in `.gitlab-ci.yml`:
```yaml
...
before_script:
- bash ci/docker_install.sh > /dev/null
...
```
Last step, run the actual tests using `phpunit`:
```yaml
...
test:app:
script:
- phpunit --configuration phpunit_myapp.xml
...
```
Finally, commit your files and push them to GitLab to see your build succeeding
(or failing).
The final `.gitlab-ci.yml` should look similar to this:
```yaml
# Select image from https://hub.docker.com/_/php/
image: php:5.6
before_script:
# Install dependencies
- ci/docker_install.sh > /dev/null
test:app:
script:
- phpunit --configuration phpunit_myapp.xml
```
### Test against different PHP versions in Docker builds
Testing against multiple versions of PHP is super easy. Just add another job
with a different docker image version and the runner will do the rest:
```yaml
before_script:
# Install dependencies
- ci/docker_install.sh > /dev/null
# We test PHP5.6
test:5.6:
image: php:5.6
script:
- phpunit --configuration phpunit_myapp.xml
# We test PHP7.0 (good luck with that)
test:7.0:
image: php:7.0
script:
- phpunit --configuration phpunit_myapp.xml
```
### Custom PHP configuration in Docker builds
There are times where you will need to customise your PHP environment by
putting your `.ini` file into `/usr/local/etc/php/conf.d/`. For that purpose
add a `before_script` action:
```yaml
before_script:
- cp my_php.ini /usr/local/etc/php/conf.d/test.ini
```
Of course, `my_php.ini` must be present in the root directory of your repository.
## Test PHP projects using the Shell executor
The shell executor runs your builds in a terminal session on your server.
Thus, in order to test your projects you first need to make sure that all
dependencies are installed.
For example, in a VM running Debian 8 we first update the cache, then we
install `phpunit` and `php5-mysql`:
```bash
sudo apt-get update -y
sudo apt-get install -y phpunit php5-mysql
```
Next, add the following snippet to your `.gitlab-ci.yml`:
```yaml
test:app:
script:
- phpunit --configuration phpunit_myapp.xml
```
Finally, push to GitLab and let the tests begin!
### Test against different PHP versions in Shell builds
The [phpenv][] project allows you to easily manage different versions of PHP
each with its own config. This is specially usefull when testing PHP projects
with the Shell executor.
You will have to install it on your build machine under the `gitlab-runner`
user following [the upstream installation guide][phpenv-installation].
Using phpenv also allows to easily configure the PHP environment with:
```
phpenv config-add my_config.ini
```
*__Important note:__ It seems `phpenv/phpenv`
[is abandoned](https://github.com/phpenv/phpenv/issues/57). There is a fork
at [madumlao/phpenv](https://github.com/madumlao/phpenv) that tries to bring
the project back to life. [CHH/phpenv](https://github.com/CHH/phpenv) also
seems like a good alternative. Picking any of the mentioned tools will work
with the basic phpenv commands. Guiding you to choose the right phpenv is out
of the scope of this tutorial.*
### Install custom extensions
Since this is a pretty bare installation of the PHP environment, you may need
some extensions that are not currently present on the build machine.
To install additional extensions simply execute:
```bash
pecl install <extension>
```
It's not advised to add this to `.gitlab-ci.yml`. You should execute this
command once, only to setup the build environment.
## Extend your tests
### Using atoum
Instead of PHPUnit, you can use any other tool to run unit tests. For example
you can use [atoum](https://github.com/atoum/atoum):
```yaml
before_script:
- wget http://downloads.atoum.org/nightly/mageekguy.atoum.phar
test:atoum:
script:
- php mageekguy.atoum.phar
```
### Using Composer
The majority of the PHP projects use Composer for managing their PHP packages.
In order to execute Composer before running your tests, simply add the
following in your `.gitlab-ci.yml`:
```yaml
...
# Composer stores all downloaded packages in the vendor/ directory.
# Do not use the following if the vendor/ directory is commited to
# your git repository.
cache:
paths:
- vendor/
before_script:
# Install composer dependencies
- curl -sS https://getcomposer.org/installer | php
- php composer.phar install
...
```
## Access private packages / dependencies
If your test suite needs to access a private repository, you need to configure
[the SSH keys](../ssh_keys/README.md) in order to be able to clone it.
## Use databases or other services
Most of the time you will need a running database in order for your tests to
run. If you are using the Docker executor you can leverage Docker's ability to
link to other containers. In GitLab Runner lingo, this can be achieved by
defining a `service`.
This functionality is covered in [the CI services](../services/README.md)
documentation.
## Testing things locally
With GitLab Runner 1.0 you can also test any changes locally. From your
terminal execute:
```bash
# Check using docker executor
gitlab-runner exec docker test:app
# Check using shell executor
gitlab-runner exec shell test:app
```
## Example project
We have set up an [Example PHP Project][php-example-repo] for your convenience
that runs on [GitLab.com](https://gitlab.com) using our publicly available
[shared runners](../runners/README.md).
Want to hack on it? Simply fork it, commit and push your changes. Within a few
moments the changes will be picked by a public runner and the build will begin.
[php-hub]: https://hub.docker.com/_/php/
[phpenv]: https://github.com/phpenv/phpenv
[phpenv-installation]: https://github.com/phpenv/phpenv#installation
[php-example-repo]: https://gitlab.com/gitlab-examples/php
## GitLab CI Services
GitLab CI uses the `services` keyword to define what docker containers should be
linked with your base image. Below is a list of examples you may use.
+ [Using MySQL](mysql.md)
+ [Using PostgreSQL](postgres.md)
+ [Using Redis](redis.md)
+ [Using Other Services](../docker/using_docker_images.md#how-to-use-other-images-as-services)
## GitLab CI Services
+ [Using MySQL](mysql.md)
+ [Using PostgreSQL](postgres.md)
+ [Using Redis](redis.md)
# Using MySQL
As many applications depend on MySQL as their database, you will eventually
need it in order for your tests to run. Below you are guided how to do this
with the Docker and Shell executors of GitLab Runner.
## Use MySQL with the Docker executor
If you are using [GitLab Runner](../runners/README.md) with the Docker executor
you basically have everything set up already.
First, in your `.gitlab-ci.yml` add:
```yaml
services:
- mysql:latest
variables:
# Configure mysql environment variables (https://hub.docker.com/_/mysql/)
MYSQL_DATABASE: el_duderino
MYSQL_ROOT_PASSWORD: mysql_strong_password
```
And then configure your application to use the database, for example:
```yaml
Host: mysql
User: root
Password: mysql_strong_password
Database: el_duderino
```
If you are wondering why we used `mysql` for the `Host`, read more at
[How is service linked to the build](../docker/using_docker_images.md#how-is-service-linked-to-the-build).
You can also use any other docker image available on [Docker Hub][hub-mysql].
For example, to use MySQL 5.5 the service becomes `mysql:5.5`.
The `mysql` image can accept some environment variables. For more details
check the documentation on [Docker Hub][hub-mysql].
## Use MySQL with the Shell executor
You can also use MySQL on manually configured servers that are using
GitLab Runner with the Shell executor.
First install the MySQL server:
```bash
sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev
```
Pick a MySQL root password (can be anything), and type it twice when asked.
*Note: As a security measure you can run `mysql_secure_installation` to
remove anonymous users, drop the test database and disable remote logins with
the root user.*
The next step is to create a user, so login to MySQL as root:
```bash
mysql -u root -p
```
Then create a user (in our case `runner`) which will be used by your
application. Change `$password` in the command below to a real strong password.
*Note: Do not type `mysql>`, this is part of the MySQL prompt.*
```bash
mysql> CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password';
```
Create the database:
```bash
mysql> CREATE DATABASE IF NOT EXISTS `el_duderino` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;
```
Grant the necessary permissions on the database:
```bash
mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `el_duderino`.* TO 'runner'@'localhost';
```
If all went well you can now quit the database session:
```bash
mysql> \q
```
Now, try to connect to the newly created database to check that everything is
in place:
```bash
mysql -u runner -p -D el_duderino
```
As a final step, configure your application to use the database, for example:
```bash
Host: localhost
User: runner
Password: $password
Database: el_duderino
```
## Example project
We have set up an [Example MySQL Project][mysql-example-repo] for your
convenience that runs on [GitLab.com](https://gitlab.com) using our publicly
available [shared runners](../runners/README.md).
Want to hack on it? Simply fork it, commit and push your changes. Within a few
moments the changes will be picked by a public runner and the build will begin.
[hub-mysql]: https://hub.docker.com/_/mysql/
[mysql-example-repo]: https://gitlab.com/gitlab-examples/mysql
# Using PostgreSQL
As many applications depend on PostgreSQL as their database, you will
eventually need it in order for your tests to run. Below you are guided how to
do this with the Docker and Shell executors of GitLab Runner.
## Use PostgreSQL with the Docker executor
If you are using [GitLab Runner](../runners/README.md) with the Docker executor
you basically have everything set up already.
First, in your `.gitlab-ci.yml` add:
```yaml
services:
- postgres:latest
variables:
POSTGRES_DB: nice_marmot
POSTGRES_USER: runner
POSTGRES_PASSWORD: ""
```
And then configure your application to use the database, for example:
```yaml
Host: postgres
User: runner
Password:
Database: nice_marmot
```
If you are wondering why we used `postgres` for the `Host`, read more at
[How is service linked to the build](../docker/using_docker_images.md#how-is-service-linked-to-the-build).
You can also use any other docker image available on [Docker Hub][hub-pg].
For example, to use PostgreSQL 9.3 the service becomes `postgres:9.3`.
The `postgres` image can accept some environment variables. For more details
check the documentation on [Docker Hub][hub-pg].
## Use PostgreSQL with the Shell executor
You can also use PostgreSQL on manually configured servers that are using
GitLab Runner with the Shell executor.
First install the PostgreSQL server:
```bash
sudo apt-get install -y postgresql postgresql-client libpq-dev
```
The next step is to create a user, so login to PostgreSQL:
```bash
sudo -u postgres psql -d template1
```
Then create a user (in our case `runner`) which will be used by your
application. Change `$password` in the command below to a real strong password.
*__Note:__ Do not type `template1=#`, this is part of the PostgreSQL prompt.*
```bash
template1=# CREATE USER runner WITH PASSWORD '$password' CREATEDB;
```
*__Note:__ Notice that we created the user with the privilege to be able to
create databases (`CREATEDB`). In the following steps we will create a database
explicitly for that user but having that privilege can be useful if in your
testing framework you have tools that drop and create databases.*
Create the database and grant all privileges on it for the user `runner`:
```bash
template1=# CREATE DATABASE nice_marmot OWNER runner;
```
If all went well you can now quit the database session:
```bash
template1=# \q
```
Now, try to connect to the newly created database with the user `runner` to
check that everything is in place.
```bash
psql -U runner -h localhost -d nice_marmot -W
```
*__Note:__ We are explicitly telling `psql` to connect to localhost in order
to use the md5 authentication. If you omit this step you will be denied access.*
Finally, configure your application to use the database, for example:
```yaml
Host: localhost
User: runner
Password: $password
Database: nice_marmot
```
## Example project
We have set up an [Example PostgreSQL Project][postgres-example-repo] for your
convenience that runs on [GitLab.com](https://gitlab.com) using our publicly
available [shared runners](../runners/README.md).
Want to hack on it? Simply fork it, commit and push your changes. Within a few
moments the changes will be picked by a public runner and the build will begin.
[hub-pg]: https://hub.docker.com/_/postgres/
[postgres-example-repo]: https://gitlab.com/gitlab-examples/postgres
# Using Redis
As many applications depend on Redis as their key-value store, you will
eventually need it in order for your tests to run. Below you are guided how to
do this with the Docker and Shell executors of GitLab Runner.
## Use Redis with the Docker executor
If you are using [GitLab Runner](../runners/README.md) with the Docker executor
you basically have everything set up already.
First, in your `.gitlab-ci.yml` add:
```yaml
services:
- redis:latest
```
Then you need to configure your application to use the Redis database, for
example:
```yaml
Host: redis
```
And that's it. Redis will now be available to be used within your testing
framework.
You can also use any other docker image available on [Docker Hub][hub-redis].
For example, to use Redis 2.8 the service becomes `redis:2.8`.
## Use Redis with the Shell executor
Redis can also be used on manually configured servers that are using GitLab
Runner with the Shell executor.
In your build machine install the Redis server:
```bash
sudo apt-get install redis-server
```
Verify that you can connect to the server with the `gitlab-runner` user:
```bash
# Try connecting the the Redis server
sudo -u gitlab-runner -H redis-cli
# Quit the session
127.0.0.1:6379> quit
```
Finally, configure your application to use the database, for example:
```yaml
Host: localhost
```
## Example project
We have set up an [Example Redis Project][redis-example-repo] for your convenience
that runs on [GitLab.com](https://gitlab.com) using our publicly available
[shared runners](../runners/README.md).
Want to hack on it? Simply fork it, commit and push your changes. Within a few
moments the changes will be picked by a public runner and the build will begin.
[hub-redis]: https://hub.docker.com/_/redis/
[redis-example-repo]: https://gitlab.com/gitlab-examples/redis
# Using SSH keys
GitLab currently doesn't have built-in support for managing SSH keys in a build
environment.
The SSH keys can be useful when:
1. You want to checkout internal submodules
2. You want to download private packages using your package manager (eg. bundler)
3. You want to deploy your application to eg. Heroku or your own server
4. You want to execute SSH commands from the build server to the remote server
5. You want to rsync files from your build server to the remote server
If anything of the above rings a bell, then you most likely need an SSH key.
## Inject keys in your build server
The most widely supported method is to inject an SSH key into your build
environment by extending your `.gitlab-ci.yml`.
This is the universal solution which works with any type of executor
(docker, shell, etc.).
### How it works
1. Create a new SSH key pair with [ssh-keygen][]
2. Add the private key as a **Secret Variable** to the project
3. Run the [ssh-agent][] during build to load the private key.
## SSH keys when using the Docker executor
You will first need to create an SSH key pair. For more information, follow the
instructions to [generate an SSH key](../ssh/README.md).
Then, create a new **Secret Variable** in your project settings on GitLab
following **Settings > Variables**. As **Key** add the name `SSH_PRIVATE_KEY`
and in the **Value** field paste the content of your _private_ key that you
created earlier.
Next you need to modify your `.gitlab-ci.yml` with a `before_script` action.
Add it to the top:
```
before_script:
# Install ssh-agent if not already installed, it is required by Docker.
# (change apt-get to yum if you use a CentOS-based image)
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY")
# For Docker builds disable host key checking. Be aware that by adding that
# you are suspectible to man-in-the-middle attacks.
# WARNING: Use this only with the Docker executor, if you use it with shell
# you will overwrite your user's SSH config.
- mkdir -p ~/.ssh
- '[[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config`
```
As a final step, add the _public_ key from the one you created earlier to the
services that you want to have an access to from within the build environment.
If you are accessing a private GitLab repository you need to add it as a
[deploy key](../ssh/README.md#deploy-keys).
That's it! You can now have access to private servers or repositories in your
build environment.
## SSH keys when using the Shell executor
If you are using the Shell executor and not Docker, it is easier to set up an
SSH key.
You can generate the SSH key from the machine that GitLab Runner is installed
on, and use that key for all projects that are run on this machine.
First, you need to login to the server that runs your builds.
Then from the terminal login as the `gitlab-runner` user and generate the SSH
key pair as described in the [SSH keys documentation](../ssh/README.md).
As a final step, add the _public_ key from the one you created earlier to the
services that you want to have an access to from within the build environment.
If you are accessing a private GitLab repository you need to add it as a
[deploy key](../ssh/README.md#deploy-keys).
Once done, try to login to the remote server in order to accept the fingerprint:
```bash
ssh <address-of-my-server>
```
For accessing repositories on GitLab.com, the `<address-of-my-server>` would be
`git@gitlab.com`.
## Example project
We have set up an [Example SSH Project][ssh-example-repo] for your convenience
that runs on [GitLab.com](https://gitlab.com) using our publicly available
[shared runners](../runners/README.md).
Want to hack on it? Simply fork it, commit and push your changes. Within a few
moments the changes will be picked by a public runner and the build will begin.
[ssh-keygen]: http://linux.die.net/man/1/ssh-keygen
[ssh-agent]: http://linux.die.net/man/1/ssh-agent
[ssh-example-repo]: https://gitlab.com/gitlab-examples/ssh-private-key/
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment