Merge branch 'ci-services-docs' into 'master'

Introduce CI documentation for services and languages

This introduces guide for PHP, a bunch of services (databases) and cleans up some docker references.

With this there are bunch of example projects in https://gitlab.com/groups/gitlab-examples.

/cc @JobV @axil 

See merge request !1917

doc/README.md

@@ -24,9 +24,21 @@
 - [Using Docker Images](ci/docker/using_docker_images.md)
 - [Using Docker Build](ci/docker/using_docker_build.md)
 - [Using Variables](ci/variables/README.md)
+- [Using SSH keys](ci/ssh_keys/README.md)
 - [User permissions](ci/permissions/README.md)
 - [API](ci/api/README.md)
 
+### CI Languages
+
++ [Testing PHP](ci/languages/php.md)
+
+### CI Services
+
++ [Using MySQL](ci/services/mysql.md)
++ [Using PostgreSQL](ci/services/postgres.md)
++ [Using Redis](ci/services/redis.md)
++ [Using Other Services](ci/docker/using_docker_images.md#how-to-use-other-images-as-services)
+
 ### CI Examples
 
 - [Test and deploy Ruby applications to Heroku](ci/examples/test-and-deploy-ruby-application-to-heroku.md)

doc/ci/README.md

@@ -9,6 +9,18 @@
 + [Using Docker Images](docker/using_docker_images.md)
 + [Using Docker Build](docker/using_docker_build.md)
 + [Using Variables](variables/README.md)
++ [Using SSH keys](ssh_keys/README.md)
+
+### Languages
+
++ [Testing PHP](languages/php.md)
+
+### Services
+
++ [Using MySQL](services/mysql.md)
++ [Using PostgreSQL](services/postgres.md)
++ [Using Redis](services/redis.md)
++ [Using Other Services](docker/using_docker_images.md#how-to-use-other-images-as-services)
 
 ### Examples
 

doc/ci/languages/README.md

+### Languages
+
+This is a list of languages you can test with GitLab CI. Each section has
+comprehensive documentation and comes with a test repository hosted on
+GitLab.com
+
++ [Testing PHP](php.md)

doc/ci/languages/php.md

+# Testing PHP projects
+
+This guide covers basic building instructions for PHP projects.
+
+There are covered two cases: testing using the Docker executor and testing
+using the Shell executor.
+
+## Test PHP projects using the Docker executor
+
+While it is possible to test PHP apps on any system, this would require manual
+configuration from the developer. To overcome this we will be using the
+official [PHP docker image][php-hub] that can be found in Docker Hub.
+
+This will allow us to test PHP projects against different versions of PHP.
+However, not everything is plug 'n' play, you still need to onfigure some
+things manually.
+
+As with every build, you need to create a valid `.gitlab-ci.yml` describing the
+build environment.
+
+Let's first specify the PHP image that will be used for the build process
+(you can read more about what an image means in the Runner's lingo reading
+about [Using Docker images](../docker/using_docker_images.md#what-is-image)).
+
+Start by adding the image to your `.gitlab-ci.yml`:
+
+```yaml
+image: php:5.6
+```
+
+The official images are great, but they lack a few useful tools for testing.
+We need to first prepare the build environment. A way to overcome this is to
+create a script which installs all prerequisites prior the actual testing is
+done.
+
+Let's create a `ci/docker_install.sh` file in the root directory of our
+repository with the following content:
+
+```bash
+#!/bin/bash
+
+# We need to install dependencies only for Docker
+[[ ! -e /.dockerinit ]] && exit 0
+
+set -xe
+
+# Install git (the php image doesn't have it) which is required by composer
+apt-get update -yqq
+apt-get install git -yqq
+
+# Install phpunit, the tool that we will use for testing
+curl -o /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar
+chmod +x /usr/local/bin/phpunit
+
+# Install mysql driver
+# Here you can install any other extension that you need
+docker-php-ext-install pdo_mysql
+```
+
+You might wonder what `docker-php-ext-install` is. In short, it is a script
+provided by the official php docker image that you can use to easilly install
+extensions. For more information read the the documentation at
+<https://hub.docker.com/_/php/>.
+
+Now that we created the script that contains all prerequisites for our build
+environment, let's add it in `.gitlab-ci.yml`:
+
+```yaml
+...
+
+before_script:
+- bash ci/docker_install.sh > /dev/null
+
+...
+```
+
+Last step, run the actual tests using `phpunit`:
+
+```yaml
+...
+
+test:app:
+  script:
+  - phpunit --configuration phpunit_myapp.xml
+
+...
+```
+
+Finally, commit your files and push them to GitLab to see your build succeeding
+(or failing).
+
+The final `.gitlab-ci.yml` should look similar to this:
+
+```yaml
+# Select image from https://hub.docker.com/_/php/
+image: php:5.6
+
+before_script:
+# Install dependencies
+- ci/docker_install.sh > /dev/null
+
+test:app:
+  script:
+  - phpunit --configuration phpunit_myapp.xml
+```
+
+### Test against different PHP versions in Docker builds
+
+Testing against multiple versions of PHP is super easy. Just add another job
+with a different docker image version and the runner will do the rest:
+
+```yaml
+before_script:
+# Install dependencies
+- ci/docker_install.sh > /dev/null
+
+# We test PHP5.6
+test:5.6:
+  image: php:5.6
+  script:
+  - phpunit --configuration phpunit_myapp.xml
+
+# We test PHP7.0 (good luck with that)
+test:7.0:
+  image: php:7.0
+  script:
+  - phpunit --configuration phpunit_myapp.xml
+```
+
+### Custom PHP configuration in Docker builds
+
+There are times where you will need to customise your PHP environment by
+putting your `.ini` file into `/usr/local/etc/php/conf.d/`. For that purpose
+add a `before_script` action:
+
+```yaml
+before_script:
+- cp my_php.ini /usr/local/etc/php/conf.d/test.ini
+```
+
+Of course, `my_php.ini` must be present in the root directory of your repository.
+
+## Test PHP projects using the Shell executor
+
+The shell executor runs your builds in a terminal session on your server.
+Thus, in order to test your projects you first need to make sure that all
+dependencies are installed.
+
+For example, in a VM running Debian 8 we first update the cache, then we
+install `phpunit` and `php5-mysql`:
+
+```bash
+sudo apt-get update -y
+sudo apt-get install -y phpunit php5-mysql
+```
+
+Next, add the following snippet to your `.gitlab-ci.yml`:
+
+```yaml
+test:app:
+  script:
+  - phpunit --configuration phpunit_myapp.xml
+```
+
+Finally, push to GitLab and let the tests begin!
+
+### Test against different PHP versions in Shell builds
+
+The [phpenv][] project allows you to easily manage different versions of PHP
+each with its own config. This is specially usefull when testing PHP projects
+with the Shell executor.
+
+You will have to install it on your build machine under the `gitlab-runner`
+user following [the upstream installation guide][phpenv-installation].
+
+Using phpenv also allows to easily configure the PHP environment with:
+
+```
+phpenv config-add my_config.ini
+```
+
+*__Important note:__ It seems `phpenv/phpenv`
+ [is abandoned](https://github.com/phpenv/phpenv/issues/57). There is a fork
+ at [madumlao/phpenv](https://github.com/madumlao/phpenv) that tries to bring
+ the project back to life. [CHH/phpenv](https://github.com/CHH/phpenv) also
+ seems like a good alternative. Picking any of the mentioned tools will work
+ with the basic phpenv commands. Guiding you to choose the right phpenv is out
+ of the scope of this tutorial.*
+
+### Install custom extensions
+
+Since this is a pretty bare installation of the PHP environment, you may need
+some extensions that are not currently present on the build machine.
+
+To install additional extensions simply execute:
+
+```bash
+pecl install <extension>
+```
+
+It's not advised to add this to `.gitlab-ci.yml`. You should execute this
+command once, only to setup the build environment.
+
+## Extend your tests
+
+### Using atoum
+
+Instead of PHPUnit, you can use any other tool to run unit tests. For example
+you can use [atoum](https://github.com/atoum/atoum):
+
+```yaml
+before_script:
+- wget http://downloads.atoum.org/nightly/mageekguy.atoum.phar
+
+test:atoum:
+  script:
+  - php mageekguy.atoum.phar
+```
+
+### Using Composer
+
+The majority of the PHP projects use Composer for managing their PHP packages.
+In order to execute Composer before running your tests, simply add the
+following in your `.gitlab-ci.yml`:
+
+```yaml
+...
+
+# Composer stores all downloaded packages in the vendor/ directory.
+# Do not use the following if the vendor/ directory is commited to
+# your git repository.
+cache:
+  paths:
+  - vendor/
+
+before_script:
+# Install composer dependencies
+- curl -sS https://getcomposer.org/installer | php
+- php composer.phar install
+
+...
+```
+
+## Access private packages / dependencies
+
+If your test suite needs to access a private repository, you need to configure
+[the SSH keys](../ssh_keys/README.md) in order to be able to clone it.
+
+## Use databases or other services
+
+Most of the time you will need a running database in order for your tests to
+run. If you are using the Docker executor you can leverage Docker's ability to
+link to other containers. In GitLab Runner lingo, this can be achieved by
+defining a `service`.
+
+This functionality is covered in [the CI services](../services/README.md)
+documentation.
+
+## Testing things locally
+
+With GitLab Runner 1.0 you can also test any changes locally. From your
+terminal execute:
+
+```bash
+# Check using docker executor
+gitlab-runner exec docker test:app
+
+# Check using shell executor
+gitlab-runner exec shell test:app
+```
+
+## Example project
+
+We have set up an [Example PHP Project][php-example-repo] for your convenience
+that runs on [GitLab.com](https://gitlab.com) using our publicly available
+[shared runners](../runners/README.md).
+
+Want to hack on it? Simply fork it, commit and push  your changes. Within a few
+moments the changes will be picked by a public runner and the build will begin.
+
+[php-hub]: https://hub.docker.com/_/php/
+[phpenv]: https://github.com/phpenv/phpenv
+[phpenv-installation]: https://github.com/phpenv/phpenv#installation
+[php-example-repo]: https://gitlab.com/gitlab-examples/php

doc/ci/services/README.md

+## GitLab CI Services
+
+GitLab CI uses the `services` keyword to define what docker containers should be
+linked with your base image. Below is a list of examples you may use.
+
++ [Using MySQL](mysql.md)
++ [Using PostgreSQL](postgres.md)
++ [Using Redis](redis.md)
++ [Using Other Services](../docker/using_docker_images.md#how-to-use-other-images-as-services)

doc/ci/services/docker-services.md

+## GitLab CI Services
+
++ [Using MySQL](mysql.md)
++ [Using PostgreSQL](postgres.md)
++ [Using Redis](redis.md)

doc/ci/services/mysql.md

+# Using MySQL
+
+As many applications depend on MySQL as their database, you will eventually
+need it in order for your tests to run. Below you are guided how to do this
+with the Docker and Shell executors of GitLab Runner.
+
+## Use MySQL with the Docker executor
+
+If you are using [GitLab Runner](../runners/README.md) with the Docker executor
+you basically have everything set up already.
+
+First, in your `.gitlab-ci.yml` add:
+
+```yaml
+services:
+  - mysql:latest
+
+variables:
+  # Configure mysql environment variables (https://hub.docker.com/_/mysql/)
+  MYSQL_DATABASE: el_duderino
+  MYSQL_ROOT_PASSWORD: mysql_strong_password
+```
+
+And then configure your application to use the database, for example:
+
+```yaml
+Host: mysql
+User: root
+Password: mysql_strong_password
+Database: el_duderino
+```
+
+If you are wondering why we used `mysql` for the `Host`, read more at
+[How is service linked to the build](../docker/using_docker_images.md#how-is-service-linked-to-the-build).
+
+You can also use any other docker image available on [Docker Hub][hub-mysql].
+For example, to use MySQL 5.5 the service becomes `mysql:5.5`.
+
+The `mysql` image can accept some environment variables. For more details
+check the documentation on [Docker Hub][hub-mysql].
+
+## Use MySQL with the Shell executor
+
+You can also use MySQL on manually configured servers that are using
+GitLab Runner with the Shell executor.
+
+First install the MySQL server:
+
+```bash
+sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev
+```
+
+Pick a MySQL root password (can be anything), and type it twice when asked.
+
+*Note: As a security measure you can run `mysql_secure_installation` to
+remove anonymous users, drop the test database and disable remote logins with
+the root user.*
+
+The next step is to create a user, so login to MySQL as root:
+
+```bash
+mysql -u root -p
+```
+
+Then create a user (in our case `runner`) which will be used by your
+application. Change `$password` in the command below to a real strong password.
+
+*Note: Do not type `mysql>`, this is part of the MySQL prompt.*
+
+```bash
+mysql> CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password';
+```
+
+Create the database:
+
+```bash
+mysql> CREATE DATABASE IF NOT EXISTS `el_duderino` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;
+```
+
+Grant the necessary permissions on the database:
+
+```bash
+mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `el_duderino`.* TO 'runner'@'localhost';
+```
+
+If all went well you can now quit the database session:
+
+```bash
+mysql> \q
+```
+
+Now, try to connect to the newly created database to check that everything is
+in place:
+
+```bash
+mysql -u runner -p -D el_duderino
+```
+
+As a final step, configure your application to use the database, for example:
+
+```bash
+Host: localhost
+User: runner
+Password: $password
+Database: el_duderino
+```
+
+## Example project
+
+We have set up an [Example MySQL Project][mysql-example-repo] for your
+convenience that runs on [GitLab.com](https://gitlab.com) using our publicly
+available [shared runners](../runners/README.md).
+
+Want to hack on it? Simply fork it, commit and push  your changes. Within a few
+moments the changes will be picked by a public runner and the build will begin.
+
+[hub-mysql]: https://hub.docker.com/_/mysql/
+[mysql-example-repo]: https://gitlab.com/gitlab-examples/mysql

doc/ci/services/postgres.md

+# Using PostgreSQL
+
+As many applications depend on PostgreSQL as their database, you will
+eventually need it in order for your tests to run. Below you are guided how to
+do this with the Docker and Shell executors of GitLab Runner.
+
+## Use PostgreSQL with the Docker executor
+
+If you are using [GitLab Runner](../runners/README.md) with the Docker executor
+you basically have everything set up already.
+
+First, in your `.gitlab-ci.yml` add:
+
+```yaml
+services:
+  - postgres:latest
+
+variables:
+  POSTGRES_DB: nice_marmot
+  POSTGRES_USER: runner
+  POSTGRES_PASSWORD: ""
+```
+
+And then configure your application to use the database, for example:
+
+```yaml
+Host: postgres
+User: runner
+Password:
+Database: nice_marmot
+```
+
+If you are wondering why we used `postgres` for the `Host`, read more at
+[How is service linked to the build](../docker/using_docker_images.md#how-is-service-linked-to-the-build).
+
+You can also use any other docker image available on [Docker Hub][hub-pg].
+For example, to use PostgreSQL 9.3 the service becomes `postgres:9.3`.
+
+The `postgres` image can accept some environment variables. For more details
+check the documentation on [Docker Hub][hub-pg].
+
+## Use PostgreSQL with the Shell executor
+
+You can also use PostgreSQL on manually configured servers that are using
+GitLab Runner with the Shell executor.
+
+First install the PostgreSQL server:
+
+```bash
+sudo apt-get install -y postgresql postgresql-client libpq-dev
+```
+
+The next step is to create a user, so login to PostgreSQL:
+
+```bash
+sudo -u postgres psql -d template1
+```
+
+Then create a user (in our case `runner`) which will be used by your
+application. Change `$password` in the command below to a real strong password.
+
+*__Note:__ Do not type `template1=#`, this is part of the PostgreSQL prompt.*
+
+```bash
+template1=# CREATE USER runner WITH PASSWORD '$password' CREATEDB;
+```
+
+*__Note:__ Notice that we created the user with the privilege to be able to
+create databases (`CREATEDB`). In the following steps we will create a database 
+explicitly for that user but having that privilege can be useful if in your
+testing framework you have tools that drop and create databases.*
+
+Create the database and grant all privileges on it for the user `runner`:
+
+```bash
+template1=# CREATE DATABASE nice_marmot OWNER runner;
+```
+
+If all went well you can now quit the database session:
+
+```bash
+template1=# \q
+```
+
+Now, try to connect to the newly created database with the user `runner` to
+check that everything is in place.
+
+```bash
+psql -U runner -h localhost -d nice_marmot -W
+```
+
+*__Note:__ We are explicitly telling `psql` to connect to localhost in order
+to use the md5 authentication. If you omit this step you will be denied access.*
+
+Finally, configure your application to use the database, for example:
+
+```yaml
+Host: localhost
+User: runner
+Password: $password
+Database: nice_marmot
+```
+
+## Example project
+
+We have set up an [Example PostgreSQL Project][postgres-example-repo] for your
+convenience that runs on [GitLab.com](https://gitlab.com) using our publicly
+available [shared runners](../runners/README.md).
+
+Want to hack on it? Simply fork it, commit and push  your changes. Within a few
+moments the changes will be picked by a public runner and the build will begin.
+
+[hub-pg]: https://hub.docker.com/_/postgres/
+[postgres-example-repo]: https://gitlab.com/gitlab-examples/postgres

doc/ci/services/redis.md

+# Using Redis
+
+As many applications depend on Redis as their key-value store, you will
+eventually need it in order for your tests to run. Below you are guided how to
+do this with the Docker and Shell executors of GitLab Runner.
+
+## Use Redis with the Docker executor
+
+If you are using [GitLab Runner](../runners/README.md) with the Docker executor
+you basically have everything set up already.
+
+First, in your `.gitlab-ci.yml` add:
+
+```yaml
+services:
+  - redis:latest
+```
+
+Then you need to configure your application to use the Redis database, for
+example:
+
+```yaml
+Host: redis
+```
+
+And that's it. Redis will now be available to be used within your testing
+framework.
+
+You can also use any other docker image available on [Docker Hub][hub-redis].
+For example, to use Redis 2.8 the service becomes `redis:2.8`.
+
+## Use Redis with the Shell executor
+
+Redis can also be used on manually configured servers that are using GitLab
+Runner with the Shell executor.
+
+In your build machine install the Redis server:
+
+```bash
+sudo apt-get install redis-server
+```
+
+Verify that you can connect to the server with the `gitlab-runner` user:
+
+```bash
+# Try connecting the the Redis server
+sudo -u gitlab-runner -H redis-cli
+
+# Quit the session
+127.0.0.1:6379> quit
+```
+
+Finally, configure your application to use the database, for example:
+
+```yaml
+Host: localhost
+```
+
+## Example project
+
+We have set up an [Example Redis Project][redis-example-repo] for your convenience
+that runs on [GitLab.com](https://gitlab.com) using our publicly available
+[shared runners](../runners/README.md).
+
+Want to hack on it? Simply fork it, commit and push  your changes. Within a few
+moments the changes will be picked by a public runner and the build will begin.
+
+[hub-redis]: https://hub.docker.com/_/redis/
+[redis-example-repo]: https://gitlab.com/gitlab-examples/redis

doc/ci/ssh_keys/README.md

+# Using SSH keys
+
+GitLab currently doesn't have built-in support for managing SSH keys in a build
+environment.
+
+The SSH keys can be useful when:
+
+1. You want to checkout internal submodules
+2. You want to download private packages using your package manager (eg. bundler)
+3. You want to deploy your application to eg. Heroku or your own server
+4. You want to execute SSH commands from the build server to the remote server
+5. You want to rsync files from your build server to the remote server
+
+If anything of the above rings a bell, then you most likely need an SSH key.
+
+## Inject keys in your build server
+
+The most widely supported method is to inject an SSH key into your build
+environment by extending your `.gitlab-ci.yml`.
+
+This is the universal solution which works with any type of executor
+(docker, shell, etc.).
+
+### How it works
+
+1. Create a new SSH key pair with [ssh-keygen][]
+2. Add the private key as a **Secret Variable** to the project
+3. Run the [ssh-agent][] during build to load the private key.
+
+## SSH keys when using the Docker executor
+
+You will first need to create an SSH key pair. For more information, follow the
+instructions to [generate an SSH key](../ssh/README.md).
+
+Then, create a new **Secret Variable** in your project settings on GitLab
+following **Settings > Variables**. As **Key** add the name `SSH_PRIVATE_KEY`
+and in the **Value** field paste the content of your _private_ key that you
+created earlier.
+
+Next you need to modify your `.gitlab-ci.yml` with a `before_script` action.
+Add it to the top:
+
+```
+before_script:
+  # Install ssh-agent if not already installed, it is required by Docker.
+  # (change apt-get to yum if you use a CentOS-based image)
+  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
+
+  # Run ssh-agent (inside the build environment)
+  - eval $(ssh-agent -s)
+
+  # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
+  - ssh-add <(echo "$SSH_PRIVATE_KEY")
+
+  # For Docker builds disable host key checking. Be aware that by adding that
+  # you are suspectible to man-in-the-middle attacks.
+  # WARNING: Use this only with the Docker executor, if you use it with shell
+  # you will overwrite your user's SSH config.
+  - mkdir -p ~/.ssh
+  - '[[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config`
+```
+
+As a final step, add the _public_ key from the one you created earlier to the
+services that you want to have an access to from within the build environment.
+If you are accessing a private GitLab repository you need to add it as a
+[deploy key](../ssh/README.md#deploy-keys).
+
+That's it! You can now have access to private servers or repositories in your
+build environment.
+
+## SSH keys when using the Shell executor
+
+If you are using the Shell executor and not Docker, it is easier to set up an
+SSH key.
+
+You can generate the SSH key from the machine that GitLab Runner is installed
+on, and use that key for all projects that are run on this machine.
+
+First, you need to login to the server that runs your builds.
+
+Then from the terminal login as the `gitlab-runner` user and generate the SSH
+key pair as described in the [SSH keys documentation](../ssh/README.md).
+
+As a final step, add the _public_ key from the one you created earlier to the
+services that you want to have an access to from within the build environment.
+If you are accessing a private GitLab repository you need to add it as a
+[deploy key](../ssh/README.md#deploy-keys).
+
+Once done, try to login to the remote server in order to accept the fingerprint:
+
+```bash
+ssh <address-of-my-server>
+```
+
+For accessing repositories on GitLab.com, the `<address-of-my-server>` would be
+`git@gitlab.com`.
+
+## Example project
+
+We have set up an [Example SSH Project][ssh-example-repo] for your convenience
+that runs on [GitLab.com](https://gitlab.com) using our publicly available
+[shared runners](../runners/README.md).
+
+Want to hack on it? Simply fork it, commit and push your changes. Within a few
+moments the changes will be picked by a public runner and the build will begin.
+
+[ssh-keygen]: http://linux.die.net/man/1/ssh-keygen
+[ssh-agent]: http://linux.die.net/man/1/ssh-agent
+[ssh-example-repo]: https://gitlab.com/gitlab-examples/ssh-private-key/