Commit 91f441c1 authored by Evan Read's avatar Evan Read

Merge branch 'docs/refactor-pgloader' into 'master'

Clean up the mysql to postgres guide

See merge request gitlab-org/gitlab-ce!24061
parents 43f9d5d7 7fcd386a
---
last_updated: 2018-02-07
last_updated: 2019-03-27
---
# Migrating from MySQL to PostgreSQL
> **Note:** This guide assumes you have a working GitLab instance with
> MySQL and want to migrate to bundled PostgreSQL database.
This guide documents how to take a working GitLab instance that uses MySQL and
migrate it to a PostgreSQL database.
## Omnibus installation
## Requirements
### Prerequisites
[pgloader](http://pgloader.io) 3.4.1+ is required.
First, we'll need to enable the bundled PostgreSQL database with up-to-date
schema. Next, we'll use [pgloader](http://pgloader.io) to migrate the data
from the old MySQL database to the new PostgreSQL one.
You can install it directly from your distribution, for example in
Debian/Ubuntu:
Here's what you'll need to have installed:
1. Search for the version:
- pgloader 3.4.1+
- Omnibus GitLab
- MySQL
```bash
apt-cache madison pgloader
```
1. If the version is 3.4.1+, install it with:
```bash
sudo apt-get install pgloader
```
### Enable bundled PostgreSQL database
If your distribution's version is too old, use PostgreSQL's repository:
```bash
# Add repository
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
# Add key
sudo apt-get install wget ca-certificates
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# Install package
sudo apt-get update
sudo apt-get install pgloader
```
For other distributions, follow the instructions in PostrgreSQL's
[download page](https://www.postgresql.org/download/) to add their repository
and then install `pgloader`.
## Omnibus GitLab installations
For [Omnibus GitLab packages](https://about.gitlab.com/install/), you'll first
need to enable the bundled PostgreSQL:
1. Stop GitLab:
``` bash
```bash
sudo gitlab-ctl stop
```
......@@ -40,39 +67,34 @@ Here's what you'll need to have installed:
and alike. You could just comment all of them out so that we'll just use
the defaults.
1. [Reconfigure GitLab] for the changes to take effect:
``` bash
sudo gitlab-ctl reconfigure
```
1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure)
for the changes to take effect.
1. Start Unicorn and PostgreSQL so that we can prepare the schema:
``` bash
```bash
sudo gitlab-ctl start unicorn
sudo gitlab-ctl start postgresql
```
1. Run the following commands to prepare the schema:
``` bash
```bash
sudo gitlab-rake db:create db:migrate
```
1. Stop Unicorn to prevent other database access from interfering with the loading of data:
``` bash
```bash
sudo gitlab-ctl stop unicorn
```
After these steps, you'll have a fresh PostgreSQL database with up-to-date schema.
### Migrate data from MySQL to PostgreSQL
Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
Next, we'll use `pgloader` to migrate the data from the old MySQL database to the
new PostgreSQL one:
1. Save the following snippet in a `commands.load` file, and edit with your
database `username`, `password` and `host`:
MySQL database `username`, `password` and `host`:
```
LOAD DATABASE
......@@ -90,7 +112,7 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
1. Start the migration:
``` bash
```bash
sudo -u gitlab-psql pgloader commands.load
```
......@@ -117,106 +139,56 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
Total import time 1894 1894 0 12.497s
```
If there is no output for more than 30 minutes, it's possible pgloader encountered an error. See
the [troubleshooting guide](#Troubleshooting) for more details.
If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See
the [troubleshooting guide](#troubleshooting) for more details.
1. Start GitLab:
``` bash
```bash
sudo gitlab-ctl start
```
Now, you can verify that everything worked by visiting GitLab.
You can now verify that everything works as expected by visiting GitLab.
### Troubleshooting
## Source installations
#### Permissions
For installations from source that use MySQL, you'll first need to
[install PostgreSQL and create a database](../install/installation.md#6-database).
Note that the PostgreSQL user that you use for the above MUST have **superuser** privileges. Otherwise, you may see
a similar message to the following:
```
debugger invoked on a CL-POSTGRES-ERROR:INSUFFICIENT-PRIVILEGE in thread
#<THREAD "lparallel" RUNNING {10078A3513}>:
Database error 42501: permission denied: "RI_ConstraintTrigger_a_20937" is a system trigger
QUERY: ALTER TABLE ci_builds DISABLE TRIGGER ALL;
2017-08-23T00:36:56.782000Z ERROR Database error 42501: permission denied: "RI_ConstraintTrigger_c_20864" is a system trigger
QUERY: ALTER TABLE approver_groups DISABLE TRIGGER ALL;
```
#### Experiencing 500 errors after the migration
If you experience 500 errors after the migration, try to clear the cache:
``` bash
sudo gitlab-rake cache:clear
```
[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
## Source installation
### Prerequisites
#### Install PostgreSQL and create database
See [installation guide](../install/installation.md#6-database).
#### Install [pgloader](http://pgloader.io) 3.4.1+
Install directly from your distro:
``` bash
sudo apt-get install pgloader
```
If this version is too old, use PostgreSQL's repository:
``` bash
# add repository
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
# add key
sudo apt-get install wget ca-certificates
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# install package
sudo apt-get update
sudo apt-get install pgloader
```
### Enable bundled PostgreSQL database
After the database is created, go on with the following steps:
1. Stop GitLab:
``` bash
```bash
sudo service gitlab stop
```
1. Switch database from MySQL to PostgreSQL
``` bash
```bash
cd /home/git/gitlab
sudo -u git mv config/database.yml config/database.yml.bak
sudo -u git cp config/database.yml.postgresql config/database.yml
sudo -u git -H chmod o-rwx config/database.yml
```
1. Install Gems related to Postgresql
``` bash
```bash
sudo -u git -H rm .bundle/config
sudo -u git -H bundle install --deployment --without development test mysql aws kerberos
```
1. Run the following commands to prepare the schema:
``` bash
```bash
sudo -u git -H bundle exec rake db:create db:migrate RAILS_ENV=production
```
After these steps, you'll have a fresh PostgreSQL database with up-to-date schema.
### Migrate data from MySQL to PostgreSQL
Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
Next, we'll use `pgloader` to migrate the data from the old MySQL database to the
new PostgreSQL one:
1. Save the following snippet in a `commands.load` file, and edit with your
MySQL `username`, `password` and `host`:
......@@ -237,7 +209,7 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
1. Start the migration:
``` bash
```bash
sudo -u postgres pgloader commands.load
```
......@@ -264,23 +236,43 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
Total import time 1894 1894 0 12.497s
```
If there is no output for more than 30 minutes, it's possible pgloader encountered an error. See
the [troubleshooting guide](#Troubleshooting) for more details.
If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See
the [troubleshooting guide](#troubleshooting) for more details.
1. Start GitLab:
``` bash
```bash
sudo service gitlab start
```
Now, you can verify that everything worked by visiting GitLab.
You can now verify that everything works as expected by visiting GitLab.
### Troubleshooting
## Troubleshooting
#### Experiencing 500 errors after the migration
Sometimes, you might encounter some errors during or after the migration.
### Database error permission denied
The PostgreSQL user that you use for the migration MUST have **superuser** privileges.
Otherwise, you may see a similar message to the following:
```
debugger invoked on a CL-POSTGRES-ERROR:INSUFFICIENT-PRIVILEGE in thread
#<THREAD "lparallel" RUNNING {10078A3513}>:
Database error 42501: permission denied: "RI_ConstraintTrigger_a_20937" is a system trigger
QUERY: ALTER TABLE ci_builds DISABLE TRIGGER ALL;
2017-08-23T00:36:56.782000Z ERROR Database error 42501: permission denied: "RI_ConstraintTrigger_c_20864" is a system trigger
QUERY: ALTER TABLE approver_groups DISABLE TRIGGER ALL;
```
### Experiencing 500 errors after the migration
If you experience 500 errors after the migration, try to clear the cache:
``` bash
```bash
# Omnibus GitLab
sudo gitlab-rake cache:clear
# Installations from source
sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production
```
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment