Commit 958af6f5 authored by Jacob Vosmaer's avatar Jacob Vosmaer

Merge branch 'master' into 'master'

Add support for AWS S3 Server-Side Encryption support

This adds support for AWS S3 SSE with S3 managed keys, this means the
data is encrypted at rest and the encryption is handled transparently to
the end user as well as in the AWS Console.

## Points to double check
I'm unsure on the best way to the handle the default. I've followed the `multipart_upload` de facto in the app. I'm happy to change this if required or if it will impact elsewhere e.g. omnibus packages

I also think I've managed to catch all of the documentation for this change as well.

### Why is this required
Many enterprises require good backup support but also for this to be encrypted. By default backups aren't encrypted, this allows at rest encryption to be supported in GitLab backups providing a layer of security should the physical media not be properly disposed of.

Relates to issue #2478.


See merge request !1280
parents e1b7fced 253d2320
...@@ -33,6 +33,7 @@ v 8.0.2 ...@@ -33,6 +33,7 @@ v 8.0.2
- Use standard Markdown font in Markdown preview instead of fixed-width font (Stan Hu) - Use standard Markdown font in Markdown preview instead of fixed-width font (Stan Hu)
- Fix Reply by email for non-UTF-8 messages. - Fix Reply by email for non-UTF-8 messages.
- Add option to use StartTLS with Reply by email IMAP server. - Add option to use StartTLS with Reply by email IMAP server.
- Allow AWS S3 Server-Side Encryption with Amazon S3-Managed Keys for backups (Paul Beattie)
v 8.0.1 v 8.0.1
- Remove git refs used internally by GitLab from network graph (Stan Hu) - Remove git refs used internally by GitLab from network graph (Stan Hu)
......
...@@ -159,7 +159,7 @@ production: &base ...@@ -159,7 +159,7 @@ production: &base
method: 'plain' # "tls" or "ssl" or "plain" method: 'plain' # "tls" or "ssl" or "plain"
bind_dn: '_the_full_dn_of_the_user_you_will_bind_with' bind_dn: '_the_full_dn_of_the_user_you_will_bind_with'
password: '_the_password_of_the_bind_user' password: '_the_password_of_the_bind_user'
# This setting specifies if LDAP server is Active Directory LDAP server. # This setting specifies if LDAP server is Active Directory LDAP server.
# For non AD servers it skips the AD specific queries. # For non AD servers it skips the AD specific queries.
# If your LDAP server is not AD, set this to false. # If your LDAP server is not AD, set this to false.
...@@ -204,13 +204,13 @@ production: &base ...@@ -204,13 +204,13 @@ production: &base
# The username will be used in paths for the user's own projects # The username will be used in paths for the user's own projects
# (like `gitlab.example.com/username/project`) and when mentioning # (like `gitlab.example.com/username/project`) and when mentioning
# them in issues, merge request and comments (like `@username`). # them in issues, merge request and comments (like `@username`).
# If the attribute specified for `username` contains an email address, # If the attribute specified for `username` contains an email address,
# the GitLab username will be the part of the email address before the '@'. # the GitLab username will be the part of the email address before the '@'.
username: ['uid', 'userid', 'sAMAccountName'] username: ['uid', 'userid', 'sAMAccountName']
email: ['mail', 'email', 'userPrincipalName'] email: ['mail', 'email', 'userPrincipalName']
# If no full name could be found at the attribute specified for `name`, # If no full name could be found at the attribute specified for `name`,
# the full name is determined using the attributes specified for # the full name is determined using the attributes specified for
# `first_name` and `last_name`. # `first_name` and `last_name`.
name: 'cn' name: 'cn'
first_name: 'givenName' first_name: 'givenName'
...@@ -252,28 +252,28 @@ production: &base ...@@ -252,28 +252,28 @@ production: &base
# arguments, followed by optional 'args' which can be either a hash or an array. # arguments, followed by optional 'args' which can be either a hash or an array.
# Documentation for this is available at http://doc.gitlab.com/ce/integration/omniauth.html # Documentation for this is available at http://doc.gitlab.com/ce/integration/omniauth.html
providers: providers:
# - { name: 'google_oauth2', # - { name: 'google_oauth2',
# label: 'Google', # label: 'Google',
# app_id: 'YOUR_APP_ID', # app_id: 'YOUR_APP_ID',
# app_secret: 'YOUR_APP_SECRET', # app_secret: 'YOUR_APP_SECRET',
# args: { access_type: 'offline', approval_prompt: '' } } # args: { access_type: 'offline', approval_prompt: '' } }
# - { name: 'twitter', # - { name: 'twitter',
# app_id: 'YOUR_APP_ID', # app_id: 'YOUR_APP_ID',
# app_secret: 'YOUR_APP_SECRET' } # app_secret: 'YOUR_APP_SECRET' }
# - { name: 'github', # - { name: 'github',
# label: 'GitHub', # label: 'GitHub',
# app_id: 'YOUR_APP_ID', # app_id: 'YOUR_APP_ID',
# app_secret: 'YOUR_APP_SECRET', # app_secret: 'YOUR_APP_SECRET',
# args: { scope: 'user:email' } } # args: { scope: 'user:email' } }
# - { name: 'gitlab', # - { name: 'gitlab',
# label: 'GitLab.com', # label: 'GitLab.com',
# app_id: 'YOUR_APP_ID', # app_id: 'YOUR_APP_ID',
# app_secret: 'YOUR_APP_SECRET', # app_secret: 'YOUR_APP_SECRET',
# args: { scope: 'api' } } # args: { scope: 'api' } }
# - { name: 'bitbucket', # - { name: 'bitbucket',
# app_id: 'YOUR_APP_ID', # app_id: 'YOUR_APP_ID',
# app_secret: 'YOUR_APP_SECRET' } # app_secret: 'YOUR_APP_SECRET' }
# - { name: 'saml', # - { name: 'saml',
# label: 'Our SAML Provider', # label: 'Our SAML Provider',
# args: { # args: {
# assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', # assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback',
...@@ -319,6 +319,8 @@ production: &base ...@@ -319,6 +319,8 @@ production: &base
# # Use multipart uploads when file size reaches 100MB, see # # Use multipart uploads when file size reaches 100MB, see
# # http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html # # http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html
# multipart_chunk_size: 104857600 # multipart_chunk_size: 104857600
# # Turns on AWS Server-Side Encryption with Amazon S3-Managed Keys for backups, this is optional
# # encryption: 'AES256'
## GitLab Shell settings ## GitLab Shell settings
gitlab_shell: gitlab_shell:
......
...@@ -229,6 +229,7 @@ if Settings.backup['upload']['connection'] ...@@ -229,6 +229,7 @@ if Settings.backup['upload']['connection']
Settings.backup['upload']['connection'] = Hash[Settings.backup['upload']['connection'].map { |k, v| [k.to_sym, v] }] Settings.backup['upload']['connection'] = Hash[Settings.backup['upload']['connection'].map { |k, v| [k.to_sym, v] }]
end end
Settings.backup['upload']['multipart_chunk_size'] ||= 104857600 Settings.backup['upload']['multipart_chunk_size'] ||= 104857600
Settings.backup['upload']['encryption'] ||= nil
# #
# Git # Git
......
...@@ -95,6 +95,8 @@ For installations from source: ...@@ -95,6 +95,8 @@ For installations from source:
aws_secret_access_key: 'secret123' aws_secret_access_key: 'secret123'
# The remote 'directory' to store your backups. For S3, this would be the bucket name. # The remote 'directory' to store your backups. For S3, this would be the bucket name.
remote_directory: 'my.s3.bucket' remote_directory: 'my.s3.bucket'
# Turns on AWS Server-Side Encryption with Amazon S3-Managed Keys for backups, this is optional
# encryption: 'AES256'
``` ```
If you are uploading your backups to S3 you will probably want to create a new If you are uploading your backups to S3 you will probably want to create a new
......
...@@ -45,7 +45,8 @@ module Backup ...@@ -45,7 +45,8 @@ module Backup
directory = connection.directories.get(remote_directory) directory = connection.directories.get(remote_directory)
if directory.files.create(key: tar_file, body: File.open(tar_file), public: false, if directory.files.create(key: tar_file, body: File.open(tar_file), public: false,
multipart_chunk_size: Gitlab.config.backup.upload.multipart_chunk_size) multipart_chunk_size: Gitlab.config.backup.upload.multipart_chunk_size,
encryption: Gitlab.config.backup.upload.encryption)
$progress.puts "done".green $progress.puts "done".green
else else
puts "uploading backup to #{remote_directory} failed".red puts "uploading backup to #{remote_directory} failed".red
...@@ -55,7 +56,7 @@ module Backup ...@@ -55,7 +56,7 @@ module Backup
def cleanup def cleanup
$progress.print "Deleting tmp directories ... " $progress.print "Deleting tmp directories ... "
backup_contents.each do |dir| backup_contents.each do |dir|
next unless File.exist?(File.join(Gitlab.config.backup.path, dir)) next unless File.exist?(File.join(Gitlab.config.backup.path, dir))
...@@ -75,7 +76,7 @@ module Backup ...@@ -75,7 +76,7 @@ module Backup
if keep_time > 0 if keep_time > 0
removed = 0 removed = 0
Dir.chdir(Gitlab.config.backup.path) do Dir.chdir(Gitlab.config.backup.path) do
file_list = Dir.glob('*_gitlab_backup.tar') file_list = Dir.glob('*_gitlab_backup.tar')
file_list.map! { |f| $1.to_i if f =~ /(\d+)_gitlab_backup.tar/ } file_list.map! { |f| $1.to_i if f =~ /(\d+)_gitlab_backup.tar/ }
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment