Commit a9a1e26f authored by Rémy Coutable's avatar Rémy Coutable

Merge branch 'ce-to-ee-2018-03-30' into 'master'

CE upstream - 2018-03-30 15:25 UTC

See merge request gitlab-org/gitlab-ee!5185
parents 5b2b4d48 f8a96a0a
......@@ -322,8 +322,17 @@ package-and-qa:
stage: build
cache: {}
when: manual
variables:
GIT_STRATEGY: none
before_script:
# We need to download the script rather than clone the repo since the
# package-and-qa job will not be able to run when the branch gets
# deleted (when merging the MR).
- apk add --update openssl
- wget https://gitlab.com/gitlab-org/gitlab-ce/raw/$CI_COMMIT_SHA/scripts/trigger-build-omnibus
- chmod 755 trigger-build-omnibus
script:
- scripts/trigger-build-omnibus
- ./trigger-build-omnibus
only:
- //@gitlab-org/gitlab-ce
- //@gitlab-org/gitlab-ee
......
......@@ -178,6 +178,9 @@ module Projects
def latest_sha
project.commit(build.ref).try(:sha).to_s
ensure
# Close any file descriptors that were opened and free libgit2 buffers
project.cleanup
end
def sha
......
---
title: Fixed gitlab:uploads:migrate task ignoring some uploads.
merge_request: 18082
author:
type: fixed
---
title: Free open file descriptors and libgit2 buffers in UpdatePagesService
merge_request:
author:
type: performance
......@@ -35,7 +35,7 @@ are:
A Runner that is specific only runs for the specified project(s). A shared Runner
can run jobs for every project that has enabled the option **Allow shared Runners**
under **Settings CI/CD**.
under **Settings > CI/CD**.
Projects with high demand of CI activity can also benefit from using specific
Runners. By having dedicated Runners you are guaranteed that the Runner is not
......@@ -76,7 +76,7 @@ Registering a specific can be done in two ways:
To create a specific Runner without having admin rights to the GitLab instance,
visit the project you want to make the Runner work for in GitLab:
1. Go to **Settings CI/CD** to obtain the token
1. Go to **Settings > CI/CD** to obtain the token
1. [Register the Runner][register]
### Making an existing shared Runner specific
......@@ -85,7 +85,7 @@ If you are an admin on your GitLab instance, you can turn any shared Runner into
a specific one, but not the other way around. Keep in mind that this is a one
way transition.
1. Go to the Runners in the admin area **Overview Runners** (`/admin/runners`)
1. Go to the Runners in the admin area **Overview > Runners** (`/admin/runners`)
and find your Runner
1. Enable any projects under **Restrict projects for this Runner** to be used
with the Runner
......@@ -101,7 +101,7 @@ can be changed afterwards under each Runner's settings.
To lock/unlock a Runner:
1. Visit your project's **Settings CI/CD**
1. Visit your project's **Settings > CI/CD**
1. Find the Runner you wish to lock/unlock and make sure it's enabled
1. Click the pencil button
1. Check the **Lock to current projects** option
......@@ -115,7 +115,7 @@ you can enable the Runner also on any other project where you have Master permis
To enable/disable a Runner in your project:
1. Visit your project's **Settings CI/CD**
1. Visit your project's **Settings > CI/CD**
1. Find the Runner you wish to enable/disable
1. Click **Enable for this project** or **Disable for this project**
......@@ -124,6 +124,13 @@ Consider that if you don't lock your specific Runner to a specific project, any
user with Master role in you project can assign your runner to another arbitrary
project without requiring your authorization, so use it with caution.
An admin can enable/disable a specific Runner for projects:
1. Navigate to **Admin > Runners**
2. Find the Runner you wish to enable/disable
3. Click edit on the Runner
4. Click **Enable** or **Disable** on the project
## Protected Runners
>
......@@ -136,7 +143,7 @@ Whenever a Runner is protected, the Runner picks only jobs created on
To protect/unprotect Runners:
1. Visit your project's **Settings CI/CD**
1. Visit your project's **Settings > CI/CD**
1. Find a Runner you want to protect/unprotect and make sure it's enabled
1. Click the pencil button besides the Runner name
1. Check the **Protected** option
......
......@@ -5,9 +5,5 @@ module Backup
def initialize
super('artifacts', JobArtifactUploader.root)
end
def create_files_dir
Dir.mkdir(app_files_dir, 0700)
end
end
end
......@@ -5,9 +5,5 @@ module Backup
def initialize
super('builds', Settings.gitlab_ci.builds_path)
end
def create_files_dir
Dir.mkdir(app_files_dir, 0700)
end
end
end
require 'open3'
require_relative 'helper'
module Backup
class Files
include Backup::Helper
attr_reader :name, :app_files_dir, :backup_tarball, :files_parent_dir
def initialize(name, app_files_dir)
......@@ -35,15 +38,22 @@ module Backup
def restore
backup_existing_files_dir
create_files_dir
run_pipeline!([%w(gzip -cd), %W(tar -C #{app_files_dir} -xf -)], in: backup_tarball)
run_pipeline!([%w(gzip -cd), %W(tar --unlink-first --recursive-unlink -C #{app_files_dir} -xf -)], in: backup_tarball)
end
def backup_existing_files_dir
timestamped_files_path = File.join(files_parent_dir, "#{name}.#{Time.now.to_i}")
timestamped_files_path = File.join(Gitlab.config.backup.path, "tmp", "#{name}.#{Time.now.to_i}")
if File.exist?(app_files_dir)
FileUtils.mv(app_files_dir, File.expand_path(timestamped_files_path))
# Move all files in the existing repos directory except . and .. to
# repositories.old.<timestamp> directory
FileUtils.mkdir_p(timestamped_files_path, mode: 0700)
files = Dir.glob(File.join(app_files_dir, "*"), File::FNM_DOTMATCH) - [File.join(app_files_dir, "."), File.join(app_files_dir, "..")]
begin
FileUtils.mv(files, timestamped_files_path)
rescue Errno::EACCES
access_denied_error(app_files_dir)
end
end
end
......
module Backup
module Helper
def access_denied_error(path)
message = <<~EOS
### NOTICE ###
As part of restore, the task tried to move existing content from #{path}.
However, it seems that directory contains files/folders that are not owned
by the user #{Gitlab.config.gitlab.user}. To proceed, please move the files
or folders inside #{path} to a secure location so that #{path} is empty and
run restore task again.
EOS
raise message
end
end
end
......@@ -5,9 +5,5 @@ module Backup
def initialize
super('lfs', Settings.lfs.storage_path)
end
def create_files_dir
Dir.mkdir(app_files_dir, 0700)
end
end
end
......@@ -5,9 +5,5 @@ module Backup
def initialize
super('pages', Gitlab.config.pages.path)
end
def create_files_dir
Dir.mkdir(app_files_dir, 0700)
end
end
end
......@@ -5,9 +5,5 @@ module Backup
def initialize
super('registry', Settings.registry.path)
end
def create_files_dir
Dir.mkdir(app_files_dir, 0700)
end
end
end
require 'yaml'
require_relative 'helper'
module Backup
class Repository
include Backup::Helper
# rubocop:disable Metrics/AbcSize
def dump
prepare
......@@ -63,18 +66,27 @@ module Backup
end
end
def restore
def prepare_directories
Gitlab.config.repositories.storages.each do |name, repository_storage|
path = repository_storage.legacy_disk_path
next unless File.exist?(path)
# Move repos dir to 'repositories.old' dir
bk_repos_path = File.join(path, '..', 'repositories.old.' + Time.now.to_i.to_s)
FileUtils.mv(path, bk_repos_path)
# This is expected from gitlab:check
FileUtils.mkdir_p(path, mode: 02770)
# Move all files in the existing repos directory except . and .. to
# repositories.old.<timestamp> directory
bk_repos_path = File.join(Gitlab.config.backup.path, "tmp", "#{name}-repositories.old." + Time.now.to_i.to_s)
FileUtils.mkdir_p(bk_repos_path, mode: 0700)
files = Dir.glob(File.join(path, "*"), File::FNM_DOTMATCH) - [File.join(path, "."), File.join(path, "..")]
begin
FileUtils.mv(files, bk_repos_path)
rescue Errno::EACCES
access_denied_error(path)
end
end
end
def restore
prepare_directories
Project.find_each(batch_size: 1000) do |project|
progress.print " * #{display_repo_path(project)} ... "
path_to_project_repo = path_to_repo(project)
......
......@@ -5,9 +5,5 @@ module Backup
def initialize
super('uploads', Rails.root.join('public/uploads'))
end
def create_files_dir
Dir.mkdir(app_files_dir)
end
end
end
......@@ -25,8 +25,8 @@ namespace :gitlab do
Upload.class_eval { include EachBatch } unless Upload < EachBatch
Upload
.where.not(store: @to_store)
.where(uploader: @uploader_class.to_s,
.where(store: [nil, ObjectStorage::Store::LOCAL],
uploader: @uploader_class.to_s,
model_type: @model_class.base_class.sti_name)
end
end
......
......@@ -8,8 +8,8 @@ msgid ""
msgstr ""
"Project-Id-Version: gitlab 1.0.0\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2018-03-27 14:40+0300\n"
"PO-Revision-Date: 2018-03-27 14:40+0300\n"
"POT-Creation-Date: 2018-03-30 17:57+0200\n"
"PO-Revision-Date: 2018-03-30 17:57+0200\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"Language: \n"
......@@ -148,6 +148,9 @@ msgstr ""
msgid "Account"
msgstr ""
msgid "Account and limit settings"
msgstr ""
msgid "Active"
msgstr ""
......@@ -385,6 +388,12 @@ msgstr ""
msgid "Assign to"
msgstr ""
msgid "Assigned Issues"
msgstr ""
msgid "Assigned Merge Requests"
msgstr ""
msgid "Assigned to :name"
msgstr ""
......@@ -412,6 +421,9 @@ msgstr ""
msgid "Auto DevOps enabled"
msgstr ""
msgid "Auto DevOps, runners amd job artifacts"
msgstr ""
msgid "Auto Review Apps and Auto Deploy need a %{kubernetes} to work correctly."
msgstr ""
......@@ -1285,6 +1297,9 @@ msgstr ""
msgid "Confidentiality"
msgstr ""
msgid "Configure the way a user creates a new account."
msgstr ""
msgid "Connect"
msgstr ""
......@@ -1345,6 +1360,9 @@ msgstr ""
msgid "ContainerRegistry|With the Docker Container Registry integrated into GitLab, every project can have its own space to store its Docker images."
msgstr ""
msgid "Continuous Integration and Deployment"
msgstr ""
msgid "Contribution"
msgstr ""
......@@ -1414,9 +1432,6 @@ msgstr ""
msgid "Create empty repository"
msgstr ""
msgid "Create file"
msgstr ""
msgid "Create epic"
msgstr ""
......@@ -1629,6 +1644,12 @@ msgstr ""
msgid "Enable Auto DevOps"
msgstr ""
msgid "Enable and configure InfluxDB metrics."
msgstr ""
msgid "Enable and configure Prometheus metrics."
msgstr ""
msgid "Enable classification control using an external service"
msgstr ""
......@@ -1949,9 +1970,6 @@ msgstr ""
msgid "GeoNodes|Replication slots:"
msgstr ""
msgid "GroupsTree|Create a project in this group."
msgstr ""
msgid "GeoNodes|Repositories checksummed:"
msgstr ""
......@@ -1997,14 +2015,6 @@ msgstr ""
msgid "GeoNodes|Wiki checksums verified:"
msgstr ""
msgid "Help"
msgstr ""
msgid "Hide value"
msgid_plural "Hide values"
msgstr[0] ""
msgstr[1] ""
msgid "GeoNodes|Wikis checksummed:"
msgstr ""
......@@ -2053,6 +2063,9 @@ msgstr ""
msgid "GitHub import"
msgstr ""
msgid "GitLab CI Linter has been moved"
msgstr ""
msgid "GitLab Runner section"
msgstr ""
......@@ -2179,6 +2192,12 @@ msgstr ""
msgid "Help"
msgstr ""
msgid "Help page"
msgstr ""
msgid "Help page text and support page url."
msgstr ""
msgid "Hide value"
msgid_plural "Hide values"
msgstr[0] ""
......@@ -2348,10 +2367,10 @@ msgstr ""
msgid "Labels can be applied to issues and merge requests to categorize them."
msgstr ""
msgid "Labels|Promote Label"
msgid "Labels|<span>Promote label</span> %{labelTitle} <span>to Group Label?</span>"
msgstr ""
msgid "Labels|Promote label %{labelTitle} to Group Label?"
msgid "Labels|Promote Label"
msgstr ""
msgid "Last %d day"
......@@ -2500,6 +2519,12 @@ msgstr ""
msgid "Messages"
msgstr ""
msgid "Metrics - Influx"
msgstr ""
msgid "Metrics - Prometheus"
msgstr ""
msgid "Metrics|Business"
msgstr ""
......@@ -2590,6 +2615,9 @@ msgstr ""
msgid "Milestones|Promote Milestone"
msgstr ""
msgid "Milestones|This action cannot be reversed."
msgstr ""
msgid "MissingSSHKeyWarningLink|add an SSH key"
msgstr ""
......@@ -2859,6 +2887,9 @@ msgstr ""
msgid "Owner"
msgstr ""
msgid "Pages"
msgstr ""
msgid "Pagination|Last »"
msgstr ""
......@@ -3524,9 +3555,18 @@ msgstr ""
msgid "Service URL"
msgstr ""
msgid "Session expiration, projects limit and attachment size."
msgstr ""
msgid "Set a password on your account to pull or push via %{protocol}."
msgstr ""
msgid "Set default and restrict visibility levels. Configure import sources and git access protocol."
msgstr ""
msgid "Set requirements for a user to sign-in. Enable mandatory two-factor authentication."
msgstr ""
msgid "Set up CI/CD"
msgstr ""
......@@ -3577,12 +3617,24 @@ msgstr ""
msgid "Sidebar|Weight"
msgstr ""
msgid "Sign-in restrictions"
msgstr ""
msgid "Sign-up restrictions"
msgstr ""
msgid "Size and domain settings for static websites"
msgstr ""
msgid "Snippets"
msgstr ""
msgid "Something went wrong on our end"
msgstr ""
msgid "Something went wrong on our end."
msgstr ""
msgid "Something went wrong trying to change the confidentiality of this issue"
msgstr ""
......@@ -3595,9 +3647,6 @@ msgstr ""
msgid "Something went wrong while fetching Dependency Scanning."
msgstr ""
msgid "Something went wrong while fetching the projects."
msgstr ""
msgid "Something went wrong while fetching SAST."
msgstr ""
......@@ -3730,6 +3779,12 @@ msgstr ""
msgid "StarProject|Star"
msgstr ""
msgid "Starred Projects"
msgstr ""
msgid "Starred Projects' Activity"
msgstr ""
msgid "Starred projects"
msgstr ""
......@@ -3957,6 +4012,9 @@ msgstr ""
msgid "There are problems accessing Git storage: "
msgstr ""
msgid "There was an error loading results"
msgstr ""
msgid "There was an error loading users activity calendar."
msgstr ""
......@@ -4228,6 +4286,9 @@ msgstr ""
msgid "To only use CI/CD features for an external repository, choose <strong>CI/CD for external repo</strong>."
msgstr ""
msgid "To validate your GitLab CI configurations, go to 'CI/CD → Pipelines' inside your project, and click on the 'CI Lint' button."
msgstr ""
msgid "To view the roadmap, add a planned start or finish date to one of your epics in this group or its subgroups. Only epics in the past 3 months and the next 3 months are shown."
msgstr ""
......@@ -4354,6 +4415,9 @@ msgstr ""
msgid "View replaced file @ "
msgstr ""
msgid "Visibility and access controls"
msgstr ""
msgid "VisibilityLevel|Internal"
msgstr ""
......@@ -4600,9 +4664,21 @@ msgstr ""
msgid "You're receiving this email because of your account on %{host}. %{manage_notifications_link} &middot; %{help_link}"
msgstr ""
msgid "Your Groups"
msgstr ""
msgid "Your Kubernetes cluster information on this page is still editable, but you are advised to disable and reconfigure"
msgstr ""
msgid "Your Projects (default)"
msgstr ""
msgid "Your Projects' Activity"
msgstr ""
msgid "Your Todos"
msgstr ""
msgid "Your changes can be committed to %{branch_name} because a merge request is open."
msgstr ""
......@@ -4624,6 +4700,11 @@ msgstr ""
msgid "among other things"
msgstr ""
msgid "and %d fixed vulnerability"
msgid_plural "and %d fixed vulnerabilities"
msgstr[0] ""
msgstr[1] ""
msgid "assign yourself"
msgstr ""
......@@ -4633,6 +4714,12 @@ msgstr ""
msgid "by"
msgstr ""
msgid "ciReport|%{type} detected no new security vulnerabilities"
msgstr ""
msgid "ciReport|%{type} detected no security vulnerabilities"
msgstr ""
msgid "ciReport|Code quality"
msgstr ""
......@@ -4690,6 +4777,12 @@ msgstr ""
msgid "ciReport|SAST:container no vulnerabilities were found"
msgstr ""
msgid "ciReport|Security scanning"
msgstr ""
msgid "ciReport|Security scanning failed loading any results"
msgstr ""
msgid "ciReport|Show complete code vulnerabilities report"
msgstr ""
......@@ -4719,6 +4812,19 @@ msgid_plural "days"
msgstr[0] ""
msgstr[1] ""
msgid "detected %d fixed vulnerability"
msgid_plural "detected %d fixed vulnerabilities"
msgstr[0] ""
msgstr[1] ""
msgid "detected %d new vulnerability"
msgid_plural "detected %d new vulnerabilities"
msgstr[0] ""
msgstr[1] ""
msgid "detected no vulnerabilities"
msgstr ""
msgid "estimateCommand|%{slash_command} will update the estimated time with the latest command."
msgstr ""
......@@ -4728,6 +4834,9 @@ msgstr ""
msgid "importing"
msgstr ""
msgid "in progress"
msgstr ""
msgid "is invalid because there is downstream lock"
msgstr ""
......
require 'spec_helper'
describe Backup::Files do
let(:progress) { StringIO.new }
let!(:project) { create(:project) }
before do
allow(progress).to receive(:puts)
allow(progress).to receive(:print)
allow(FileUtils).to receive(:mkdir_p).and_return(true)
allow(FileUtils).to receive(:mv).and_return(true)
allow(File).to receive(:exist?).and_return(true)
allow(File).to receive(:realpath).with("/var/gitlab-registry").and_return("/var/gitlab-registry")
allow(File).to receive(:realpath).with("/var/gitlab-registry/..").and_return("/var")
allow_any_instance_of(String).to receive(:color) do |string, _color|
string
end
allow_any_instance_of(described_class).to receive(:progress).and_return(progress)
end
describe '#restore' do
subject { described_class.new('registry', '/var/gitlab-registry') }
let(:timestamp) { Time.utc(2017, 3, 22) }
around do |example|
Timecop.freeze(timestamp) { example.run }
end
describe 'folders with permission' do
before do
allow(subject).to receive(:run_pipeline!).and_return(true)
allow(subject).to receive(:backup_existing_files).and_return(true)
allow(Dir).to receive(:glob).with("/var/gitlab-registry/*", File::FNM_DOTMATCH).and_return(["/var/gitlab-registry/.", "/var/gitlab-registry/..", "/var/gitlab-registry/sample1"])
end
it 'moves all necessary files' do
allow(subject).to receive(:backup_existing_files).and_call_original
expect(FileUtils).to receive(:mv).with(["/var/gitlab-registry/sample1"], File.join(Gitlab.config.backup.path, "tmp", "registry.#{Time.now.to_i}"))
subject.restore
end
it 'raises no errors' do
expect { subject.restore }.not_to raise_error
end
it 'calls tar command with unlink' do
expect(subject).to receive(:run_pipeline!).with([%w(gzip -cd), %w(tar --unlink-first --recursive-unlink -C /var/gitlab-registry -xf -)], any_args)
subject.restore
end
end
describe 'folders without permissions' do
before do
allow(FileUtils).to receive(:mv).and_raise(Errno::EACCES)
allow(subject).to receive(:run_pipeline!).and_return(true)
end
it 'shows error message' do
expect(subject).to receive(:access_denied_error).with("/var/gitlab-registry")
subject.restore
end
end
end
end
......@@ -7,6 +7,8 @@ describe Backup::Repository do
before do
allow(progress).to receive(:puts)
allow(progress).to receive(:print)
allow(FileUtils).to receive(:mkdir_p).and_return(true)
allow(FileUtils).to receive(:mv).and_return(true)
allow_any_instance_of(String).to receive(:color) do |string, _color|
string
......@@ -68,6 +70,17 @@ describe Backup::Repository do
end
end
end
describe 'folders without permissions' do
before do
allow(FileUtils).to receive(:mv).and_raise(Errno::EACCES)
end
it 'shows error message' do
expect(subject).to receive(:access_denied_error)
subject.restore
end
end
end
describe '#empty_repo?' do
......
......@@ -20,9 +20,23 @@ describe 'gitlab:uploads:migrate rake tasks' do
run_rake_task("gitlab:uploads:migrate", *args)
end
it 'enqueue jobs in batch' do
expect(ObjectStorage::MigrateUploadsWorker).to receive(:enqueue!).exactly(4).times
shared_examples 'enqueue jobs in batch' do |batch:|
it do
expect(ObjectStorage::MigrateUploadsWorker)
.to receive(:perform_async).exactly(batch).times
.and_return("A fake job.")
run
run
end
end
it_behaves_like 'enqueue jobs in batch', batch: 4
context 'Upload has store = nil' do
before do
Upload.where(model: projects).update_all(store: nil)
end
it_behaves_like 'enqueue jobs in batch', batch: 4
end
end
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment