Merge branch 'master' into sh-headless-chrome-support

.gitlab/merge_request_templates/Database Changes.md

+Remove this section and replace it with a description of your MR. Also follow the
+checklist below and check off any tasks that are done. If a certain task can not
+be done you should explain so in the MR body. You are free to remove any
+sections that do not apply to your MR.
+
+When gathering statistics (e.g. the output of `EXPLAIN ANALYZE`) you should make
+sure your database has enough data. Having around 10 000 rows in the tables
+being queries should provide a reasonable estimate of how a query will behave.
+Also make sure that PostgreSQL uses the following settings:
+
+* `random_page_cost`: `1`
+* `work_mem`: `16MB`
+* `maintenance_work_mem`: at least `64MB`
+* `shared_buffers`: at least `256MB`
+
+If you have access to GitLab.com's staging environment you should also run your
+measurements there, and include the results in this MR.
+
+## Database Checklist
+
+When adding migrations:
+
+- [ ] Updated `db/schema.rb`
+- [ ] Added a `down` method so the migration can be reverted
+- [ ] Added the output of the migration(s) to the MR body
+- [ ] Added the execution time of the migration(s) to the MR body
+- [ ] Added tests for the migration in `spec/migrations` if necessary (e.g. when
+  migrating data)
+- [ ] Made sure the migration won't interfere with a running GitLab cluster,
+  for example by disabling transactions for long running migrations
+
+When adding or modifying queries to improve performance:
+
+- [ ] Included the raw SQL queries of the relevant queries
+- [ ] Included the output of `EXPLAIN ANALYZE` and execution timings of the
+  relevant queries
+- [ ] Added tests for the relevant changes
+
+When adding indexes:
+
+- [ ] Described the need for these indexes in the MR body
+- [ ] Made sure existing indexes can not be reused instead
+
+When adding foreign keys to existing tables:
+
+- [ ] Included a migration to remove orphaned rows in the source table
+- [ ] Removed any instances of `dependent: ...` that may no longer be necessary
+
+When adding tables:
+
+- [ ] Ordered columns based on their type sizes in descending order
+- [ ] Added foreign keys if necessary
+- [ ] Added indexes if necessary
+
+When removing columns, tables, indexes or other structures:
+
+- [ ] Removed these in a post-deployment migration
+- [ ] Made sure the application no longer uses (or ignores) these structures
+
+## General Checklist
+
+- [ ] [Changelog entry](https://docs.gitlab.com/ce/development/changelog.html) added, if necessary
+- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
+- [ ] API support added
+- [ ] Tests added for this feature/bug
+- Review
+  - [ ] Has been reviewed by UX
+  - [ ] Has been reviewed by Frontend
+  - [ ] Has been reviewed by Backend
+  - [ ] Has been reviewed by Database
+- [ ] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
+- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
+- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

.rubocop.yml

@@ -1045,7 +1045,7 @@ RSpec/BeforeAfterAll:
 RSpec/DescribeClass:
   Enabled: false
 
-# Use `described_class` for tested class / module.
+# Checks that the second argument to `describe` specifies a method.
 RSpec/DescribeMethod:
   Enabled: false
 
@@ -1053,8 +1053,7 @@ RSpec/DescribeMethod:
 RSpec/DescribeSymbol:
   Enabled: true
 
-# Checks that the second argument to top level describe is the tested method
-# name.
+# Checks that tests use `described_class`.
 RSpec/DescribedClass:
   Enabled: true
 
@@ -1099,6 +1098,11 @@ RSpec/FilePath:
 RSpec/Focus:
   Enabled: true
 
+# Checks the arguments passed to `before`, `around`, and `after`.
+RSpec/HookArgument:
+  Enabled: true
+  EnforcedStyle: implicit
+
 # Configuration parameters: EnforcedStyle, SupportedStyles.
 # SupportedStyles: is_expected, should
 RSpec/ImplicitExpect:

.rubocop_todo.yml

@@ -70,12 +70,6 @@ RSpec/EmptyLineAfterFinalLet:
 RSpec/EmptyLineAfterSubject:
   Enabled: false
 
-# Offense count: 78
-# Configuration parameters: EnforcedStyle, SupportedStyles.
-# SupportedStyles: implicit, each, example
-RSpec/HookArgument:
-  Enabled: false
-
 # Offense count: 9
 # Configuration parameters: EnforcedStyle, SupportedStyles.
 # SupportedStyles: it_behaves_like, it_should_behave_like

CHANGELOG.md

@@ -2,6 +2,11 @@
 documentation](doc/development/changelog.md) for instructions on adding your own
 entry.
 
+## 9.4.4 (2017-08-09)
+
+- Remove hidden symlinks from project import files.
+- Disallow Git URLs that include a username or hostname beginning with a non-alphanumeric character.
+
 ## 9.4.3 (2017-07-31)
 
 - Fix Prometheus client PID reuse bug. !13130
@@ -226,6 +231,11 @@ entry.
 - Log rescued exceptions to Sentry.
 - Remove remaining N+1 queries in merge requests API with emojis and labels.
 
+## 9.3.10 (2017-08-09)
+
+- Remove hidden symlinks from project import files.
+- Disallow Git URLs that include a username or hostname beginning with a non-alphanumeric character.
+
 ## 9.3.9 (2017-07-20)
 
 - Fix an infinite loop when handling user-supplied regular expressions.
@@ -498,6 +508,11 @@ entry.
 - Remove foreigh key on ci_trigger_schedules only if it exists.
 - Allow translation of Pipeline Schedules.
 
+## 9.2.10 (2017-08-09)
+
+- Remove hidden symlinks from project import files.
+- Disallow Git URLs that include a username or hostname beginning with a non-alphanumeric character.
+
 ## 9.2.9 (2017-07-20)
 
 - Fix an infinite loop when handling user-supplied regular expressions.
@@ -753,6 +768,11 @@ entry.
 - Fix preemptive scroll bar on user activity calendar.
 - Pipeline chat notifications convert seconds to minutes and hours.
 
+## 9.1.10 (2017-08-09)
+
+- Remove hidden symlinks from project import files.
+- Disallow Git URLs that include a username or hostname beginning with a non-alphanumeric character.
+
 ## 9.1.9 (2017-07-20)
 
 - Fix an infinite loop when handling user-supplied regular expressions.
@@ -1076,6 +1096,11 @@ entry.
 - Only send chat notifications for the default branch.
 - Don't fill in the default kubernetes namespace.
 
+## 9.0.13 (2017-08-09)
+
+- Remove hidden symlinks from project import files.
+- Disallow Git URLs that include a username or hostname beginning with a non-alphanumeric character.
+
 ## 9.0.12 (2017-07-20)
 
 - Fix an infinite loop when handling user-supplied regular expressions.
@@ -1456,6 +1481,11 @@ entry.
 - Change development tanuki favicon colors to match logo color order.
 - API issues - support filtering by iids.
 
+## 8.17.8 (2017-08-09)
+
+- Remove hidden symlinks from project import files.
+- Disallow Git URLs that include a username or hostname beginning with a non-alphanumeric character.
+
 ## 8.17.7 (2017-07-19)
 
 - Renders 404 if given project is not readable by the user on Todos dashboard.

GITALY_SERVER_VERSION

-0.29.0
+0.30.0

Gemfile

@@ -64,7 +64,7 @@ gem 'gpgme'
 # LDAP Auth
 # GitLab fork with several improvements to original library. For full list of changes
 # see https://github.com/intridea/omniauth-ldap/compare/master...gitlabhq:master
-gem 'gitlab_omniauth-ldap', '~> 2.0.3', require: 'omniauth-ldap'
+gem 'gitlab_omniauth-ldap', '~> 2.0.4', require: 'omniauth-ldap'
 gem 'net-ldap'
 
 # Git Wiki
@@ -402,7 +402,7 @@ group :ed25519 do
 end
 
 # Gitaly GRPC client
-gem 'gitaly', '~> 0.26.0'
+gem 'gitaly', '~> 0.27.0'
 
 gem 'toml-rb', '~> 0.3.15', require: false
 

Gemfile.lock

@@ -271,7 +271,7 @@ GEM
       po_to_json (>= 1.0.0)
       rails (>= 3.2.0)
     gherkin-ruby (0.3.2)
-    gitaly (0.26.0)
+    gitaly (0.27.0)
       google-protobuf (~> 3.1)
       grpc (~> 1.0)
     github-linguist (4.7.6)
@@ -290,7 +290,7 @@ GEM
       mime-types (>= 1.16, < 3)
       posix-spawn (~> 0.3)
     gitlab-markup (1.5.1)
-    gitlab_omniauth-ldap (2.0.3)
+    gitlab_omniauth-ldap (2.0.4)
       net-ldap (~> 0.16)
       omniauth (~> 1.3)
       pyu-ruby-sasl (>= 0.0.3.3, < 0.1)
@@ -740,6 +740,7 @@ GEM
     rubocop-gitlab-security (0.0.6)
       rubocop (>= 0.47.1)
     rubocop-rspec (1.15.1)
+      rubocop (>= 0.42.0)
     ruby-fogbugz (0.2.1)
       crack (~> 0.4)
     ruby-prof (0.16.2)
@@ -981,11 +982,11 @@ DEPENDENCIES
   gettext (~> 3.2.2)
   gettext_i18n_rails (~> 1.8.0)
   gettext_i18n_rails_js (~> 1.2.0)
-  gitaly (~> 0.26.0)
+  gitaly (~> 0.27.0)
   github-linguist (~> 4.7.0)
   gitlab-flowdock-git-hook (~> 1.0.1)
   gitlab-markup (~> 1.5.1)
-  gitlab_omniauth-ldap (~> 2.0.3)
+  gitlab_omniauth-ldap (~> 2.0.4)
   gollum-lib (~> 4.2)
   gollum-rugged_adapter (~> 0.4.4)
   gon (~> 6.1.0)

app/assets/javascripts/boards/components/modal/list.js

 /* global ListIssue */
-/* global bp */
 
 import Vue from 'vue';
+import bp from '../../../breakpoints';
 
 const ModalStore = gl.issueBoards.ModalStore;
 

app/assets/javascripts/breakpoints.js

-/* eslint-disable func-names, space-before-function-paren, wrap-iife, one-var, no-var, one-var-declaration-per-line, quotes, no-shadow, prefer-arrow-callback, prefer-template, consistent-return, no-return-assign, new-parens, no-param-reassign, max-len */
+export const breakpoints = {
+  lg: 1200,
+  md: 992,
+  sm: 768,
+  xs: 0,
+};
 
-var Breakpoints = (function() {
-  var BreakpointInstance, instance;
+const BreakpointInstance = {
+  windowWidth: () => window.innerWidth,
+  getBreakpointSize() {
+    const windowWidth = this.windowWidth();
 
-  function Breakpoints() {}
+    const breakpoint = Object.keys(breakpoints).find(key => windowWidth > breakpoints[key]);
 
-  instance = null;
+    return breakpoint;
+  },
+};
 
-  BreakpointInstance = (function() {
-    var BREAKPOINTS;
-
-    BREAKPOINTS = ["xs", "sm", "md", "lg"];
-
-    function BreakpointInstance() {
-      this.setup();
-    }
-
-    BreakpointInstance.prototype.setup = function() {
-      var allDeviceSelector, els;
-      allDeviceSelector = BREAKPOINTS.map(function(breakpoint) {
-        return ".device-" + breakpoint;
-      });
-      if ($(allDeviceSelector.join(",")).length) {
-        return;
-      }
-      // Create all the elements
-      els = $.map(BREAKPOINTS, function(breakpoint) {
-        return "<div class='device-" + breakpoint + " visible-" + breakpoint + "'></div>";
-      });
-      return $("body").append(els.join(''));
-    };
-
-    BreakpointInstance.prototype.visibleDevice = function() {
-      var allDeviceSelector;
-      allDeviceSelector = BREAKPOINTS.map(function(breakpoint) {
-        return ".device-" + breakpoint;
-      });
-      return $(allDeviceSelector.join(",")).filter(":visible");
-    };
-
-    BreakpointInstance.prototype.getBreakpointSize = function() {
-      var $visibleDevice;
-      $visibleDevice = this.visibleDevice;
-      // TODO: Consider refactoring in light of turbolinks removal.
-      // the page refreshed via turbolinks
-      if (!$visibleDevice().length) {
-        this.setup();
-      }
-      $visibleDevice = this.visibleDevice();
-      return $visibleDevice.attr("class").split("visible-")[1];
-    };
-
-    return BreakpointInstance;
-  })();
-
-  Breakpoints.get = function() {
-    return instance != null ? instance : instance = new BreakpointInstance;
-  };
-
-  return Breakpoints;
-})();
-
-$(() => { window.bp = Breakpoints.get(); });
-
-window.Breakpoints = Breakpoints;
+export default BreakpointInstance;

app/assets/javascripts/build.js

 /* eslint-disable func-names, wrap-iife, no-use-before-define,
 consistent-return, prefer-rest-params */
-/* global Breakpoints */
-
 import _ from 'underscore';
+import bp from './breakpoints';
 import { bytesToKiB } from './lib/utils/number_utils';
 
 window.Build = (function () {
@@ -34,8 +33,6 @@ window.Build = (function () {
     this.$scrollBottomBtn = $('.js-scroll-down');
 
     clearTimeout(Build.timeout);
-    // Init breakpoint checker
-    this.bp = Breakpoints.get();
 
     this.initSidebar();
     this.populateJobs(this.buildStage);
@@ -230,7 +227,7 @@ window.Build = (function () {
   };
 
   Build.prototype.shouldHideSidebarForViewport = function () {
-    const bootstrapBreakpoint = this.bp.getBreakpointSize();
+    const bootstrapBreakpoint = bp.getBreakpointSize();
     return bootstrapBreakpoint === 'xs' || bootstrapBreakpoint === 'sm';
   };
 

app/assets/javascripts/dispatcher.js

@@ -76,6 +76,7 @@ import initLegacyFilters from './init_legacy_filters';
 import initIssuableSidebar from './init_issuable_sidebar';
 import GpgBadges from './gpg_badges';
 import UserFeatureHelper from './helpers/user_feature_helper';
+import initChangesDropdown from './init_changes_dropdown';
 
 (function() {
   var Dispatcher;
@@ -228,6 +229,7 @@ import UserFeatureHelper from './helpers/user_feature_helper';
           break;
         case 'projects:compare:show':
           new gl.Diff();
+          initChangesDropdown();
           break;
         case 'projects:branches:new':
         case 'projects:branches:create':
@@ -320,6 +322,7 @@ import UserFeatureHelper from './helpers/user_feature_helper';
             container: '.js-commit-pipeline-graph',
           }).bindEvents();
           initNotes();
+          initChangesDropdown();
           $('.commit-info.branches').load(document.querySelector('.js-commit-box').dataset.commitPath);
           break;
         case 'projects:commit:pipelines':

app/assets/javascripts/fly_out_nav.js

-/* global bp */
 import Cookies from 'js-cookie';
-import './breakpoints';
+import bp from './breakpoints';
 
 export const canShowActiveSubItems = (el) => {
   const isHiddenByMedia = bp.getBreakpointSize() === 'sm' || bp.getBreakpointSize() === 'md';
@@ -49,7 +48,8 @@ export const hideSubLevelItems = (el) => {
 
   el.classList.remove('is-showing-fly-out');
   el.classList.remove('is-over');
-  subItems.style.display = 'none';
+  subItems.style.display = '';
+  subItems.style.transform = '';
   subItems.classList.remove('is-above');
 };
 

app/assets/javascripts/init_changes_dropdown.js

+import stickyMonitor from './lib/utils/sticky';
+
+export default () => {
+  stickyMonitor(document.querySelector('.js-diff-files-changed'));
+
+  $('.js-diff-stats-dropdown').glDropdown({
+    filterable: true,
+    remoteFilter: false,
+  });
+};

app/assets/javascripts/issuable_context.js

 /* eslint-disable func-names, space-before-function-paren, wrap-iife, no-new, comma-dangle, quotes, prefer-arrow-callback, consistent-return, one-var, no-var, one-var-declaration-per-line, no-underscore-dangle, max-len */
-/* global bp */
-
 import Cookies from 'js-cookie';
+import bp from './breakpoints';
 import UsersSelect from './users_select';
 
 const PARTICIPANTS_ROW_COUNT = 7;

app/assets/javascripts/jobs/components/header.vue

@@ -40,7 +40,7 @@
             label: 'New issue',
             path: this.job.new_issue_path,
             cssClass: 'js-new-issue btn btn-new btn-inverted visible-md-block visible-lg-block',
-            type: 'ujs-link',
+            type: 'link',
           });
         }
 

app/assets/javascripts/main.js

 /* eslint-disable func-names, space-before-function-paren, no-var, quotes, consistent-return, prefer-arrow-callback, comma-dangle, object-shorthand, no-new, max-len, no-multi-spaces, import/newline-after-import, import/first */
-/* global bp */
 /* global Flash */
 /* global ConfirmDangerModal */
 /* global Aside */
@@ -66,7 +65,7 @@ import './api';
 import './aside';
 import './autosave';
 import loadAwardsHandler from './awards_handler';
-import './breakpoints';
+import bp from './breakpoints';
 import './broadcast_message';
 import './build';
 import './build_artifacts';

app/assets/javascripts/merge_request_tabs.js

 /* eslint-disable no-new, class-methods-use-this */
-/* global Breakpoints */
 /* global Flash */
 /* global notes */
 
 import Cookies from 'js-cookie';
-import './breakpoints';
 import './flash';
 import BlobForkSuggestion from './blob/blob_fork_suggestion';
-import stickyMonitor from './lib/utils/sticky';
+import initChangesDropdown from './init_changes_dropdown';
+import bp from './breakpoints';
 
 /* eslint-disable max-len */
 // MergeRequestTabs
@@ -134,7 +133,7 @@ import stickyMonitor from './lib/utils/sticky';
         this.destroyPipelinesView();
       } else if (this.isDiffAction(action)) {
         this.loadDiff($target.attr('href'));
-        if (Breakpoints.get().getBreakpointSize() !== 'lg') {
+        if (bp.getBreakpointSize() !== 'lg') {
           this.shrinkView();
         }
         if (this.diffViewType() === 'parallel') {
@@ -145,7 +144,7 @@ import stickyMonitor from './lib/utils/sticky';
         this.resetViewContainer();
         this.mountPipelinesView();
       } else {
-        if (Breakpoints.get().getBreakpointSize() !== 'xs') {
+        if (bp.getBreakpointSize() !== 'xs') {
           this.expandView();
         }
         this.resetViewContainer();
@@ -267,9 +266,7 @@ import stickyMonitor from './lib/utils/sticky';
           const $container = $('#diffs');
           $container.html(data.html);
 
-          this.initChangesDropdown();
-
-          stickyMonitor(document.querySelector('.js-diff-files-changed'));
+          initChangesDropdown();
 
           if (typeof gl.diffNotesCompileComponents !== 'undefined') {
             gl.diffNotesCompileComponents();
@@ -319,13 +316,6 @@ import stickyMonitor from './lib/utils/sticky';
       });
     }
 
-    initChangesDropdown() {
-      $('.js-diff-stats-dropdown').glDropdown({
-        filterable: true,
-        remoteFilter: false,
-      });
-    }
-
     // Show or hide the loading spinner
     //
     // status - Boolean, true to show, false to hide
@@ -401,7 +391,7 @@ import stickyMonitor from './lib/utils/sticky';
 
       // Screen space on small screens is usually very sparse
       // So we dont affix the tabs on these
-      if (Breakpoints.get().getBreakpointSize() === 'xs' || !$tabs.length) return;
+      if (bp.getBreakpointSize() === 'xs' || !$tabs.length) return;
 
       /**
         If the browser does not support position sticky, it returns the position as static.

app/assets/javascripts/monitoring/components/monitoring_column.vue

 <script>
-  /* global Breakpoints */
   import d3 from 'd3';
   import monitoringLegends from './monitoring_legends.vue';
   import monitoringFlag from './monitoring_flag.vue';
@@ -8,6 +7,7 @@
   import eventHub from '../event_hub';
   import measurements from '../utils/measurements';
   import { formatRelevantDigits } from '../../lib/utils/number_utils';
+  import bp from '../../breakpoints';
 
   const bisectDate = d3.bisector(d => d.time).left;
 
@@ -42,7 +42,6 @@
         yScale: {},
         margin: {},
         data: [],
-        breakpointHandler: Breakpoints.get(),
         unitOfDisplay: '',
         areaColorRgb: '#8fbce8',
         lineColorRgb: '#1f78d1',
@@ -96,7 +95,7 @@
 
     methods: {
       draw() {
-        const breakpointSize = this.breakpointHandler.getBreakpointSize();
+        const breakpointSize = bp.getBreakpointSize();
         const query = this.columnData.queries[0];
         this.margin = measurements.large.margin;
         if (breakpointSize === 'xs' || breakpointSize === 'sm') {

app/assets/javascripts/new_sidebar.js

 import Cookies from 'js-cookie';
 import _ from 'underscore';
-/* global bp */
-import './breakpoints';
+import bp from './breakpoints';
 
 export default class NewNavSidebar {
   constructor() {

app/assets/javascripts/projects/project_new.js

@@ -36,7 +36,7 @@ const bindEvents = () => {
 
   $('.how_to_import_link').on('click', (e) => {
     e.preventDefault();
-    $('.how_to_import_link').next('.modal').show();
+    $(e.currentTarget).next('.modal').show();
   });
 
   $('.modal-header .close').on('click', () => {

app/assets/javascripts/sidebar_height_manager.js

 import _ from 'underscore';
+import Cookies from 'js-cookie';
 
 export default {
   init() {
     if (!this.initialized) {
+      if (Cookies.get('new_nav') === 'true' && $('.js-issuable-sidebar').length) return;
+
       this.$window = $(window);
       this.$rightSidebar = $('.js-right-sidebar');
       this.$navHeight = $('.navbar-gitlab').outerHeight() +

app/assets/javascripts/vue_shared/components/header_ci_component.vue

@@ -120,7 +120,7 @@ export default {
         </a>
 
         <a
-          v-if="action.type === 'ujs-link'"
+          v-else-if="action.type === 'ujs-link'"
           :href="action.path"
           data-method="post"
           rel="nofollow"
@@ -129,7 +129,7 @@ export default {
         </a>
 
         <button
-          v-else="action.type === 'button'"
+          v-else-if="action.type === 'button'"
           @click="onClickAction(action)"
           :disabled="action.isLoading"
           :class="action.cssClass"

app/assets/javascripts/wikis.js

-/* global Breakpoints */
-
-import './breakpoints';
+import bp from './breakpoints';
 
 export default class Wikis {
   constructor() {
-    this.bp = Breakpoints.get();
     this.sidebarEl = document.querySelector('.js-wiki-sidebar');
     this.sidebarExpanded = false;
 
@@ -41,15 +38,15 @@ export default class Wikis {
     this.renderSidebar();
   }
 
-  sidebarCanCollapse() {
-    const bootstrapBreakpoint = this.bp.getBreakpointSize();
+  static sidebarCanCollapse() {
+    const bootstrapBreakpoint = bp.getBreakpointSize();
     return bootstrapBreakpoint === 'xs' || bootstrapBreakpoint === 'sm';
   }
 
   renderSidebar() {
     if (!this.sidebarEl) return;
     const { classList } = this.sidebarEl;
-    if (this.sidebarExpanded || !this.sidebarCanCollapse()) {
+    if (this.sidebarExpanded || !Wikis.sidebarCanCollapse()) {
       if (!classList.contains('right-sidebar-expanded')) {
         classList.remove('right-sidebar-collapsed');
         classList.add('right-sidebar-expanded');

app/assets/stylesheets/framework/dropdowns.scss

@@ -725,9 +725,9 @@
 }
 
 // TODO: change global style and remove mixin
-@mixin new-style-dropdown {
-  .dropdown-menu,
-  .dropdown-menu-nav {
+@mixin new-style-dropdown($selector: '') {
+  #{$selector}.dropdown-menu,
+  #{$selector}.dropdown-menu-nav {
     .divider {
       margin: 6px 0;
     }
@@ -773,7 +773,7 @@
     }
   }
 
-  .dropdown-menu-align-right {
+  #{$selector}.dropdown-menu-align-right {
     margin-top: 2px;
   }
 }

app/assets/stylesheets/framework/highlight.scss

@@ -47,7 +47,7 @@
       font-family: $monospace_font;
       display: block;
       font-size: $code_font_size !important;
-      line-height: 19px;
+      min-height: 19px;
       white-space: nowrap;
 
       i {

app/assets/stylesheets/framework/typography.scss

@@ -339,6 +339,8 @@ a > code {
   @extend .ref-name;
 }
 
+@include new-style-dropdown('.git-revision-dropdown');
+
 /**
  * Apply Markdown typography
  *

app/assets/stylesheets/new_sidebar.scss

@@ -103,6 +103,7 @@ $new-sidebar-collapsed-width: 50px;
 
   &.sidebar-icons-only {
     width: $new-sidebar-collapsed-width;
+    overflow-x: hidden;
 
     .badge,
     .project-title {

app/assets/stylesheets/pages/diff.scss

@@ -574,10 +574,14 @@
   @media (min-width: $screen-sm-min) {
     position: -webkit-sticky;
     position: sticky;
-    top: 84px;
+    top: 34px;
     background-color: $white-light;
     z-index: 190;
 
+    &.diff-files-changed-merge-request {
+      top: 84px;
+    }
+
     + .files,
     + .alert {
       margin-top: 1px;

app/assets/stylesheets/pages/note_form.scss

@@ -108,6 +108,7 @@
   background-color: $orange-50;
   border-radius: $border-radius-default $border-radius-default 0 0;
   border: 1px solid $border-gray-normal;
+  border-bottom: none;
   padding: 3px 12px;
   margin: auto;
   align-items: center;
@@ -132,22 +133,9 @@
   }
 }
 
-.not-confidential {
-  padding: 0;
-  border-top: none;
-}
-
-.right-sidebar-expanded {
-  .md-area {
-    border-radius: 0;
-    border-top: none;
-  }
-}
-
-.right-sidebar-collapsed {
-  .confidential-issue-warning {
-    border-bottom: none;
-  }
+.confidential-issue-warning + .md-area {
+  border-top-left-radius: 0;
+  border-top-right-radius: 0;
 }
 
 .discussion-form {

app/assets/stylesheets/pages/projects.scss

@@ -566,14 +566,14 @@ a.deploy-project-label {
     &::before {
       content: "OR";
       position: absolute;
-      left: 0;
-      top: 40%;
+      left: -10px;
+      top: 50%;
       z-index: 10;
       padding: 8px 0;
       text-align: center;
       background-color: $white-light;
       color: $gl-text-color-tertiary;
-      transform: translateX(-50%);
+      transform: translateY(-50%);
       font-size: 12px;
       font-weight: bold;
       line-height: 20px;
@@ -581,8 +581,8 @@ a.deploy-project-label {
       // Mobile
       @media (max-width: $screen-xs-max) {
         left: 50%;
-        top: 10px;
-        transform: translateY(-50%);
+        top: 0;
+        transform: translateX(-50%);
         padding: 0 8px;
       }
     }

app/controllers/admin/appearances_controller.rb

@@ -45,7 +45,7 @@ class Admin::AppearancesController < Admin::ApplicationController
 
   # Use callbacks to share common setup or constraints between actions.
   def set_appearance
-    @appearance = Appearance.last || Appearance.new
+    @appearance = Appearance.current || Appearance.new
   end
 
   # Only allow a trusted parameter "white list" through.

app/controllers/concerns/authenticates_with_two_factor.rb

@@ -69,7 +69,7 @@ module AuthenticatesWithTwoFactor
     if U2fRegistration.authenticate(user, u2f_app_id, user_params[:device_response], session[:challenge])
       # Remove any lingering user data from login
       session.delete(:otp_user_id)
-      session.delete(:challenges)
+      session.delete(:challenge)
 
       remember_me(user) if user_params[:remember_me] == '1'
       sign_in(user)

app/controllers/dashboard/projects_controller.rb

@@ -52,8 +52,10 @@ class Dashboard::ProjectsController < Dashboard::ApplicationController
   end
 
   def load_events
-    @events = Event.in_projects(load_projects(params.merge(non_public: true)))
-    @events = event_filter.apply_filter(@events).with_associations
-    @events = @events.limit(20).offset(params[:offset] || 0)
+    projects = load_projects(params.merge(non_public: true))
+
+    @events = EventCollection
+      .new(projects, offset: params[:offset].to_i, filter: event_filter)
+      .to_a
   end
 end

app/controllers/dashboard_controller.rb

@@ -29,9 +29,9 @@ class DashboardController < Dashboard::ApplicationController
         current_user.authorized_projects
       end
 
-    @events = Event.in_projects(projects)
-    @events = @event_filter.apply_filter(@events).with_associations
-    @events = @events.limit(20).offset(params[:offset] || 0)
+    @events = EventCollection
+      .new(projects, offset: params[:offset].to_i, filter: @event_filter)
+      .to_a
   end
 
   def set_show_full_reference

app/controllers/groups_controller.rb

@@ -160,9 +160,9 @@ class GroupsController < Groups::ApplicationController
   end
 
   def load_events
-    @events = Event.in_projects(@projects)
-    @events = event_filter.apply_filter(@events).with_associations
-    @events = @events.limit(20).offset(params[:offset] || 0)
+    @events = EventCollection
+      .new(@projects, offset: params[:offset].to_i, filter: event_filter)
+      .to_a
   end
 
   def user_actions

app/controllers/projects/issues_controller.rb

@@ -257,18 +257,6 @@ class Projects::IssuesController < Projects::ApplicationController
     return render_404 unless @project.feature_available?(:issues, current_user)
   end
 
-  def redirect_to_external_issue_tracker
-    external = @project.external_issue_tracker
-
-    return unless external
-
-    if action_name == 'new'
-      redirect_to external.new_issue_path
-    else
-      redirect_to external.issue_tracker_path
-    end
-  end
-
   def issue_params
     params.require(:issue).permit(*issue_params_attributes)
   end

app/controllers/projects_controller.rb

@@ -7,6 +7,7 @@ class ProjectsController < Projects::ApplicationController
   before_action :repository, except: [:index, :new, :create]
   before_action :assign_ref_vars, only: [:show], if: :repo_exists?
   before_action :tree, only: [:show], if: [:repo_exists?, :project_view_files?]
+  before_action :project_export_enabled, only: [:export, :download_export, :remove_export, :generate_new_export]
 
   # Authorize
   before_action :authorize_admin_project!, only: [:edit, :update, :housekeeping, :download_export, :export, :remove_export, :generate_new_export]
@@ -301,10 +302,11 @@ class ProjectsController < Projects::ApplicationController
   end
 
   def load_events
-    @events = @project.events.recent
-    @events = event_filter.apply_filter(@events).with_associations
-    limit = (params[:limit] || 20).to_i
-    @events = @events.limit(limit).offset(params[:offset] || 0)
+    projects = Project.where(id: @project.id)
+
+    @events = EventCollection
+      .new(projects, offset: params[:offset].to_i, filter: event_filter)
+      .to_a
   end
 
   def project_params
@@ -389,4 +391,8 @@ class ProjectsController < Projects::ApplicationController
 
     url_for(params)
   end
+
+  def project_export_enabled
+    render_404 unless current_application_settings.project_export_enabled?
+  end
 end

app/controllers/unicorn_test_controller.rb

+# :nocov:
 if Rails.env.test?
   class UnicornTestController < ActionController::Base
     def pid
       render plain: Process.pid.to_s
     end
-  
+
     def kill
       Process.kill(params[:signal], Process.pid)
       render plain: 'Bye!'
     end
   end
 end
+# :nocov:

app/finders/admin/projects_finder.rb

@@ -18,7 +18,7 @@ class Admin::ProjectsFinder
   end
 
   def execute
-    items = Project.with_statistics
+    items = Project.without_deleted.with_statistics
     items = items.in_namespace(namespace_id) if namespace_id.present?
     items = items.where(visibility_level: visibility_level) if visibility_level.present?
     items = items.with_push if with_push.present?

app/helpers/appearances_helper.rb

@@ -20,7 +20,7 @@ module AppearancesHelper
   end
 
   def brand_item
-    @appearance ||= Appearance.first
+    @appearance ||= Appearance.current
   end
 
   def brand_header_logo

app/helpers/application_settings_helper.rb

@@ -146,6 +146,7 @@ module ApplicationSettingsHelper
       :plantuml_enabled,
       :plantuml_url,
       :polling_interval_multiplier,
+      :project_export_enabled,
       :prometheus_metrics_enabled,
       :recaptcha_enabled,
       :recaptcha_private_key,

app/models/appearance.rb

@@ -8,7 +8,27 @@ class Appearance < ActiveRecord::Base
   validates :logo,        file_size: { maximum: 1.megabyte }
   validates :header_logo, file_size: { maximum: 1.megabyte }
 
+  validate :single_appearance_row, on: :create
+
   mount_uploader :logo,         AttachmentUploader
   mount_uploader :header_logo,  AttachmentUploader
   has_many :uploads, as: :model, dependent: :destroy # rubocop:disable Cop/ActiveRecordDependent
+
+  CACHE_KEY = 'current_appearance'.freeze
+
+  after_commit :flush_redis_cache
+
+  def self.current
+    Rails.cache.fetch(CACHE_KEY) { first }
+  end
+
+  def flush_redis_cache
+    Rails.cache.delete(CACHE_KEY)
+  end
+
+  def single_appearance_row
+    if self.class.any?
+      errors.add(:single_appearance_row, 'Only 1 appearances row can exist')
+    end
+  end
 end

app/models/application_setting.rb

@@ -241,6 +241,7 @@ class ApplicationSetting < ActiveRecord::Base
       performance_bar_allowed_group_id: nil,
       plantuml_enabled: false,
       plantuml_url: nil,
+      project_export_enabled: true,
       recaptcha_enabled: false,
       repository_checks_enabled: true,
       repository_storages: ['default'],

app/models/broadcast_message.rb

@@ -14,9 +14,15 @@ class BroadcastMessage < ActiveRecord::Base
   default_value_for :color, '#E75E40'
   default_value_for :font,  '#FFFFFF'
 
+  CACHE_KEY = 'broadcast_message_current'.freeze
+
+  after_commit :flush_redis_cache
+
   def self.current
-    Rails.cache.fetch("broadcast_message_current", expires_in: 1.minute) do
-      where('ends_at > :now AND starts_at <= :now', now: Time.zone.now).order([:created_at, :id]).to_a
+    Rails.cache.fetch(CACHE_KEY) do
+      where('ends_at > :now AND starts_at <= :now', now: Time.zone.now)
+        .reorder(id: :asc)
+        .to_a
     end
   end
 
@@ -31,4 +37,8 @@ class BroadcastMessage < ActiveRecord::Base
   def ended?
     ends_at < Time.zone.now
   end
+
+  def flush_redis_cache
+    Rails.cache.delete(CACHE_KEY)
+  end
 end

app/models/event.rb

@@ -48,6 +48,7 @@ class Event < ActiveRecord::Base
   belongs_to :author, class_name: "User"
   belongs_to :project
   belongs_to :target, polymorphic: true # rubocop:disable Cop/PolymorphicAssociations
+  has_one :push_event_payload, foreign_key: :event_id
 
   # For Hash only
   serialize :data # rubocop:disable Cop/ActiveRecordSerialize
@@ -55,19 +56,51 @@ class Event < ActiveRecord::Base
   # Callbacks
   after_create :reset_project_activity
   after_create :set_last_repository_updated_at, if: :push?
+  after_create :replicate_event_for_push_events_migration
 
   # Scopes
   scope :recent, -> { reorder(id: :desc) }
   scope :code_push, -> { where(action: PUSHED) }
 
-  scope :in_projects, ->(projects) do
-    where(project_id: projects.pluck(:id)).recent
+  scope :in_projects, -> (projects) do
+    sub_query = projects
+      .except(:order)
+      .select(1)
+      .where('projects.id = events.project_id')
+
+    where('EXISTS (?)', sub_query).recent
+  end
+
+  scope :with_associations, -> do
+    # We're using preload for "push_event_payload" as otherwise the association
+    # is not always available (depending on the query being built).
+    includes(:author, :project, project: :namespace)
+      .preload(:target, :push_event_payload)
   end
 
-  scope :with_associations, -> { includes(:author, :project, project: :namespace).preload(:target) }
   scope :for_milestone_id, ->(milestone_id) { where(target_type: "Milestone", target_id: milestone_id) }
 
+  self.inheritance_column = 'action'
+
   class << self
+    def find_sti_class(action)
+      if action.to_i == PUSHED
+        PushEvent
+      else
+        Event
+      end
+    end
+
+    def subclass_from_attributes(attrs)
+      # Without this Rails will keep calling this method on the returned class,
+      # resulting in an infinite loop.
+      return unless self == Event
+
+      action = attrs.with_indifferent_access[inheritance_column].to_i
+
+      PushEvent if action == PUSHED
+    end
+
     # Update Gitlab::ContributionsCalendar#activity_dates if this changes
     def contributions
       where("action = ? OR (target_type IN (?) AND action IN (?)) OR (target_type = ? AND action = ?)",
@@ -290,6 +323,16 @@ class Event < ActiveRecord::Base
     @commits ||= (data[:commits] || []).reverse
   end
 
+  def commit_title
+    commit = commits.last
+
+    commit[:message] if commit
+  end
+
+  def commit_id
+    commit_to || commit_from
+  end
+
   def commits_count
     data[:total_commits_count] || commits.count || 0
   end
@@ -385,6 +428,16 @@ class Event < ActiveRecord::Base
     user ? author_id == user.id : false
   end
 
+  # We're manually replicating data into the new table since database triggers
+  # are not dumped to db/schema.rb. This could mean that a new installation
+  # would not have the triggers in place, thus losing events data in GitLab
+  # 10.0.
+  def replicate_event_for_push_events_migration
+    new_attributes = attributes.with_indifferent_access.except(:title, :data)
+
+    EventForMigration.create!(new_attributes)
+  end
+
   private
 
   def recent_update?

app/models/event_collection.rb

+# A collection of events to display in an event list.
+#
+# An EventCollection is meant to be used for displaying events to a user (e.g.
+# in a controller), it's not suitable for building queries that are used for
+# building other queries.
+class EventCollection
+  # To prevent users from putting too much pressure on the database by cycling
+  # through thousands of events we put a limit on the number of pages.
+  MAX_PAGE = 10
+
+  # projects - An ActiveRecord::Relation object that returns the projects for
+  #            which to retrieve events.
+  # filter - An EventFilter instance to use for filtering events.
+  def initialize(projects, limit: 20, offset: 0, filter: nil)
+    @projects = projects
+    @limit = limit
+    @offset = offset
+    @filter = filter
+  end
+
+  # Returns an Array containing the events.
+  def to_a
+    return [] if current_page > MAX_PAGE
+
+    relation = if Gitlab::Database.join_lateral_supported?
+                 relation_with_join_lateral
+               else
+                 relation_without_join_lateral
+               end
+
+    relation.with_associations.to_a
+  end
+
+  private
+
+  # Returns the events relation to use when JOIN LATERAL is not supported.
+  #
+  # This relation simply gets all the events for all authorized projects, then
+  # limits that set.
+  def relation_without_join_lateral
+    events = filtered_events.in_projects(projects)
+
+    paginate_events(events)
+  end
+
+  # Returns the events relation to use when JOIN LATERAL is supported.
+  #
+  # This relation is built using JOIN LATERAL, producing faster queries than a
+  # regular LIMIT + OFFSET approach.
+  def relation_with_join_lateral
+    projects_for_lateral = projects.select(:id).to_sql
+
+    lateral = filtered_events
+      .limit(limit_for_join_lateral)
+      .where('events.project_id = projects_for_lateral.id')
+      .to_sql
+
+    # The outer query does not need to re-apply the filters since the JOIN
+    # LATERAL body already takes care of this.
+    outer = base_relation
+      .from("(#{projects_for_lateral}) projects_for_lateral")
+      .joins("JOIN LATERAL (#{lateral}) AS #{Event.table_name} ON true")
+
+    paginate_events(outer)
+  end
+
+  def filtered_events
+    @filter ? @filter.apply_filter(base_relation) : base_relation
+  end
+
+  def paginate_events(events)
+    events.limit(@limit).offset(@offset)
+  end
+
+  def base_relation
+    # We want to have absolute control over the event queries being built, thus
+    # we're explicitly opting out of any default scopes that may be set.
+    Event.unscoped.recent
+  end
+
+  def limit_for_join_lateral
+    # Applying the OFFSET on the inside of a JOIN LATERAL leads to incorrect
+    # results. To work around this we need to increase the inner limit for every
+    # page.
+    #
+    # This means that on page 1 we use LIMIT 20, and an outer OFFSET of 0. On
+    # page 2 we use LIMIT 40 and an outer OFFSET of 20.
+    @limit + @offset
+  end
+
+  def current_page
+    (@offset / @limit) + 1
+  end
+
+  def projects
+    @projects.except(:order)
+  end
+end

app/models/event_for_migration.rb

+# This model is used to replicate events between the old "events" table and the
+# new "events_for_migration" table that will replace "events" in GitLab 10.0.
+class EventForMigration < ActiveRecord::Base
+  self.table_name = 'events_for_migration'
+end

app/models/push_event.rb

+class PushEvent < Event
+  # This validation exists so we can't accidentally use PushEvent with a
+  # different "action" value.
+  validate :validate_push_action
+
+  # Authors are required as they're used to display who pushed data.
+  #
+  # We're just validating the presence of the ID here as foreign key constraints
+  # should ensure the ID points to a valid user.
+  validates :author_id, presence: true
+
+  # The project is required to build links to commits, commit ranges, etc.
+  #
+  # We're just validating the presence of the ID here as foreign key constraints
+  # should ensure the ID points to a valid project.
+  validates :project_id, presence: true
+
+  # The "data" field must not be set for push events since it's not used and a
+  # waste of space.
+  validates :data, absence: true
+
+  # These fields are also not used for push events, thus storing them would be a
+  # waste.
+  validates :target_id, absence: true
+  validates :target_type, absence: true
+
+  def self.sti_name
+    PUSHED
+  end
+
+  def push?
+    true
+  end
+
+  def push_with_commits?
+    !!(commit_from && commit_to)
+  end
+
+  def tag?
+    return super unless push_event_payload
+
+    push_event_payload.tag?
+  end
+
+  def branch?
+    return super unless push_event_payload
+
+    push_event_payload.branch?
+  end
+
+  def valid_push?
+    return super unless push_event_payload
+
+    push_event_payload.ref.present?
+  end
+
+  def new_ref?
+    return super unless push_event_payload
+
+    push_event_payload.created?
+  end
+
+  def rm_ref?
+    return super unless push_event_payload
+
+    push_event_payload.removed?
+  end
+
+  def commit_from
+    return super unless push_event_payload
+
+    push_event_payload.commit_from
+  end
+
+  def commit_to
+    return super unless push_event_payload
+
+    push_event_payload.commit_to
+  end
+
+  def ref_name
+    return super unless push_event_payload
+
+    push_event_payload.ref
+  end
+
+  def ref_type
+    return super unless push_event_payload
+
+    push_event_payload.ref_type
+  end
+
+  def branch_name
+    return super unless push_event_payload
+
+    ref_name
+  end
+
+  def tag_name
+    return super unless push_event_payload
+
+    ref_name
+  end
+
+  def commit_title
+    return super unless push_event_payload
+
+    push_event_payload.commit_title
+  end
+
+  def commit_id
+    commit_to || commit_from
+  end
+
+  def commits_count
+    return super unless push_event_payload
+
+    push_event_payload.commit_count
+  end
+
+  def validate_push_action
+    return if action == PUSHED
+
+    errors.add(:action, "the action #{action.inspect} is not valid")
+  end
+end

app/models/push_event_payload.rb

+class PushEventPayload < ActiveRecord::Base
+  include ShaAttribute
+
+  belongs_to :event, inverse_of: :push_event_payload
+
+  validates :event_id, :commit_count, :action, :ref_type, presence: true
+  validates :commit_title, length: { maximum: 70 }
+
+  sha_attribute :commit_from
+  sha_attribute :commit_to
+
+  enum action: {
+    created: 0,
+    removed: 1,
+    pushed: 2
+  }
+
+  enum ref_type: {
+    branch: 0,
+    tag: 1
+  }
+end

app/models/redirect_route.rb

@@ -8,5 +8,13 @@ class RedirectRoute < ActiveRecord::Base
     presence: true,
     uniqueness: { case_sensitive: false }
 
-  scope :matching_path_and_descendants, -> (path) { where('redirect_routes.path = ? OR redirect_routes.path LIKE ?', path, "#{sanitize_sql_like(path)}/%") }
+  scope :matching_path_and_descendants, -> (path) do
+    wheres = if Gitlab::Database.postgresql?
+               'LOWER(redirect_routes.path) = LOWER(?) OR LOWER(redirect_routes.path) LIKE LOWER(?)'
+             else
+               'redirect_routes.path = ? OR redirect_routes.path LIKE ?'
+             end
+             
+    where(wheres, path, "#{sanitize_sql_like(path)}/%")
+  end
 end

app/services/ci/register_job_service.rb

@@ -85,13 +85,13 @@ module Ci
     end
 
     def register_failure
-      failed_attempt_counter.increase
-      attempt_counter.increase
+      failed_attempt_counter.increment
+      attempt_counter.increment
     end
 
     def register_success(job)
       job_queue_duration_seconds.observe({ shared_runner: @runner.shared? }, Time.now - job.created_at)
-      attempt_counter.increase
+      attempt_counter.increment
     end
 
     def failed_attempt_counter

app/services/event_create_service.rb

@@ -71,7 +71,14 @@ class EventCreateService
   end
 
   def push(project, current_user, push_data)
-    create_event(project, current_user, Event::PUSHED, data: push_data)
+    # We're using an explicit transaction here so that any errors that may occur
+    # when creating push payload data will result in the event creation being
+    # rolled back as well.
+    Event.transaction do
+      event = create_event(project, current_user, Event::PUSHED)
+
+      PushEventPayloadService.new(event, push_data).execute
+    end
 
     Users::ActivityService.new(current_user, 'push').execute
   end

app/services/projects/update_pages_service.rb

@@ -172,11 +172,11 @@ module Projects
     end
 
     def register_attempt
-      pages_deployments_total_counter.increase
+      pages_deployments_total_counter.increment
     end
 
     def register_failure
-      pages_deployments_failed_total_counter.increase
+      pages_deployments_failed_total_counter.increment
     end
 
     def pages_deployments_total_counter

app/services/push_event_payload_service.rb

+# Service class for creating push event payloads as stored in the
+# "push_event_payloads" table.
+#
+# Example:
+#
+#     data = Gitlab::DataBuilder::Push.build(...)
+#     event = Event.create(...)
+#
+#     PushEventPayloadService.new(event, data).execute
+class PushEventPayloadService
+  # event - The event this push payload belongs to.
+  # push_data - A Hash produced by `Gitlab::DataBuilder::Push.build` to use for
+  #             building the push payload.
+  def initialize(event, push_data)
+    @event = event
+    @push_data = push_data
+  end
+
+  # Creates and returns a new PushEventPayload row.
+  #
+  # This method will raise upon encountering validation errors.
+  #
+  # Returns an instance of PushEventPayload.
+  def execute
+    @event.build_push_event_payload(
+      commit_count: commit_count,
+      action: action,
+      ref_type: ref_type,
+      commit_from: commit_from_id,
+      commit_to: commit_to_id,
+      ref: trimmed_ref,
+      commit_title: commit_title,
+      event_id: @event.id
+    )
+
+    @event.push_event_payload.save!
+    @event.push_event_payload
+  end
+
+  # Returns the commit title to use.
+  #
+  # The commit title is limited to the first line and a maximum of 70
+  # characters.
+  def commit_title
+    commit = @push_data.fetch(:commits).last
+
+    return nil unless commit && commit[:message]
+
+    raw_msg = commit[:message]
+
+    # Find where the first line ends, without turning the entire message into an
+    # Array of lines (this is a waste of memory for large commit messages).
+    index = raw_msg.index("\n")
+    message = index ? raw_msg[0..index] : raw_msg
+
+    message.strip.truncate(70)
+  end
+
+  def commit_from_id
+    if create?
+      nil
+    else
+      revision_before
+    end
+  end
+
+  def commit_to_id
+    if remove?
+      nil
+    else
+      revision_after
+    end
+  end
+
+  def commit_count
+    @push_data.fetch(:total_commits_count)
+  end
+
+  def ref
+    @push_data.fetch(:ref)
+  end
+
+  def revision_before
+    @push_data.fetch(:before)
+  end
+
+  def revision_after
+    @push_data.fetch(:after)
+  end
+
+  def trimmed_ref
+    Gitlab::Git.ref_name(ref)
+  end
+
+  def create?
+    Gitlab::Git.blank_ref?(revision_before)
+  end
+
+  def remove?
+    Gitlab::Git.blank_ref?(revision_after)
+  end
+
+  def action
+    if create?
+      :created
+    elsif remove?
+      :removed
+    else
+      :pushed
+    end
+  end
+
+  def ref_type
+    if Gitlab::Git.tag_ref?(ref)
+      :tag
+    else
+      :branch
+    end
+  end
+end

app/views/admin/application_settings/_form.html.haml

@@ -48,6 +48,12 @@
         = select(:application_setting, :enabled_git_access_protocol, [['Both SSH and HTTP(S)', nil], ['Only SSH', 'ssh'], ['Only HTTP(S)', 'http']], {}, class: 'form-control')
         %span.help-block#clone-protocol-help
           Allow only the selected protocols to be used for Git access.
+    .form-group
+      .col-sm-offset-2.col-sm-10
+        .checkbox
+          = f.label :project_export_enabled do
+            = f.check_box :project_export_enabled
+            Project export enabled
 
   %fieldset
     %legend Account and Limit Settings

app/views/dashboard/projects/_blank_state_admin_welcome.html.haml

@@ -28,6 +28,6 @@
       %h3.blank-state-title
         Create a group
       %p.blank-state-text
-        Groups are a great way to organise projects and people.
+        Groups are a great way to organize projects and people.
       = link_to new_group_path, class: "btn btn-new" do
         New group

app/views/events/_commit.html.haml

 %li.commit
   .commit-row-title
-    = link_to truncate_sha(commit[:id]), project_commit_path(project, commit[:id]), class: "commit-sha", alt: '', title: truncate_sha(commit[:id])
+    = link_to truncate_sha(event.commit_id), project_commit_path(project, event.commit_id), class: "commit-sha", alt: '', title: truncate_sha(event.commit_id)
     ·
-    = markdown event_commit_title(commit[:message]), project: project, pipeline: :single_line, author: event.author
+    = markdown event_commit_title(event.commit_title), project: project, pipeline: :single_line, author: event.author

app/views/events/_event_push.atom.haml

 %div{ xmlns: "http://www.w3.org/1999/xhtml" }
-  - event.commits.first(15).each do |commit|
-    %p
-      %strong= commit[:author][:name]
-      = link_to "(##{truncate_sha(commit[:id])})", project_commit_path(event.project, id: commit[:id])
-      %i
-        at
-        = commit[:timestamp].to_time.to_s(:short)
-    %blockquote= markdown(escape_once(commit[:message]), pipeline: :atom, project: event.project, author: event.author)
-  - if event.commits_count > 15
+  %p
+    %strong= event.author_name
+    = link_to "(#{truncate_sha(event.commit_id)})", project_commit_path(event.project, event.commit_id)
+    %i
+      at
+      = event.created_at.to_s(:short)
+  %blockquote= markdown(escape_once(event.commit_title), pipeline: :atom, project: event.project, author: event.author)
+  - if event.commits_count > 1
     %p
       %i
         \... and
-        = pluralize(event.commits_count - 15, "more commit")
+        = pluralize(event.commits_count - 1, "more commit")

app/views/events/event/_push.html.haml

@@ -14,9 +14,7 @@
 - if event.push_with_commits?
   .event-body
     %ul.well-list.event_commits
-      - few_commits = event.commits[0...2]
-      - few_commits.each do |commit|
-        = render "events/commit", commit: commit, project: project, event: event
+      = render "events/commit", project: project, event: event
 
       - create_mr = event.new_ref? && create_mr_button?(project.default_branch, event.ref_name, project) && event.authored_by?(current_user)
       - if event.commits_count > 1
@@ -44,9 +42,6 @@
           = link_to create_mr_path(project.default_branch, event.ref_name, project) do
             Create Merge Request
 - elsif event.rm_ref?
-  - repository = project.repository
-  - last_commit = repository.commit(event.commit_from)
-  - if last_commit
-    .event-body
-      %ul.well-list.event_commits
-        = render "events/commit", commit: last_commit, project: project, event: event
+  .event-body
+    %ul.well-list.event_commits
+      = render "events/commit", project: project, event: event

app/views/import/fogbugz/new_user_map.html.haml

@@ -10,7 +10,7 @@
     Customize how FogBugz email addresses and usernames are imported into GitLab.
     In the next step, you'll be able to select the projects you want to import.
   %p
-    The user map is a mapping of the FogBugz users that participated on your projects to the way their email address and usernames wil be imported into GitLab. You can change this by populating the table below.
+    The user map is a mapping of the FogBugz users that participated on your projects to the way their email address and usernames will be imported into GitLab. You can change this by populating the table below.
   %ul
     %li
       %strong Default: Map a FogBugz account ID to a full name

app/views/projects/_export.html.haml

+- return unless current_application_settings.project_export_enabled?
+
+- project = local_assigns.fetch(:project)
+- expanded = Rails.env.test?
+
+%section.settings
+  .settings-header
+    %h4
+      Export project
+    %button.btn.js-settings-toggle
+      = expanded ? 'Collapse' : 'Expand'
+    %p
+      Export this project with all its related data in order to move your project to a new GitLab instance. Once the export is finished, you can import the file from the "New Project" page.
+  .settings-content.no-animate{ class: ('expanded' if expanded) }
+    .bs-callout.bs-callout-info
+      %p.append-bottom-0
+        %p
+          The following items will be exported:
+        %ul
+          %li Project and wiki repositories
+          %li Project uploads
+          %li Project configuration including web hooks and services
+          %li Issues with comments, merge requests with diffs and comments, labels, milestones, snippets, and other project entities
+        %p
+          The following items will NOT be exported:
+        %ul
+          %li Job traces and artifacts
+          %li LFS objects
+          %li Container registry images
+          %li CI variables
+          %li Any encrypted tokens
+    %p
+      Once the exported file is ready, you will receive a notification email with a download link.
+    - if project.export_project_path
+      = link_to 'Download export',  download_export_project_path(project),
+              rel: 'nofollow', download: '', method: :get, class: "btn btn-default"
+      = link_to 'Generate new export',  generate_new_export_project_path(project),
+              method: :post, class: "btn btn-default"
+    - else
+      = link_to 'Export project', export_project_path(project),
+              method: :post, class: "btn btn-default"

app/views/projects/_md_preview.html.haml

 - referenced_users = local_assigns.fetch(:referenced_users, nil)
 
 - if defined?(@issue) && @issue.confidential?
-  %li.confidential-issue-warning
+  .confidential-issue-warning
     = confidential_icon(@issue)
     %span This is a confidential issue. Your comment will not be visible to the public.
-- else
-  %li.confidential-issue-warning.not-confidential
 
 .md-area
   .md-header

app/views/projects/diffs/_diffs.html.haml

@@ -2,8 +2,9 @@
 - show_whitespace_toggle = local_assigns.fetch(:show_whitespace_toggle, true)
 - can_create_note = !@diff_notes_disabled && can?(current_user, :create_note, diffs.project)
 - diff_files = diffs.diff_files
+- merge_request = local_assigns.fetch(:merge_request, false)
 
-.content-block.oneline-block.files-changed.diff-files-changed.js-diff-files-changed
+.content-block.oneline-block.files-changed.diff-files-changed.js-diff-files-changed{ class: ("diff-files-changed-merge-request" if merge_request) }
   .files-changed-inner
     .inline-parallel-buttons
       - if !diffs_expanded? && diff_files.any? { |diff_file| diff_file.collapsed? }

app/views/projects/edit.html.haml

@@ -161,42 +161,7 @@
         = render 'merge_request_settings', form: f
         = f.submit 'Save changes', class: "btn btn-save"
 
-  %section.settings
-    .settings-header
-      %h4
-        Export project
-      %button.btn.js-settings-toggle
-        = expanded ? 'Collapse' : 'Expand'
-      %p
-        Export this project with all its related data in order to move your project to a new GitLab instance. Once the export is finished, you can import the file from the "New Project" page.
-    .settings-content.no-animate{ class: ('expanded' if expanded) }
-      .bs-callout.bs-callout-info
-        %p.append-bottom-0
-          %p
-            The following items will be exported:
-          %ul
-            %li Project and wiki repositories
-            %li Project uploads
-            %li Project configuration including web hooks and services
-            %li Issues with comments, merge requests with diffs and comments, labels, milestones, snippets, and other project entities
-          %p
-            The following items will NOT be exported:
-          %ul
-            %li Job traces and artifacts
-            %li LFS objects
-            %li Container registry images
-            %li CI variables
-            %li Any encrypted tokens
-      %p
-        Once the exported file is ready, you will receive a notification email with a download link.
-      - if @project.export_project_path
-        = link_to 'Download export',  download_export_project_path(@project),
-                rel: 'nofollow', download: '', method: :get, class: "btn btn-default"
-        = link_to 'Generate new export',  generate_new_export_project_path(@project),
-                method: :post, class: "btn btn-default"
-      - else
-        = link_to 'Export project', export_project_path(@project),
-                method: :post, class: "btn btn-default"
+  = render 'export', project: @project
 
   %section.settings.advanced-settings
     .settings-header
@@ -205,7 +170,7 @@
       %button.btn.js-settings-toggle
         = expanded ? 'Collapse' : 'Expand'
       %p
-        Perform advanced options such as housekeeping, exporting, archiveing, renameing, transfering, or removeing your project.
+        Perform advanced options such as housekeeping, exporting, archiving, renaming, transferring, or removing your project.
     .settings-content.no-animate{ class: ('expanded' if expanded) }
       .sub-section
         %h4 Housekeeping
@@ -274,7 +239,7 @@
                 %li Be careful. Changing the project's namespace can have unintended side effects.
                 %li You can only transfer the project to namespaces you manage.
                 %li You will need to update your local repositories to point to the new location.
-                %li Project visibility level will be changed to match namespace rules when transfering to a group.
+                %li Project visibility level will be changed to match namespace rules when transferring to a group.
             = f.submit 'Transfer project', class: "btn btn-remove js-confirm-danger", data: { "confirm-danger-message" => transfer_project_message(@project) }
       - if @project.forked? && can?(current_user, :remove_fork_project, @project)
         .sub-section

app/views/projects/jobs/_sidebar.html.haml

@@ -75,7 +75,7 @@
             Pipeline
             = link_to "##{@build.pipeline.id}", project_pipeline_path(@project, @build.pipeline), class: 'link-commit'
             from
-            = link_to "#{@build.pipeline.ref}", project_branch_path(@project, @build.pipeline.ref), class: 'link-commit'
+            = link_to "#{@build.pipeline.ref}", project_ref_path(@project, @build.pipeline.ref), class: 'link-commit ref-name'
           %button.dropdown-menu-toggle{ type: 'button', 'data-toggle' => 'dropdown' }
             %span.stage-selection More
             = icon('chevron-down')

app/views/projects/merge_requests/diffs/_diffs.html.haml

 - if @merge_request_diff.collected? || @merge_request_diff.overflow?
   = render 'projects/merge_requests/diffs/versions'
-  = render "projects/diffs/diffs", diffs: @diffs, environment: @environment
+  = render "projects/diffs/diffs", diffs: @diffs, environment: @environment, merge_request: true
 - elsif @merge_request_diff.empty?
   .nothing-here-block Nothing to merge from #{@merge_request.source_branch} into #{@merge_request.target_branch}

app/views/shared/issuable/_sidebar.html.haml

@@ -3,7 +3,7 @@
   = page_specific_javascript_bundle_tag('common_vue')
   = page_specific_javascript_bundle_tag('sidebar')
 
-%aside.right-sidebar.js-right-sidebar{ data: { "offset-top" => "50", "spy" => "affix", signed: { in: current_user.present? } }, class: sidebar_gutter_collapsed_class, 'aria-live' => 'polite' }
+%aside.right-sidebar.js-right-sidebar.js-issuable-sidebar{ data: { "offset-top" => ("50" unless show_new_nav?), "spy" => ("affix" unless show_new_nav?), signed: { in: current_user.present? } }, class: sidebar_gutter_collapsed_class, 'aria-live' => 'polite' }
   .issuable-sidebar{ data: { endpoint: "#{issuable_json_path(issuable)}" } }
     - can_edit_issuable = can?(current_user, :"admin_#{issuable.to_ability_name}", @project)
     .block.issuable-sidebar-header

bin/changelog

@@ -14,54 +14,107 @@ Options = Struct.new(
   :dry_run,
   :force,
   :merge_request,
-  :title
+  :title,
+  :type
 )
+INVALID_TYPE = -1
 
 class ChangelogOptionParser
-  def self.parse(argv)
-    options = Options.new
+  Type = Struct.new(:name, :description)
+  TYPES = [
+    Type.new('added', 'New feature'),
+    Type.new('fixed', 'Bug fix'),
+    Type.new('changed', 'Feature change'),
+    Type.new('deprecated', 'New deprecation'),
+    Type.new('removed', 'Feature removal'),
+    Type.new('security', 'Security fix'),
+    Type.new('other', 'Other')
+  ].freeze
+  TYPES_OFFSET = 1
+
+  class << self
+    def parse(argv)
+      options = Options.new
+
+      parser = OptionParser.new do |opts|
+        opts.banner = "Usage: #{__FILE__} [options] [title]\n\n"
+
+        # Note: We do not provide a shorthand for this in order to match the `git
+        # commit` interface
+        opts.on('--amend', 'Amend the previous commit') do |value|
+          options.amend = value
+        end
+
+        opts.on('-f', '--force', 'Overwrite an existing entry') do |value|
+          options.force = value
+        end
+
+        opts.on('-m', '--merge-request [integer]', Integer, 'Merge Request ID') do |value|
+          options.merge_request = value
+        end
+
+        opts.on('-n', '--dry-run', "Don't actually write anything, just print") do |value|
+          options.dry_run = value
+        end
+
+        opts.on('-u', '--git-username', 'Use Git user.name configuration as the author') do |value|
+          options.author = git_user_name if value
+        end
+
+        opts.on('-t', '--type [string]', String, "The category of the change, valid options are: #{TYPES.map(&:name).join(', ')}") do |value|
+          options.type = parse_type(value)
+        end
+
+        opts.on('-h', '--help', 'Print help message') do
+          $stdout.puts opts
+          exit
+        end
+      end
 
-    parser = OptionParser.new do |opts|
-      opts.banner = "Usage: #{__FILE__} [options] [title]\n\n"
+      parser.parse!(argv)
 
-      # Note: We do not provide a shorthand for this in order to match the `git
-      # commit` interface
-      opts.on('--amend', 'Amend the previous commit') do |value|
-        options.amend = value
-      end
+      # Title is everything that remains, but let's clean it up a bit
+      options.title = argv.join(' ').strip.squeeze(' ').tr("\r\n", '')
 
-      opts.on('-f', '--force', 'Overwrite an existing entry') do |value|
-        options.force = value
-      end
+      options
+    end
 
-      opts.on('-m', '--merge-request [integer]', Integer, 'Merge Request ID') do |value|
-        options.merge_request = value
-      end
+    def read_type
+      read_type_message
 
-      opts.on('-n', '--dry-run', "Don't actually write anything, just print") do |value|
-        options.dry_run = value
-      end
+      type = TYPES[$stdin.getc.to_i - TYPES_OFFSET]
+      assert_valid_type!(type)
 
-      opts.on('-u', '--git-username', 'Use Git user.name configuration as the author') do |value|
-        options.author = git_user_name if value
-      end
+      type.name
+    end
+
+    private
 
-      opts.on('-h', '--help', 'Print help message') do
-        $stdout.puts opts
-        exit
+    def parse_type(name)
+      type_found = TYPES.find do |type|
+        type.name == name
       end
+      type_found ? type_found.name : INVALID_TYPE
     end
 
-    parser.parse!(argv)
-
-    # Title is everything that remains, but let's clean it up a bit
-    options.title = argv.join(' ').strip.squeeze(' ').tr("\r\n", '')
+    def read_type_message
+      $stdout.puts "\n>> Please specify the index for the category of your change:"
+      TYPES.each_with_index do |type, index|
+        $stdout.puts "#{index + TYPES_OFFSET}. #{type.description}"
+      end
+      $stdout.print "\n?> "
+    end
 
-    options
-  end
+    def assert_valid_type!(type)
+      unless type
+        $stderr.puts "Invalid category index, please select an index between 1 and #{TYPES.length}"
+        exit 1
+      end
+    end
 
-  def self.git_user_name
-    %x{git config user.name}.strip
+    def git_user_name
+      %x{git config user.name}.strip
+    end
   end
 end
 
@@ -72,8 +125,12 @@ class ChangelogEntry
     @options = options
 
     assert_feature_branch!
-    assert_new_file!
     assert_title!
+    assert_new_file!
+
+    # Read type from $stdin unless is already set
+    options.type ||= ChangelogOptionParser.read_type
+    assert_valid_type!
 
     $stdout.puts "\e[32mcreate\e[0m #{file_path}"
     $stdout.puts contents
@@ -90,7 +147,8 @@ class ChangelogEntry
     yaml_content = YAML.dump(
       'title'         => title,
       'merge_request' => options.merge_request,
-      'author'        => options.author
+      'author'        => options.author,
+      'type'          => options.type
     )
     remove_trailing_whitespace(yaml_content)
   end
@@ -129,6 +187,12 @@ class ChangelogEntry
       " to use the title from the previous commit."
   end
 
+  def assert_valid_type!
+    return unless options.type && options.type == INVALID_TYPE
+
+    fail_with 'Invalid category given!'
+  end
+
   def title
     if options.title.empty?
       last_commit_subject

changelogs/unreleased/21949-add-type-to-changelog.yml

+---
+title: Added type to CHANGELOG entries
+merge_request:
+author: Jacopo Beschi @jacopo-beschi

changelogs/unreleased/29811-fix-line-number-alignment.yml

+---
+title: Fix the alignment of line numbers to lines of code in code viewer 
+merge_request: 13403
+author: Trevor Flynn
\ No newline at end of file

changelogs/unreleased/35435-pending-delete-project-error-in-admin-interface-fix.yml

+---
+title: Project pending delete no longer return 500 error in admins projects view
+merge_request: 13389
+author:

changelogs/unreleased/36158-new-issue-button.yml

+---
+title: Fixes new issue button for failed job returning 404
+merge_request:
+author:

changelogs/unreleased/36185-or-separator.yml

+---
+title: Align OR separator to center in new project page
+merge_request:
+author:

changelogs/unreleased/36213-return-is_admin-in-users-api-when-current_user-is-admin.yml

+---
+title: Include the `is_admin` field in the `GET /users/:id` API when current user
+  is an admin
+merge_request:
+author:
+type: fixed

changelogs/unreleased/appearances-caching-and-schema.yml

+---
+title: Cache Appearance instances in Redis
+merge_request:
+author:

changelogs/unreleased/broadcast-messages-cache.yml

+---
+title: Better caching and indexing of broadcast messages
+merge_request:
+author:

changelogs/unreleased/bump-omniauth-ldap-gem-version-2-0-4.yml

+---
+title: Bumps omniauth-ldap gem version to 2.0.4
+merge_request: 13465
+author:

changelogs/unreleased/disable-project-export.yml

+---
+title: Add option to disable project export on instance
+merge_request: 13211
+author: Robin Bobbitt

changelogs/unreleased/fix-import-symbolink-links.yml

+---
+title: Remove hidden symlinks from project import files
+merge_request:
+author:

changelogs/unreleased/migrate-events-into-a-new-format.yml

+---
+title: Migrate events into a new format to reduce the storage necessary and improve performance
+merge_request:
+author:

changelogs/unreleased/mk-fix-case-insensitive-redirect-matching.yml

+---
+title: Fix destroy of case-insensitive conflicting redirects
+merge_request: 13357
+author:

changelogs/unreleased/rs-alphanumeric-ssh-params.yml

+---
+title: Disallow Git URLs that include a username or hostname beginning with a non-alphanumeric
+  character
+merge_request:
+author:

changelogs/unreleased/use-a-specialized-class-for-querying-events.yml

+---
+title: Use a specialized class for querying events to improve performance
+merge_request:
+author:

changelogs/unreleased/zj-ref-path-monospace.yml

+---
+title: Use project_ref_path to create the link to a branch to fix links that 404
+merge_request:
+author:

config/routes/repository.rb

@@ -3,6 +3,9 @@
 resource :repository, only: [:create] do
   member do
     get ':ref/archive', constraints: { format: Gitlab::PathRegex.archive_formats_regex, ref: /.+/ }, action: 'archive', as: 'archive'
+    
+    # deprecated since GitLab 9.5
+    get 'archive', constraints: { format: Gitlab::PathRegex.archive_formats_regex }, as: 'archive_alternative'
   end
 end
 

db/migrate/20170608152747_prepare_events_table_for_push_events_migration.rb

+# See http://doc.gitlab.com/ce/development/migration_style_guide.html
+# for more information on how to write migrations for GitLab.
+
+class PrepareEventsTableForPushEventsMigration < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  # Set this constant to true if this migration requires downtime.
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  def up
+    # The order of these columns is deliberate and results in the following
+    # columns and sizes:
+    #
+    # * id (4 bytes)
+    # * project_id (4 bytes)
+    # * author_id (4 bytes)
+    # * target_id (4 bytes)
+    # * created_at (8 bytes)
+    # * updated_at (8 bytes)
+    # * action (2 bytes)
+    # * target_type (variable)
+    #
+    # Unfortunately we can't make the "id" column a bigint/bigserial as Rails 4
+    # does not support this properly.
+    create_table :events_for_migration do |t|
+      t.references :project,
+                   index: true,
+                   foreign_key: { on_delete: :cascade }
+
+      t.integer :author_id, index: true, null: false
+      t.integer :target_id
+
+      t.timestamps_with_timezone null: false
+
+      t.integer :action, null: false, limit: 2, index: true
+      t.string :target_type
+
+      t.index %i[target_type target_id]
+    end
+
+    # t.references doesn't like it when the column name doesn't make the table
+    # name so we have to add the foreign key separately.
+    add_concurrent_foreign_key(:events_for_migration, :users, column: :author_id)
+  end
+
+  def down
+    drop_table :events_for_migration
+  end
+end

db/migrate/20170608152748_create_push_event_payloads_tables.rb

+# See http://doc.gitlab.com/ce/development/migration_style_guide.html
+# for more information on how to write migrations for GitLab.
+
+class CreatePushEventPayloadsTables < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  # Set this constant to true if this migration requires downtime.
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  def up
+    create_table :push_event_payloads, id: false do |t|
+      t.bigint :commit_count, null: false
+
+      t.integer :event_id, null: false
+      t.integer :action, null: false, limit: 2
+      t.integer :ref_type, null: false, limit: 2
+
+      t.binary :commit_from
+      t.binary :commit_to
+
+      t.text :ref
+      t.string :commit_title, limit: 70
+
+      t.index :event_id, unique: true
+    end
+
+    # We're adding a foreign key to the _shadow_ table, and this is deliberate.
+    # By using the shadow table we don't have to recreate/revalidate this
+    # foreign key after swapping the "events_for_migration" and "events" tables.
+    #
+    # The "events_for_migration" table has a foreign key to "projects.id"
+    # ensuring that project removals also remove events from the shadow table
+    # (and thus also from this table).
+    add_concurrent_foreign_key(
+      :push_event_payloads,
+      :events_for_migration,
+      column: :event_id
+    )
+  end
+
+  def down
+    drop_table :push_event_payloads
+  end
+end

db/migrate/20170727123534_add_index_on_events_project_id_id.rb

+# See http://doc.gitlab.com/ce/development/migration_style_guide.html
+# for more information on how to write migrations for GitLab.
+
+class AddIndexOnEventsProjectIdId < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  DOWNTIME = false
+
+  COLUMNS = %i[project_id id].freeze
+  TABLES = %i[events events_for_migration].freeze
+
+  disable_ddl_transaction!
+
+  def up
+    TABLES.each do |table|
+      add_concurrent_index(table, COLUMNS) unless index_exists?(table, COLUMNS)
+
+      # We remove the index _after_ adding the new one since MySQL doesn't let
+      # you remove an index when a foreign key exists for the same column.
+      if index_exists?(table, :project_id)
+        remove_concurrent_index(table, :project_id)
+      end
+    end
+  end
+
+  def down
+    TABLES.each do |table|
+      unless index_exists?(table, :project_id)
+        add_concurrent_index(table, :project_id)
+      end
+
+      unless index_exists?(table, COLUMNS)
+        remove_concurrent_index(table, COLUMNS)
+      end
+    end
+  end
+end

db/migrate/20170809133343_add_broadcast_messages_index.rb

+# See http://doc.gitlab.com/ce/development/migration_style_guide.html
+# for more information on how to write migrations for GitLab.
+
+class AddBroadcastMessagesIndex < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  # Set this constant to true if this migration requires downtime.
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  COLUMNS = %i[starts_at ends_at id].freeze
+
+  def up
+    add_concurrent_index :broadcast_messages, COLUMNS
+  end
+
+  def down
+    remove_concurrent_index :broadcast_messages, COLUMNS
+  end
+end

db/migrate/20170809134534_add_broadcast_message_not_null_constraints.rb

+# See http://doc.gitlab.com/ce/development/migration_style_guide.html
+# for more information on how to write migrations for GitLab.
+
+class AddBroadcastMessageNotNullConstraints < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  # Set this constant to true if this migration requires downtime.
+  DOWNTIME = false
+
+  COLUMNS = %i[starts_at ends_at created_at updated_at message_html]
+
+  def change
+    COLUMNS.each do |column|
+      change_column_null :broadcast_messages, column, false
+    end
+  end
+end

db/migrate/20170809142252_cleanup_appearances_schema.rb

+# See http://doc.gitlab.com/ce/development/migration_style_guide.html
+# for more information on how to write migrations for GitLab.
+
+class CleanupAppearancesSchema < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  # Set this constant to true if this migration requires downtime.
+  DOWNTIME = false
+
+  NOT_NULL_COLUMNS = %i[title description description_html created_at updated_at]
+
+  TIME_COLUMNS = %i[created_at updated_at]
+
+  def up
+    NOT_NULL_COLUMNS.each do |column|
+      change_column_null :appearances, column, false
+    end
+
+    TIME_COLUMNS.each do |column|
+      change_column :appearances, column, :datetime_with_timezone
+    end
+  end
+
+  def down
+    NOT_NULL_COLUMNS.each do |column|
+      change_column_null :appearances, column, true
+    end
+
+    TIME_COLUMNS.each do |column|
+      change_column :appearances, column, :datetime # rubocop: disable Migration/Datetime
+    end
+  end
+end

db/migrate/20170809161910_add_project_export_enabled_to_application_settings.rb

+class AddProjectExportEnabledToApplicationSettings < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+  disable_ddl_transaction!
+
+  DOWNTIME = false
+
+  def up
+    add_column_with_default(:application_settings, :project_export_enabled, :boolean, default: true)
+  end
+
+  def down
+    remove_column(:application_settings, :project_export_enabled)
+  end
+end

db/post_migrate/20170627101016_schedule_event_migrations.rb

+# See http://doc.gitlab.com/ce/development/migration_style_guide.html
+# for more information on how to write migrations for GitLab.
+
+class ScheduleEventMigrations < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+
+  DOWNTIME = false
+  BUFFER_SIZE = 1000
+
+  disable_ddl_transaction!
+
+  class Event < ActiveRecord::Base
+    include EachBatch
+
+    self.table_name = 'events'
+  end
+
+  def up
+    jobs = []
+
+    Event.each_batch(of: 1000) do |relation|
+      min, max = relation.pluck('MIN(id), MAX(id)').first
+
+      if jobs.length == BUFFER_SIZE
+        # We push multiple jobs at a time to reduce the time spent in
+        # Sidekiq/Redis operations. We're using this buffer based approach so we
+        # don't need to run additional queries for every range.
+        BackgroundMigrationWorker.perform_bulk(jobs)
+        jobs.clear
+      end
+
+      jobs << ['MigrateEventsToPushEventPayloads', [min, max]]
+    end
+
+    BackgroundMigrationWorker.perform_bulk(jobs) unless jobs.empty?
+  end
+
+  def down
+  end
+end

db/schema.rb

@@ -11,7 +11,7 @@
 #
 # It's strongly recommended that you check this file into your version control system.
 
-ActiveRecord::Schema.define(version: 20170807160457) do
+ActiveRecord::Schema.define(version: 20170809161910) do
 
   # These are extensions that must be enabled in order to support this database
   enable_extension "plpgsql"
@@ -28,13 +28,13 @@ ActiveRecord::Schema.define(version: 20170807160457) do
   end
 
   create_table "appearances", force: :cascade do |t|
-    t.string "title"
-    t.text "description"
+    t.string "title", null: false
+    t.text "description", null: false
     t.string "header_logo"
     t.string "logo"
     t.datetime "created_at", null: false
     t.datetime "updated_at", null: false
-    t.text "description_html"
+    t.text "description_html", null: false
     t.integer "cached_markdown_version"
   end
 
@@ -127,6 +127,7 @@ ActiveRecord::Schema.define(version: 20170807160457) do
     t.string "help_page_support_url"
     t.integer "performance_bar_allowed_group_id"
     t.boolean "password_authentication_enabled"
+    t.boolean "project_export_enabled", default: true, null: false
   end
 
   create_table "audit_events", force: :cascade do |t|
@@ -163,16 +164,18 @@ ActiveRecord::Schema.define(version: 20170807160457) do
 
   create_table "broadcast_messages", force: :cascade do |t|
     t.text "message", null: false
-    t.datetime "starts_at"
-    t.datetime "ends_at"
-    t.datetime "created_at"
-    t.datetime "updated_at"
+    t.datetime "starts_at", null: false
+    t.datetime "ends_at", null: false
+    t.datetime "created_at", null: false
+    t.datetime "updated_at", null: false
     t.string "color"
     t.string "font"
-    t.text "message_html"
+    t.text "message_html", null: false
     t.integer "cached_markdown_version"
   end
 
+  add_index "broadcast_messages", ["starts_at", "ends_at", "id"], name: "index_broadcast_messages_on_starts_at_and_ends_at_and_id", using: :btree
+
   create_table "chat_names", force: :cascade do |t|
     t.integer "user_id", null: false
     t.integer "service_id", null: false
@@ -530,10 +533,25 @@ ActiveRecord::Schema.define(version: 20170807160457) do
   add_index "events", ["action"], name: "index_events_on_action", using: :btree
   add_index "events", ["author_id"], name: "index_events_on_author_id", using: :btree
   add_index "events", ["created_at"], name: "index_events_on_created_at", using: :btree
-  add_index "events", ["project_id"], name: "index_events_on_project_id", using: :btree
+  add_index "events", ["project_id", "id"], name: "index_events_on_project_id_and_id", using: :btree
   add_index "events", ["target_id"], name: "index_events_on_target_id", using: :btree
   add_index "events", ["target_type"], name: "index_events_on_target_type", using: :btree
 
+  create_table "events_for_migration", force: :cascade do |t|
+    t.integer "project_id"
+    t.integer "author_id", null: false
+    t.integer "target_id"
+    t.datetime "created_at", null: false
+    t.datetime "updated_at", null: false
+    t.integer "action", limit: 2, null: false
+    t.string "target_type"
+  end
+
+  add_index "events_for_migration", ["action"], name: "index_events_for_migration_on_action", using: :btree
+  add_index "events_for_migration", ["author_id"], name: "index_events_for_migration_on_author_id", using: :btree
+  add_index "events_for_migration", ["project_id", "id"], name: "index_events_for_migration_on_project_id_and_id", using: :btree
+  add_index "events_for_migration", ["target_type", "target_id"], name: "index_events_for_migration_on_target_type_and_target_id", using: :btree
+
   create_table "feature_gates", force: :cascade do |t|
     t.string "feature_key", null: false
     t.string "key", null: false
@@ -1254,6 +1272,19 @@ ActiveRecord::Schema.define(version: 20170807160457) do
 
   add_index "protected_tags", ["project_id"], name: "index_protected_tags_on_project_id", using: :btree
 
+  create_table "push_event_payloads", id: false, force: :cascade do |t|
+    t.integer "commit_count", limit: 8, null: false
+    t.integer "event_id", null: false
+    t.integer "action", limit: 2, null: false
+    t.integer "ref_type", limit: 2, null: false
+    t.binary "commit_from"
+    t.binary "commit_to"
+    t.text "ref"
+    t.string "commit_title", limit: 70
+  end
+
+  add_index "push_event_payloads", ["event_id"], name: "index_push_event_payloads_on_event_id", unique: true, using: :btree
+
   create_table "redirect_routes", force: :cascade do |t|
     t.integer "source_id", null: false
     t.string "source_type", null: false
@@ -1654,6 +1685,8 @@ ActiveRecord::Schema.define(version: 20170807160457) do
   add_foreign_key "deployments", "projects", name: "fk_b9a3851b82", on_delete: :cascade
   add_foreign_key "environments", "projects", name: "fk_d1c8c1da6a", on_delete: :cascade
   add_foreign_key "events", "projects", name: "fk_0434b48643", on_delete: :cascade
+  add_foreign_key "events_for_migration", "projects", on_delete: :cascade
+  add_foreign_key "events_for_migration", "users", column: "author_id", name: "fk_edfd187b6f", on_delete: :cascade
   add_foreign_key "forked_project_links", "projects", column: "forked_to_project_id", name: "fk_434510edb0", on_delete: :cascade
   add_foreign_key "gpg_keys", "users", on_delete: :cascade
   add_foreign_key "gpg_signatures", "gpg_keys", on_delete: :nullify
@@ -1696,6 +1729,7 @@ ActiveRecord::Schema.define(version: 20170807160457) do
   add_foreign_key "protected_tag_create_access_levels", "protected_tags"
   add_foreign_key "protected_tag_create_access_levels", "users"
   add_foreign_key "protected_tags", "projects", name: "fk_8e4af87648", on_delete: :cascade
+  add_foreign_key "push_event_payloads", "events_for_migration", column: "event_id", name: "fk_36c74129da", on_delete: :cascade
   add_foreign_key "releases", "projects", name: "fk_47fe2a0596", on_delete: :cascade
   add_foreign_key "services", "projects", name: "fk_71cce407f9", on_delete: :cascade
   add_foreign_key "snippets", "projects", name: "fk_be41fd4bb7", on_delete: :cascade

doc/README.md

+---
+toc: false
+---
+
 # GitLab Documentation
 
 Welcome to [GitLab](https://about.gitlab.com/), a Git-based fully featured
@@ -26,6 +30,9 @@ Shortcuts to GitLab's most visited docs:
 | [Configuring `.gitlab-ci.yml`](ci/yaml/README.md) | [SSH authentication](ssh/README.md) |
 | [Using Docker images](ci/docker/using_docker_images.md) | [GitLab Pages](user/project/pages/index.md) |
 
+- [User documentation](user/index.md)
+- [Administrator documentation](#administrator-documentation)
+
 ## Getting started with GitLab
 
 - [GitLab Basics](gitlab-basics/README.md): Start working on your command line and on GitLab.
@@ -36,7 +43,6 @@ Shortcuts to GitLab's most visited docs:
 
 ### User account
 
-- [User documentation](user/index.md): Learn how to use GitLab and explore its features
 - [User account](user/profile/index.md): Manage your account
   - [Authentication](topics/authentication/index.md): Account security with two-factor authentication, setup your ssh keys and deploy keys for secure access to your projects.
   - [Profile settings](user/profile/index.md#profile-settings): Manage your profile settings, two factor authentication and more.

doc/administration/high_availability/nfs.md

@@ -42,6 +42,10 @@ GitLab does not recommend using EFS with GitLab.
   are allocated. For smaller volumes, users may experience decent performance
   for a period of time due to 'Burst Credits'. Over a period of weeks to months
   credits may run out and performance will bottom out.
+- To keep "Burst Credits" available, it may be necessary to provision more space
+  with 'dummy data'.  However, this may get expensive.
+- Another option to maintain "Burst Credits" is to use FS Cache on the server so
+  that AWS doesn't always have to go into EFS to access files.
 - For larger volumes, allocated IOPS may not be the problem. Workloads where
   many small files are written in a serialized manner are not well-suited for EFS.
   EBS with an NFS server on top will perform much better.

doc/api/events.md

@@ -79,7 +79,6 @@ Example response:
     "target_id":160,
     "target_type":"Issue",
     "author_id":25,
-    "data":null,
     "target_title":"Qui natus eos odio tempore et quaerat consequuntur ducimus cupiditate quis.",
     "created_at":"2017-02-09T10:43:19.667Z",
     "author":{
@@ -99,7 +98,6 @@ Example response:
     "target_id":159,
     "target_type":"Issue",
     "author_id":21,
-    "data":null,
     "target_title":"Nostrum enim non et sed optio illo deleniti non.",
     "created_at":"2017-02-09T10:43:19.426Z",
     "author":{
@@ -151,7 +149,6 @@ Example response:
     "target_id": 830,
     "target_type": "Issue",
     "author_id": 1,
-    "data": null,
     "target_title": "Public project search field",
     "author": {
       "name": "Dmitriy Zaporozhets",
@@ -166,7 +163,7 @@ Example response:
   {
     "title": null,
     "project_id": 15,
-    "action_name": "opened",
+    "action_name": "pushed",
     "target_id": null,
     "target_type": null,
     "author_id": 1,
@@ -179,31 +176,14 @@ Example response:
       "web_url": "http://localhost:3000/root"
     },
     "author_username": "john",
-    "data": {
-      "before": "50d4420237a9de7be1304607147aec22e4a14af7",
-      "after": "c5feabde2d8cd023215af4d2ceeb7a64839fc428",
-      "ref": "refs/heads/master",
-      "user_id": 1,
-      "user_name": "Dmitriy Zaporozhets",
-      "repository": {
-        "name": "gitlabhq",
-        "url": "git@dev.gitlab.org:gitlab/gitlabhq.git",
-        "description": "GitLab: self hosted Git management software. \r\nDistributed under the MIT License.",
-        "homepage": "https://dev.gitlab.org/gitlab/gitlabhq"
-      },
-      "commits": [
-        {
-          "id": "c5feabde2d8cd023215af4d2ceeb7a64839fc428",
-          "message": "Add simple search to projects in public area",
-          "timestamp": "2013-05-13T18:18:08+00:00",
-          "url": "https://dev.gitlab.org/gitlab/gitlabhq/commit/c5feabde2d8cd023215af4d2ceeb7a64839fc428",
-          "author": {
-            "name": "Dmitriy Zaporozhets",
-            "email": "dmitriy.zaporozhets@gmail.com"
-          }
-        }
-      ],
-      "total_commits_count": 1
+    "push_data": {
+      "commit_count": 1,
+      "action": "pushed",
+      "ref_type": "branch",
+      "commit_from": "50d4420237a9de7be1304607147aec22e4a14af7",
+      "commit_to": "c5feabde2d8cd023215af4d2ceeb7a64839fc428",
+      "ref": "master",
+      "commit_title": "Add simple search to projects in public area"
     },
     "target_title": null
   },
@@ -214,7 +194,6 @@ Example response:
     "target_id": 840,
     "target_type": "Issue",
     "author_id": 1,
-    "data": null,
     "target_title": "Finish & merge Code search PR",
     "author": {
       "name": "Dmitriy Zaporozhets",
@@ -233,7 +212,6 @@ Example response:
     "target_id": 1312,
     "target_type": "Note",
     "author_id": 1,
-    "data": null,
     "target_title": null,
     "created_at": "2015-12-04T10:33:58.089Z",
     "note": {
@@ -305,7 +283,6 @@ Example response:
     "target_iid":160,
     "target_type":"Issue",
     "author_id":25,
-    "data":null,
     "target_title":"Qui natus eos odio tempore et quaerat consequuntur ducimus cupiditate quis.",
     "created_at":"2017-02-09T10:43:19.667Z",
     "author":{
@@ -326,7 +303,6 @@ Example response:
     "target_iid":159,
     "target_type":"Issue",
     "author_id":21,
-    "data":null,
     "target_title":"Nostrum enim non et sed optio illo deleniti non.",
     "created_at":"2017-02-09T10:43:19.426Z",
     "author":{

doc/development/changelog.md

@@ -15,11 +15,14 @@ following format:
 title: "Going through change[log]s"
 merge_request: 1972
 author: Ozzy Osbourne
+type: added
 ```
 
 The `merge_request` value is a reference to a merge request that adds this
 entry, and the `author` key is used to give attribution to community
 contributors. **Both are optional**.
+The `type` field maps the category of the change,
+valid options are: added, fixed, changed, deprecated, removed, security, other. **Type field is mandatory**.
 
 Community contributors and core team members are encouraged to add their name to
 the `author` field. GitLab team members **should not**.
@@ -94,6 +97,19 @@ Its simplest usage is to provide the value for `title`:
 $ bin/changelog 'Hey DZ, I added a feature to GitLab!'
 ```
 
+At this point the script would ask you to select the category of the change (mapped to the `type` field in the entry):
+
+```text
+>> Please specify the category of your change:
+1. New feature
+2. Bug fix
+3. Feature change
+4. New deprecation
+5. Feature removal
+6. Security fix
+7. Other
+```
+
 The entry filename is based on the name of the current Git branch. If you run
 the command above on a branch called `feature/hey-dz`, it will generate a
 `changelogs/unreleased/feature-hey-dz.yml` file.
@@ -106,26 +122,29 @@ create changelogs/unreleased/my-feature.yml
 title: Hey DZ, I added a feature to GitLab!
 merge_request:
 author:
+type:
 ```
 If you're working on the GitLab EE repository, the entry will be added to
 `changelogs/unreleased-ee/` instead.
 
 #### Arguments
 
-| Argument            | Shorthand | Purpose                                       |
-| -----------------   | --------- | --------------------------------------------- |
-| [`--amend`]         |           | Amend the previous commit                     |
-| [`--force`]         | `-f`      | Overwrite an existing entry                   |
-| [`--merge-request`] | `-m`      | Set merge request ID                          |
-| [`--dry-run`]       | `-n`      | Don't actually write anything, just print     |
-| [`--git-username`]  | `-u`      | Use Git user.name configuration as the author |
-| [`--help`]          | `-h`      | Print help message                            |
+| Argument            | Shorthand | Purpose                                                                                                    |
+| -----------------   | --------- | ---------------------------------------------------------------------------------------------------------- |
+| [`--amend`]         |           | Amend the previous commit                                                                                  |
+| [`--force`]         | `-f`      | Overwrite an existing entry                                                                                |
+| [`--merge-request`] | `-m`      | Set merge request ID                                                                                       |
+| [`--dry-run`]       | `-n`      | Don't actually write anything, just print                                                                  |
+| [`--git-username`]  | `-u`      | Use Git user.name configuration as the author                                                              |
+| [`--type`]          | `-t`      | The category of the change, valid options are: added, fixed, changed, deprecated, removed, security, other |
+| [`--help`]          | `-h`      | Print help message                                                                                         |
 
 [`--amend`]: #-amend
 [`--force`]: #-force-or-f
 [`--merge-request`]: #-merge-request-or-m
 [`--dry-run`]: #-dry-run-or-n
 [`--git-username`]: #-git-username-or-u
+[`--type`]: #-type-or-t
 [`--help`]: #-help
 
 ##### `--amend`
@@ -147,6 +166,7 @@ create changelogs/unreleased/feature-hey-dz.yml
 title: Added an awesome new feature to GitLab
 merge_request:
 author:
+type:
 ```
 
 ##### `--force` or `-f`
@@ -164,6 +184,7 @@ create changelogs/unreleased/feature-hey-dz.yml
 title: Hey DZ, I added a feature to GitLab!
 merge_request: 1983
 author:
+type:
 ```
 
 ##### `--merge-request` or `-m`
@@ -178,6 +199,7 @@ create changelogs/unreleased/feature-hey-dz.yml
 title: Hey DZ, I added a feature to GitLab!
 merge_request: 1983
 author:
+type:
 ```
 
 ##### `--dry-run` or `-n`
@@ -192,6 +214,7 @@ create changelogs/unreleased/feature-hey-dz.yml
 title: Added an awesome new feature to GitLab
 merge_request:
 author:
+type:
 
 $ ls changelogs/unreleased/
 ```
@@ -211,6 +234,21 @@ create changelogs/unreleased/feature-hey-dz.yml
 title: Hey DZ, I added a feature to GitLab!
 merge_request:
 author: Jane Doe
+type:
+```
+
+##### `--type` or `-t`
+
+Use the **`--type`** or **`-t`** argument to provide the `type` value:
+
+```text
+$ bin/changelog 'Hey DZ, I added a feature to GitLab!' -t added
+create changelogs/unreleased/feature-hey-dz.yml
+---
+title: Hey DZ, I added a feature to GitLab!
+merge_request:
+author:
+type: added
 ```
 
 ### History and Reasoning

doc/development/testing.md

@@ -157,8 +157,9 @@ trade-off:
 
 - Unit tests are usually cheap, and you should consider them like the basement
   of your house: you need them to be confident that your code is behaving
-  correctly. However if you run only unit tests without integration / system tests, you might [miss] the [big] [picture]!
-- Integration tests are a bit more expensive, but don't abuse them. A feature test
+  correctly. However if you run only unit tests without integration / system
+  tests, you might [miss] the [big] [picture]!
+- Integration tests are a bit more expensive, but don't abuse them. A system test
   is often better than an integration test that is stubbing a lot of internals.
 - System tests are expensive (compared to unit tests), even more if they require
   a JavaScript driver. Make sure to follow the guidelines in the [Speed](#test-speed)
@@ -188,24 +189,34 @@ Please consult the [dedicated "Frontend testing" guide](./fe_guide/testing.md).
 ### General Guidelines
 
 - Use a single, top-level `describe ClassName` block.
-- Use `described_class` instead of repeating the class name being described
-  (_this is enforced by RuboCop_).
 - Use `.method` to describe class methods and `#method` to describe instance
   methods.
 - Use `context` to test branching logic.
-- Use multi-line `do...end` blocks for `before` and `after`, even when it would
-  fit on a single line.
 - Don't assert against the absolute value of a sequence-generated attribute (see [Gotchas](gotchas.md#dont-assert-against-the-absolute-value-of-a-sequence-generated-attribute)).
-- Don't supply the `:each` argument to hooks since it's the default.
-- Prefer `not_to` to `to_not` (_this is enforced by RuboCop_).
 - Try to match the ordering of tests to the ordering within the class.
 - Try to follow the [Four-Phase Test][four-phase-test] pattern, using newlines
   to separate phases.
-- Try to use `Gitlab.config.gitlab.host` rather than hard coding `'localhost'`
+- Use `Gitlab.config.gitlab.host` rather than hard coding `'localhost'`
+- Don't assert against the absolute value of a sequence-generated attribute (see
+  [Gotchas](gotchas.md#dont-assert-against-the-absolute-value-of-a-sequence-generated-attribute)).
+- Don't supply the `:each` argument to hooks since it's the default.
 - On `before` and `after` hooks, prefer it scoped to `:context` over `:all`
 
 [four-phase-test]: https://robots.thoughtbot.com/four-phase-test
 
+### Automatic retries and flaky tests detection
+
+On our CI, we use [rspec-retry] to automatically retry a failing example a few
+times (see [`spec/spec_helper.rb`] for the precise retries count).
+
+We also use a home-made `RspecFlaky::Listener` listener which records flaky
+examples in a JSON report file on `master` (`retrieve-tests-metadata` and `update-tests-metadata` jobs), and warns when a new flaky example
+is detected in any other branch (`flaky-examples-check` job). In the future, the
+`flaky-examples-check` job will not be allowed to fail.
+
+[rspec-retry]: https://github.com/NoRedInk/rspec-retry
+[`spec/spec_helper.rb`]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/spec_helper.rb
+
 ### `let` variables
 
 GitLab's RSpec suite has made extensive use of `let` variables to reduce
@@ -276,6 +287,15 @@ complexity of RSpec expectations.They should be placed under
 a certain type of specs only (e.g. features, requests etc.) but shouldn't be if
 they apply to multiple type of specs.
 
+#### have_gitlab_http_status
+
+Prefer `have_gitlab_http_status` over `have_http_status` because the former
+could also show the response body whenever the status mismatched. This would
+be very useful whenever some tests start breaking and we would love to know
+why without editing the source and rerun the tests.
+
+This is especially useful whenever it's showing 500 internal server error.
+
 ### Shared contexts
 
 All shared contexts should be be placed under `spec/support/shared_contexts/`.