Merge branch 'fix-git-blame-syntax-highlighting' into 'master'

Fix git blame syntax highlighting when different commits break up lines

### What does this MR do?

This MR fixes a bug where syntax highlighting would not work properly in certain lines when doing a `git blame` view of a file. Also, this MR reuses the Rugments lexer and formatter for each `git blame` commit block, which should make things faster and reduce overhead.

### Why was this MR needed?

When doing a `git blame`, GitLab feeds the Rugments lexer/formatter blocks of code based on the latest commit for each line.  However, this can cause lexer to fail because the state is cleared between each block of code, which causes `highlight` to fallback to the plaintext lexer. For example, notice this unhighlighted line in [this file](https://gitlab.common-lisp.net/cmucl/cmucl/blame/master/src/assembly/assemfile.lisp):

![image](https://gitlab.com/gitlab-org/gitlab-ce/uploads/b403fe5c2b883882a9eeea7e0409c583/image.png)

The fixed version looks like this:

![image](https://gitlab.com/stanhu/gitlab-ce/uploads/32c4d7c8ff15a7d59b364dd988f7c657/image.png)

This MR requires a patch to rugments to allow the `continue` option to be passed to the lexer.

* https://github.com/rumpelsepp/rugments/pull/24
* https://github.com/rumpelsepp/rugments/pull/23
* Also submitted to rouge: https://github.com/jneen/rouge/pull/267

### What are the relevant issue numbers?

Closes #1521

See merge request !697

CHANGELOG

@@ -12,6 +12,7 @@ v 7.12.0 (unreleased)
   - Disable "New Issue" and "New Merge Request" buttons when features are disabled in project settings (Stan Hu)
   - Remove Rack Attack monkey patches and bump to version 4.3.0 (Stan Hu)
   - Fix clone URL losing selection after a single click in Safari and Chrome (Stan Hu)
+  - Fix git blame syntax highlighting when different commits break up lines (Stan Hu)
   - Allow to configure location of the `.gitlab_shell_secret` file. (Jakub Jirutka)
   - Disabled expansion of top/bottom blobs for new file diffs
   - Update Asciidoctor gem to version 1.5.2. (Jakub Jirutka)

Gemfile

@@ -277,4 +277,4 @@ end
 gem "newrelic_rpm"
 
 gem 'octokit', '3.7.0'
-gem "rugments"
+gem "rugments", "~> 1.0.0.beta7"

Gemfile.lock

@@ -531,7 +531,7 @@ GEM
     rubyntlm (0.5.0)
     rubypants (0.2.0)
     rugged (0.22.2)
-    rugments (1.0.0.beta6)
+    rugments (1.0.0.beta7)
     safe_yaml (0.9.7)
     sanitize (2.1.0)
       nokogiri (>= 1.4.4)
@@ -781,7 +781,7 @@ DEPENDENCIES
   rqrcode-rails3
   rspec-rails (= 2.99)
   rubocop (= 0.28.0)
-  rugments
+  rugments (~> 1.0.0.beta7)
   sanitize (~> 2.0)
   sass-rails (~> 4.0.2)
   sdoc

app/helpers/blob_helper.rb

 module BlobHelper
-  def highlight(blob_name, blob_content, nowrap = false)
-    formatter = Rugments::Formatters::HTML.new(
+  def highlight(blob_name, blob_content, nowrap: false, continue: false)
+    @formatter ||= Rugments::Formatters::HTML.new(
       nowrap: nowrap,
       cssclass: 'code highlight',
       lineanchors: true,
@@ -8,11 +8,11 @@ module BlobHelper
     )
 
     begin
-      lexer = Rugments::Lexer.guess(filename: blob_name, source: blob_content)
-      result = formatter.format(lexer.lex(blob_content)).html_safe
+      @lexer ||= Rugments::Lexer.guess(filename: blob_name, source: blob_content).new
+      result = @formatter.format(@lexer.lex(blob_content, continue: continue)).html_safe
     rescue
       lexer = Rugments::Lexers::PlainText
-      result = formatter.format(lexer.lex(blob_content)).html_safe
+      result = @formatter.format(lexer.lex(blob_content)).html_safe
     end
 
     result

app/views/projects/blame/show.html.haml

@@ -32,5 +32,5 @@
                 %code
                   :erb
                     <% lines.each do |line| %>
-                      <%= highlight(@blob.name, line, true).html_safe %>
+                      <%= highlight(@blob.name, line, nowrap: true, continue: true).html_safe %>
                     <% end %>

spec/helpers/blob_helper_spec.rb

+require 'spec_helper'
+
+describe BlobHelper do
+  describe 'highlight' do
+    let(:blob_name) { 'test.lisp' }
+    let(:no_context_content) { ":type \"assem\"))" }
+    let(:blob_content) { "(make-pathname :defaults name\n#{no_context_content}" }
+    let(:split_content) { blob_content.split("\n") }
+
+    it 'should return plaintext for unknown lexer context' do
+      result = highlight(blob_name, no_context_content, nowrap: true, continue: false)
+      expect(result).to eq('<span id="LC1" class="line">:type &quot;assem&quot;))</span>')
+    end
+
+    it 'should highlight single block' do
+      expected = %Q[<span id="LC1" class="line"><span class="p">(</span><span class="nb">make-pathname</span> <span class="ss">:defaults</span> <span class="nv">name</span></span>
+<span id="LC2" class="line"><span class="ss">:type</span> <span class="s">&quot;assem&quot;</span><span class="p">))</span></span>]
+
+      expect(highlight(blob_name, blob_content, nowrap: true, continue: false)).to eq(expected)
+    end
+
+    it 'should highlight continued blocks' do
+      # Both lines have LC1 as ID since formatter doesn't support continue at the moment
+      expected = [
+        '<span id="LC1" class="line"><span class="p">(</span><span class="nb">make-pathname</span> <span class="ss">:defaults</span> <span class="nv">name</span></span>',
+        '<span id="LC1" class="line"><span class="ss">:type</span> <span class="s">&quot;assem&quot;</span><span class="p">))</span></span>'
+      ]
+
+      result = split_content.map{ |content| highlight(blob_name, content, nowrap: true, continue: true) }
+      expect(result).to eq(expected)
+    end
+  end
+end