Merge branch 'gitlab-ce-asciidoc-include' into 'master'

Add support for AsciiDoc include directive

Closes #18045

See merge request gitlab-org/gitlab-ce!28417

Gemfile

@@ -130,6 +130,7 @@ gem 'org-ruby', '~> 0.9.12'
 gem 'creole', '~> 0.5.0'
 gem 'wikicloth', '0.8.1'
 gem 'asciidoctor', '~> 1.5.8'
+gem 'asciidoctor-include-ext', '~> 0.3.1', require: false
 gem 'asciidoctor-plantuml', '0.0.8'
 gem 'rouge', '~> 3.1'
 gem 'truncato', '~> 0.7.11'

Gemfile.lock

@@ -67,6 +67,8 @@ GEM
       faraday_middleware-multi_json (~> 0.0)
       oauth2 (~> 1.0)
     asciidoctor (1.5.8)
+    asciidoctor-include-ext (0.3.1)
+      asciidoctor (>= 1.5.6, < 3.0.0)
     asciidoctor-plantuml (0.0.8)
       asciidoctor (~> 1.5)
     ast (2.4.0)
@@ -1024,6 +1026,7 @@ DEPENDENCIES
   apollo_upload_server (~> 2.0.0.beta3)
   asana (~> 0.8.1)
   asciidoctor (~> 1.5.8)
+  asciidoctor-include-ext (~> 0.3.1)
   asciidoctor-plantuml (= 0.0.8)
   attr_encrypted (~> 3.1.0)
   awesome_print

app/helpers/markup_helper.rb

@@ -263,6 +263,11 @@ module MarkupHelper
   end
 
   def asciidoc_unsafe(text, context = {})
+    context.merge!(
+      commit:         @commit,
+      ref:            @ref,
+      requested_path: @path
+    )
     Gitlab::Asciidoc.render(text, context)
   end
 

changelogs/unreleased/asciidoc-include-directive.yml

+---
+title: Add support for AsciiDoc include directive
+merge_request: 28417
+author: "Jakub Jirutka & Guillaume Grossetie"
+type: added

doc/user/asciidoc.md

+# AsciiDoc
+
+GitLab uses the [Asciidoctor](https://asciidoctor.org) gem to convert AsciiDoc content to HTML5.
+Consult the [Asciidoctor User Manual](https://asciidoctor.org/docs/user-manual) for a complete Asciidoctor reference.
+
+## Syntax
+
+Here's a brief reference of the most commonly used AsciiDoc syntax.
+You can find the full documentation for the AsciiDoc syntax at https://asciidoctor.org/docs.
+
+### Paragraphs
+
+```asciidoc
+A normal paragraph.
+Line breaks are not preserved.
+```
+
+Line comments, which are lines that start with `//`, are skipped:
+
+```
+// this is a comment
+```
+
+A blank line separates paragraphs.
+
+A paragraph with the `[%hardbreaks]` option will preserve line breaks:
+
+```asciidoc
+[%hardbreaks]
+This paragraph carries the `hardbreaks` option.
+Notice how line breaks are now preserved.
+```
+
+An indented (literal) paragraph disables text formatting,
+preserves spaces and line breaks, and is displayed in a
+monospaced font:
+
+```asciidoc
+ This literal paragraph is indented with one space.
+ As a consequence, *text formatting*, spaces,
+ and lines breaks will be preserved.
+```
+
+An admonition paragraph grabs the reader's attention:
+
+```asciidoc
+NOTE: This is a brief reference, please read the full documentation at https://asciidoctor.org/docs.
+
+TIP: Lists can be indented. Leading whitespace is not significant.
+```
+
+### Text Formatting
+
+**Constrained (applied at word boundaries)**
+
+```asciidoc
+*strong importance* (aka bold)
+_stress emphasis_ (aka italic)
+`monospaced` (aka typewriter text)
+"`double`" and '`single`' typographic quotes
++passthrough text+ (substitutions disabled)
+`+literal text+` (monospaced with substitutions disabled)
+```
+
+**Unconstrained (applied anywhere)**
+
+```asciidoc
+**C**reate+**R**ead+**U**pdate+**D**elete
+fan__freakin__tastic
+``mono``culture
+```
+
+**Replacements**
+
+```asciidoc
+A long time ago in a galaxy far, far away...
+(C) 1976 Arty Artisan
+I believe I shall--no, actually I won't.
+```
+
+**Macros**
+
+```asciidoc
+// where c=specialchars, q=quotes, a=attributes, r=replacements, m=macros, p=post_replacements, etc.
+The European icon:flag[role=blue] is blue & contains pass:[************] arranged in a icon:circle-o[role=yellow].
+The pass:c[->] operator is often referred to as the stabby lambda.
+Since `pass:[++]` has strong priority in AsciiDoc, you can rewrite pass:c,a,r[C++ => C{pp}].
+// activate stem support by adding `:stem:` to the document header
+stem:[sqrt(4) = 2]
+```
+
+### Attributes
+
+```asciidoc
+// define attributes in the document header
+:name: value
+```
+
+```asciidoc
+:url-gem: https://rubygems.org/gems/asciidoctor
+
+You can download and install Asciidoctor {asciidoctor-version} from {url-gem}.
+C{pp} is not required, only Ruby.
+Use a leading backslash to output a word enclosed in curly braces, like \{name}.
+```
+
+### Links
+
+```asciidoc
+https://example.org/page[A webpage]
+link:../path/to/file.txt[A local file]
+xref:document.adoc[A sibling document]
+mailto:hello@example.org[Email to say hello!]
+```
+
+### Anchors
+
+```asciidoc
+[[idname,reference text]]
+// or written using normal block attributes as `[#idname,reftext=reference text]`
+A paragraph (or any block) with an anchor (aka ID) and reftext.
+
+See <<idname>> or <<idname,optional text of internal link>>.
+
+xref:document.adoc#idname[Jumps to anchor in another document].
+
+This paragraph has a footnote.footnote:[This is the text of the footnote.]
+```
+
+### Lists
+
+#### Unordered
+
+```asciidoc
+* level 1
+** level 2
+*** level 3
+**** level 4
+***** etc.
+* back at level 1
++
+Attach a block or paragraph to a list item using a list continuation (which you can enclose in an open block).
+
+.Some Authors
+[circle]
+- Edgar Allen Poe
+- Sheri S. Tepper
+- Bill Bryson
+```
+
+#### Ordered
+
+```asciidoc
+. Step 1
+. Step 2
+.. Step 2a
+.. Step 2b
+. Step 3
+
+.Remember your Roman numerals?
+[upperroman]
+. is one
+. is two
+. is three
+```
+
+#### Checklist
+
+```asciidoc
+* [x] checked
+* [ ] not checked
+```
+#### Callout
+
+```asciidoc
+// enable callout bubbles by adding `:icons: font` to the document header
+[,ruby]
+----
+puts 'Hello, World!' # <1>
+----
+<1> Prints `Hello, World!` to the console.
+```
+
+#### Description
+
+```asciidoc
+first term:: description of first term
+second term::
+description of second term
+```
+### Document Structure
+
+#### Header
+
+```asciidoc
+= Document Title
+Author Name <author@example.org>
+v1.0, 2019-01-01
+```
+#### Sections
+
+```asciidoc
+= Document Title (Level 0)
+== Level 1
+=== Level 2
+==== Level 3
+===== Level 4
+====== Level 5
+== Back at Level 1
+```
+
+#### Includes
+
+```asciidoc
+include::basics.adoc[]
+
+// define -a allow-uri-read to allow content to be read from URI
+include::https://example.org/installation.adoc[]
+```
+### Blocks
+
+```asciidoc
+--
+open - a general-purpose content wrapper; useful for enclosing content to attach to a list item
+--
+```
+
+```asciidoc
+// recognized types include CAUTION, IMPORTANT, NOTE, TIP, and WARNING
+// enable admonition icons by setting `:icons: font` in the document header
+[NOTE]
+====
+admonition - a notice for the reader, ranging in severity from a tip to an alert
+====
+```
+
+```asciidoc
+====
+example - a demonstration of the concept being documented
+====
+```
+
+```asciidoc
+.Toggle Me
+[%collapsible]
+====
+collapsible - these details are revealed by clicking the title
+====
+```
+
+```asciidoc
+****
+sidebar - auxiliary content that can be read independently of the main content
+****
+```
+
+```asciidoc
+....
+literal - an exhibit that features program output
+....
+```
+
+```asciidoc
+----
+listing - an exhibit that features program input, source code, or the contents of a file
+----
+```
+
+```asciidoc
+[,language]
+----
+source - a listing that is embellished with (colorized) syntax highlighting
+----
+```
+
+```asciidoc
+\```language
+fenced code - a shorthand syntax for the source block
+\```
+```
+
+```asciidoc
+[,attribution,citetitle]
+____
+quote - a quotation or excerpt; attribution with title of source are optional
+____
+```
+
+```asciidoc
+[verse,attribution,citetitle]
+____
+verse - a literary excerpt, often a poem; attribution with title of source are optional
+____
+```
+
+```asciidoc
+++++
+pass - content passed directly to the output document; often raw HTML
+++++
+```
+
+```asciidoc
+// activate stem support by adding `:stem:` to the document header
+[stem]
+++++
+x = y^2
+++++
+```
+
+```asciidoc
+////
+comment - content which is not included in the output document
+////
+```
+
+### Tables
+
+```asciidoc
+.Table Attributes
+[cols=>1h;2d,width=50%,frame=topbot]
+|===
+| Attribute Name | Values
+
+| options
+| header,footer,autowidth
+
+| cols
+| colspec[;colspec;...]
+
+| grid
+| all \| cols \| rows \| none
+
+| frame
+| all \| sides \| topbot \| none
+
+| stripes
+| all \| even \| odd \| none
+
+| width
+| (0%..100%)
+
+| format
+| psv {vbar} csv {vbar} dsv
+|===
+```
+
+### Multimedia
+
+```asciidoc
+image::screenshot.png[block image,800,450]
+
+Press image:reload.svg[reload,16,opts=interactive] to reload the page.
+
+video::movie.mp4[width=640,start=60,end=140,options=autoplay]
+
+video::aHjpOzsQ9YI[youtube]
+
+video::300817511[vimeo]
+```
+
+### Breaks
+
+```asciidoc
+// thematic break (aka horizontal rule)
+---
+```
+
+```asciidoc
+// page break
+<<<
+```
+

doc/user/project/repository/index.md

@@ -68,7 +68,7 @@ according to the markup language.
 | Plain text | `txt` |
 | [Markdown](../../markdown.md) | `mdown`, `mkd`, `mkdn`, `md`, `markdown` |
 | [reStructuredText](http://docutils.sourceforge.net/rst.html) | `rst` |
-| [Asciidoc](https://asciidoctor.org/docs/what-is-asciidoc/) | `adoc`, `ad`, `asciidoc` |
+| [AsciiDoc](../../asciidoc.md) | `adoc`, `ad`, `asciidoc` |
 | [Textile](https://txstyle.org/) | `textile` |
 | [rdoc](http://rdoc.sourceforge.net/doc/index.html)  | `rdoc` |
 | [Orgmode](https://orgmode.org/) | `org` |

lib/gitlab/asciidoc.rb

 # frozen_string_literal: true
 
 require 'asciidoctor'
-require 'asciidoctor/converter/html5'
-require "asciidoctor-plantuml"
+require 'asciidoctor-plantuml'
+require 'asciidoctor/extensions'
+require 'gitlab/asciidoc/html5_converter'
 
 module Gitlab
   # Parser/renderer for the AsciiDoc format that uses Asciidoctor and filters
   # the resulting HTML through HTML pipeline filters.
   module Asciidoc
-    DEFAULT_ADOC_ATTRS = [
-      'showtitle', 'idprefix=user-content-', 'idseparator=-', 'env=gitlab',
-      'env-gitlab', 'source-highlighter=html-pipeline', 'icons=font',
-      'outfilesuffix=.adoc'
-    ].freeze
+    MAX_INCLUDE_DEPTH = 5
+    DEFAULT_ADOC_ATTRS = {
+        'showtitle' => true,
+        'idprefix' => 'user-content-',
+        'idseparator' => '-',
+        'env' => 'gitlab',
+        'env-gitlab' => '',
+        'source-highlighter' => 'html-pipeline',
+        'icons' => 'font',
+        'outfilesuffix' => '.adoc',
+        'max-include-depth' => MAX_INCLUDE_DEPTH
+    }.freeze
 
     # Public: Converts the provided Asciidoc markup into HTML.
     #
     # input         - the source text in Asciidoc format
+    # context       - :commit, :project, :ref, :requested_path
     #
     def self.render(input, context)
+      extensions = proc do
+        include_processor ::Gitlab::Asciidoc::IncludeProcessor.new(context)
+      end
+
       asciidoc_opts = { safe: :secure,
                         backend: :gitlab_html5,
-                        attributes: DEFAULT_ADOC_ATTRS }
+                        attributes: DEFAULT_ADOC_ATTRS,
+                        extensions: extensions }
 
       context[:pipeline] = :ascii_doc
 
@@ -40,29 +54,5 @@ module Gitlab
         conf.txt_enable = false
       end
     end
-
-    class Html5Converter < Asciidoctor::Converter::Html5Converter
-      extend Asciidoctor::Converter::Config
-
-      register_for 'gitlab_html5'
-
-      def stem(node)
-        return super unless node.style.to_sym == :latexmath
-
-        %(<pre#{id_attribute(node)} data-math-style="display"><code>#{node.content}</code></pre>)
-      end
-
-      def inline_quoted(node)
-        return super unless node.type.to_sym == :latexmath
-
-        %(<code#{id_attribute(node)} data-math-style="inline">#{node.text}</code>)
-      end
-
-      private
-
-      def id_attribute(node)
-        node.id ? %( id="#{node.id}") : nil
-      end
-    end
   end
 end

lib/gitlab/asciidoc/html5_converter.rb

+# frozen_string_literal: true
+
+require 'asciidoctor'
+require 'asciidoctor/converter/html5'
+
+module Gitlab
+  module Asciidoc
+    class Html5Converter < Asciidoctor::Converter::Html5Converter
+      extend Asciidoctor::Converter::Config
+
+      register_for 'gitlab_html5'
+
+      def stem(node)
+        return super unless node.style.to_sym == :latexmath
+
+        %(<pre#{id_attribute(node)} data-math-style="display"><code>#{node.content}</code></pre>)
+      end
+
+      def inline_quoted(node)
+        return super unless node.type.to_sym == :latexmath
+
+        %(<code#{id_attribute(node)} data-math-style="inline">#{node.text}</code>)
+      end
+
+      private
+
+      def id_attribute(node)
+        node.id ? %( id="#{node.id}") : nil
+      end
+    end
+  end
+end

lib/gitlab/asciidoc/include_processor.rb

+# frozen_string_literal: true
+
+require 'asciidoctor/include_ext/include_processor'
+
+module Gitlab
+  module Asciidoc
+    # Asciidoctor extension for processing includes (macro include::[]) within
+    # documents inside the same repository.
+    class IncludeProcessor < Asciidoctor::IncludeExt::IncludeProcessor
+      extend ::Gitlab::Utils::Override
+
+      def initialize(context)
+        super(logger: Gitlab::AppLogger)
+
+        @context = context
+        @repository = context[:project].try(:repository)
+
+        # Note: Asciidoctor calls #freeze on extensions, so we can't set new
+        # instance variables after initialization.
+        @cache = {
+            uri_types: {}
+        }
+      end
+
+      protected
+
+      override :include_allowed?
+      def include_allowed?(target, reader)
+        doc = reader.document
+
+        return false if doc.attributes.fetch('max-include-depth').to_i < 1
+        return false if target_uri?(target)
+
+        true
+      end
+
+      override :resolve_target_path
+      def resolve_target_path(target, reader)
+        return unless repository.try(:exists?)
+
+        base_path = reader.include_stack.empty? ? requested_path : reader.file
+        path = resolve_relative_path(target, base_path)
+
+        path if Gitlab::Git::Blob.find(repository, ref, path)
+      end
+
+      override :read_lines
+      def read_lines(filename, selector)
+        blob = read_blob(ref, filename)
+
+        if selector
+          blob.data.each_line.select.with_index(1, &selector)
+        else
+          blob.data
+        end
+      end
+
+      override :unresolved_include!
+      def unresolved_include!(target, reader)
+        reader.unshift_line("*[ERROR: include::#{target}[] - unresolved directive]*")
+      end
+
+      private
+
+      attr_accessor :context, :repository, :cache
+
+      # Gets a Blob at a path for a specific revision.
+      # This method will check that the Blob exists and contains readable text.
+      #
+      # revision - The String SHA1.
+      # path     - The String file path.
+      #
+      # Returns a Blob
+      def read_blob(ref, filename)
+        blob = repository&.blob_at(ref, filename)
+
+        raise 'Blob not found' unless blob
+        raise 'File is not readable' unless blob.readable_text?
+
+        blob
+      end
+
+      # Resolves the given relative path of file in repository into canonical
+      # path based on the specified base_path.
+      #
+      # Examples:
+      #
+      #   # File in the same directory as the current path
+      #   resolve_relative_path("users.adoc", "doc/api/README.adoc")
+      #   # => "doc/api/users.adoc"
+      #
+      #   # File in the same directory, which is also the current path
+      #   resolve_relative_path("users.adoc", "doc/api")
+      #   # => "doc/api/users.adoc"
+      #
+      #   # Going up one level to a different directory
+      #   resolve_relative_path("../update/7.14-to-8.0.adoc", "doc/api/README.adoc")
+      #   # => "doc/update/7.14-to-8.0.adoc"
+      #
+      # Returns a String
+      def resolve_relative_path(path, base_path)
+        p = Pathname(base_path)
+        p = p.dirname unless p.extname.empty?
+        p += path
+
+        p.cleanpath.to_s
+      end
+
+      def current_commit
+        cache[:current_commit] ||= context[:commit] || repository&.commit(ref)
+      end
+
+      def ref
+        context[:ref] || context[:project].default_branch
+      end
+
+      def requested_path
+        cache[:requested_path] ||= Addressable::URI.unescape(context[:requested_path])
+      end
+
+      def uri_type(path)
+        cache[:uri_types][path] ||= current_commit&.uri_type(path)
+      end
+    end
+  end
+end

spec/lib/gitlab/asciidoc_spec.rb

@@ -3,20 +3,23 @@ require 'nokogiri'
 
 module Gitlab
   describe Asciidoc do
-    let(:input) { '<b>ascii</b>' }
-    let(:context) { {} }
-    let(:html) { 'H<sub>2</sub>O' }
+    include FakeBlobHelpers
+
+    before do
+      allow_any_instance_of(ApplicationSetting).to receive(:current).and_return(::ApplicationSetting.create_from_defaults)
+    end
 
     context "without project" do
-      before do
-        allow_any_instance_of(ApplicationSetting).to receive(:current).and_return(::ApplicationSetting.create_from_defaults)
-      end
+      let(:input) { '<b>ascii</b>' }
+      let(:context) { {} }
+      let(:html) { 'H<sub>2</sub>O' }
 
       it "converts the input using Asciidoctor and default options" do
         expected_asciidoc_opts = {
             safe: :secure,
             backend: :gitlab_html5,
-            attributes: described_class::DEFAULT_ADOC_ATTRS
+            attributes: described_class::DEFAULT_ADOC_ATTRS,
+            extensions: be_a(Proc)
         }
 
         expect(Asciidoctor).to receive(:convert)
@@ -30,7 +33,8 @@ module Gitlab
           expected_asciidoc_opts = {
               safe: :secure,
               backend: :gitlab_html5,
-              attributes: described_class::DEFAULT_ADOC_ATTRS
+              attributes: described_class::DEFAULT_ADOC_ATTRS,
+              extensions: be_a(Proc)
           }
 
           expect(Asciidoctor).to receive(:convert)
@@ -105,6 +109,174 @@ module Gitlab
       end
     end
 
+    context 'with project' do
+      let(:context) do
+        {
+          commit:         commit,
+          project:        project,
+          ref:            ref,
+          requested_path: requested_path
+        }
+      end
+      let(:commit)         { project.commit(ref) }
+      let(:project)        { create(:project, :repository) }
+      let(:ref)            { 'asciidoc' }
+      let(:requested_path) { '/' }
+
+      context 'include directive' do
+        subject(:output) { render(input, context) }
+
+        let(:input) { "Include this:\n\ninclude::#{include_path}[]" }
+
+        before do
+          current_file = requested_path
+          current_file += 'README.adoc' if requested_path.end_with? '/'
+
+          create_file(current_file, "= AsciiDoc\n")
+        end
+
+        context 'with path to non-existing file' do
+          let(:include_path) { 'not-exists.adoc' }
+
+          it 'renders Unresolved directive placeholder' do
+            is_expected.to include("<strong>[ERROR: include::#{include_path}[] - unresolved directive]</strong>")
+          end
+        end
+
+        shared_examples :invalid_include do
+          let(:include_path) { 'dk.png' }
+
+          before do
+            allow(project.repository).to receive(:blob_at).and_return(blob)
+          end
+
+          it 'does not read the blob' do
+            expect(blob).not_to receive(:data)
+          end
+
+          it 'renders Unresolved directive placeholder' do
+            is_expected.to include("<strong>[ERROR: include::#{include_path}[] - unresolved directive]</strong>")
+          end
+        end
+
+        context 'with path to a binary file' do
+          let(:blob) { fake_blob(path: 'dk.png', binary: true) }
+          include_examples :invalid_include
+        end
+
+        context 'with path to file in external storage' do
+          let(:blob) { fake_blob(path: 'dk.png', lfs: true) }
+
+          before do
+            allow(Gitlab.config.lfs).to receive(:enabled).and_return(true)
+            project.update_attribute(:lfs_enabled, true)
+          end
+
+          include_examples :invalid_include
+        end
+
+        context 'with path to a textual file' do
+          let(:include_path) { 'sample.adoc' }
+
+          before do
+            create_file(file_path, "Content from #{include_path}")
+          end
+
+          shared_examples :valid_include do
+            [
+              ['/doc/sample.adoc',  'doc/sample.adoc',     'absolute path'],
+              ['sample.adoc',       'doc/api/sample.adoc', 'relative path'],
+              ['./sample.adoc',     'doc/api/sample.adoc', 'relative path with leading ./'],
+              ['../sample.adoc',    'doc/sample.adoc',     'relative path to a file up one directory'],
+              ['../../sample.adoc', 'sample.adoc',         'relative path for a file up multiple directories']
+            ].each do |include_path_, file_path_, desc|
+
+              context "the file is specified by #{desc}" do
+                let(:include_path) { include_path_ }
+                let(:file_path) { file_path_ }
+
+                it 'includes content of the file' do
+                  is_expected.to include('<p>Include this:</p>')
+                  is_expected.to include("<p>Content from #{include_path}</p>")
+                end
+              end
+            end
+          end
+
+          context 'when requested path is a file in the repo' do
+            let(:requested_path) { 'doc/api/README.adoc' }
+
+            include_examples :valid_include
+
+            context 'without a commit (only ref)' do
+              let(:commit) { nil }
+              include_examples :valid_include
+            end
+          end
+
+          context 'when requested path is a directory in the repo' do
+            let(:requested_path) { 'doc/api/' }
+
+            include_examples :valid_include
+
+            context 'without a commit (only ref)' do
+              let(:commit) { nil }
+              include_examples :valid_include
+            end
+          end
+        end
+
+        context 'recursive includes with relative paths' do
+          let(:input) do
+            <<~ADOC
+              Source: requested file
+
+              include::doc/README.adoc[]
+
+              include::license.adoc[]
+            ADOC
+          end
+
+          before do
+            create_file 'doc/README.adoc', <<~ADOC
+              Source: doc/README.adoc
+
+              include::../license.adoc[]
+
+              include::api/hello.adoc[]
+            ADOC
+            create_file 'license.adoc', <<~ADOC
+              Source: license.adoc
+            ADOC
+            create_file 'doc/api/hello.adoc', <<~ADOC
+              Source: doc/api/hello.adoc
+
+              include::./common.adoc[]
+            ADOC
+            create_file 'doc/api/common.adoc', <<~ADOC
+              Source: doc/api/common.adoc
+            ADOC
+          end
+
+          it 'includes content of the included files recursively' do
+            expect(output.gsub(/<[^>]+>/, '').gsub(/\n\s*/, "\n").strip).to eq <<~ADOC.strip
+              Source: requested file
+              Source: doc/README.adoc
+              Source: license.adoc
+              Source: doc/api/hello.adoc
+              Source: doc/api/common.adoc
+              Source: license.adoc
+            ADOC
+          end
+        end
+
+        def create_file(path, content)
+          project.repository.create_file(project.creator, path, content,
+            message: "Add #{path}", branch_name: 'asciidoc')
+        end
+      end
+    end
+
     def render(*args)
       described_class.render(*args)
     end