How to Contribute to Solr Documentation | Apache Solr Reference Guide 7.5

The Lucene/Solr project has made it easy for anyone to contribute to the Solr Reference Guide with a patch.

The Guide is written in simple AsciiDoc-formatted files, and the source lives in the main Lucene/Solr source repository, right alongside the code.

The following sections give an overview of how to work with AsciiDoc-format files, how our documentation is generated, and how to do a Ref Guide release.

Once you have a patch you’d like to contribute, you can submit it with a JIRA issue, the same as any other patch. See also the Solr wiki section on How To Contribute for information on submitting patches.

Working with AsciiDoc Files

AsciiDoc Syntax Cheatsheet

The definitive manual on AsciiDoc syntax is in the Asciidoctor User Manual. To help people get started, however, here is a simpler cheat sheet.

AsciiDoc vs Asciidoctor Syntax

We use tools from the Asciidoctor project to build the HTML and PDF versions of the Ref Guide. Asciidoctor is a Ruby port of the original AsciiDoc project, which was mostly abandoned several years ago.

While much of the syntax between the two is the same, there are many conventions supported by Asciidoctor that did not exist in AsciiDoc. While the Asciidoctor project has tried to provide back-compatibility with the older project, that may not be true forever. For this reason, it’s strongly recommended to only use the Asciidoctor User Manual as a reference for any syntax that’s not described here.

Basic AsciiDoc Syntax

Bold

Put asterisks around text to make it bold.

More info: http://asciidoctor.org/docs/user-manual/#bold-and-italic

Italics

Use underlines on either side of a string to put text into italics.

More info: http://asciidoctor.org/docs/user-manual/#bold-and-italic

Headings

Equal signs (=) are used for heading levels. Each equal sign is a level. Each page can only have one top level (i.e., only one section with a single =).

Levels should be appropriately nested. During the build, validation occurs to ensure that level 3s are preceded by level 2s, level 4s are preceded by level 3s, etc. Including out-of-sequence heading levels (such as a level 3 then a level 5) will not fail the build, but will produce an error.

More info: http://asciidoctor.org/docs/user-manual/#sections

Code Examples

Use backticks ` for text that should be monospaced, such as code or a class name in the body of a paragraph.

More info: http://asciidoctor.org/docs/user-manual/#mono

Longer code examples can be separated from text with source blocks. These allow defining the syntax being used so the code is properly highlighted.

Example Source Block

[source,xml]
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

If your code block will include line breaks, put 4 hyphens (----) before and after the entire block.

More info: http://asciidoctor.org/docs/user-manual/#source-code-blocks

Source Block Syntax Highlighting

The PDF and HTML outputs use Pygments to add syntax highlighting to code examples. This is done by adding the language of the code block after the source, as shown in the above example source block (xml in that case).

Pygments has a long selection of lexers available. You can see the full list at http://pygments.org/docs/lexers. Use one of the valid short names to get syntax highlighting for that language.

Ideally, we will have an appropriate lexer to use for all source blocks, but that’s not possible. When in doubt, choose text, or leave it blank.

Importing Code Snippets from Other Files

The build system has the ability to "include" snippets located in other files — even non-AsciiDoc files such as *.java source code files.

Snippets are bounded by "tag" comments placed at the start and end of the section you would like to import. Opening tags look like: // tag::snippetName[]. Closing tags follow the format: // end::snippetName[]. Snippets can be dropped into an .adoc file using an include directive, following the format: include::PathToFileWithSnippet[tag=snippetName]. Note that when relative paths are provided in these directives, those paths are resolved relative to the location of the AsciiDoc file that the include appears in.

Snippets can be included directly from any file in the Lucene/Solr GIT repo by refering to a solr-root-path variable prior to the file path, for example:

[source,java,indent=0]
----
include::{solr-root-path}core/src/java/org/apache/solr/core/SolrCore.java[tag=something]
----

When building the PDF or HTML versions of the Guide, the solr-root-path attribute will be automatically set correctly for the (temporary) build/solr-ref-guide/content directory used by Ant.

In order for editors (such as ATOM) to be able to offer "live preview" of the *.adoc files using these includes, the solr-root-path attribute must also be set as a document level attribute in each file, with the correct relative path.

For example, using-solrj.adoc sets solr-root-path in it’s header, along with an example-source-dir attribute (that depends on solr-root-path) in order to reduce redundancy in the many include:: directives it specifies…

= Using SolrJ
:solr-root-path: ../../
:example-source-dir: {solr-root-path}solrj/src/test/org/apache/solr/client/ref_guide_examples/
...
[source,java,indent=0]
----
include::{example-source-dir}UsingSolrJRefGuideExamplesTest.java[tag=solrj-solrclient-timeouts]
----
...
[source,java,indent=0]
----
include::{example-source-dir}UsingSolrJRefGuideExamplesTest.java[tag=solrj-other-apis]
----
...

For more information on the include directive, see the documentation at http://asciidoctor.org/docs/user-manual/#include-partial.

Block Titles

Titles can be added to most blocks (images, source blocks, tables, etc.) by simply prefacing the title with a period (.). For example, to add a title to the source block example above:

.Example ID field
[source,xml]
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />

Links

Link to Sites on the Internet

When converting content to HTML or PDF, Asciidoctor will automatically render many link types (such as http: and mailto:) without any additional syntax.

However, you can add a name to a link by adding the URI followed by square brackets:

http://lucene.apache.org/solr[Solr Website]

Link to Other Pages/Sections of the Guide

A warning up front, linking to other pages can be a little bit painful. There are slightly different rules depending on the type of link you want to create, and where you are linking from.

The build process includes a validation for internal or inter-page links, so if you can build the docs locally, you can use that to verify you constructed your link properly (or pay attention to the Jenkins build after your commit).

With all of the below examples, you can add text to display as the link title by adding a comma after the section reference followed by the display text, as in:

<<schema-api.adoc#modify-the-schema,Modify the Schema>>

Link to a Section on the Same Page

To link to an anchor (or section title) on the same page, you can simply use double angle brackets (<< >>) around the anchor/heading/section title you want to link to. Any section title (a heading that starts with equal signs) automatically becomes an anchor during conversion and is available for deep linking.

Example

If I have a section on a page that looks like this (from defining-fields.adoc):

== Field Properties

Field definitions can have the following properties:

To link to this section from another part of the same defining-fields.adoc page, I simply need to put the section title in double angle brackets, as in:

See also the <<Field Properties>> section.

The section title will be used as the display text; to customize that add a comma after the the section title, then the text you want used for display.

More info: http://asciidoctor.org/docs/user-manual/#internal-cross-references

Link to a Section with an Anchor ID

When linking to any section (on the same page or another one), you must also be aware of any pre-defined anchors that may be in use (these will be in double brackets, like [[ ]]). When the page is converted, those will be the references your link needs to point to.

Example

Take this example from configsets-api.adoc:

[[configsets-create]]
== Create a ConfigSet

To link to this section, there are two approaches depending on where you are linking from:

From the same page, simply use the anchor name: <<configsets-create>>.
From another page, use the page name and the anchor name: <<configsets-api.adoc#configsets-create>>.

Link to Another Page

To link to another page or a section on another page, you must refer to the full filename and refer to the section you want to link to.

Unfortunately, when you want to refer the reader to another page without deep-linking to a section, you cannot simply put the other file name in angle brackets and call it a day. This is due to the PDF conversion - once all the pages are combined into one big page for one big PDF, the lack of a specific reference causes inter-page links to fail.

So, you must always link to a specific section. If all you want is a reference to the top of another page, you can use the implicit id of the page — the filename w/o the .adoc extension — as your anchor reference.

Example

To construct a link to the upgrading-solr.adoc page, we need to refer to the file name (upgrading-solr.adoc), then use the page id (upgrading-solr) as the anchor reference. As in:

For more information about upgrades, see <<upgrading-solr.adoc#upgrading-solr>>.

Link to a Section on Another Page

Linking to a section is the same conceptually as linking to the top of a page, you just need to take a little extra care to format the anchor ID in your link reference properly.

When you link to a section on another page, you must make a simple conversion of the title into the format the section ID will be created during the conversion. These are the rules that transform the sections:

All characters are lower-cased.
- Using security.json with Solr becomes using security.json with solr
All non-alpha characters are removed, with the exception of hyphens (so all periods, commas, ampersands, parentheses, etc., are stripped).
- using security.json with solr becomes using security json with solr
All whitespaces are replaced with hyphens.
- using security json with solr becomes using-security-json-with-solr

Example

The file schema-api.adoc has a section "Modify the Schema" that looks like this:

== Modify the Schema

`POST /_collection_/schema`

To link from to this section from another page, you would create a link structured like this:

the file name of the page with the section (schema-api.adoc),
then the hash symbol (#),
then the converted section title (modify-the-schema),
then a comma and any link title for display.

The link in context would look like this:

For more information, see the section <<schema-api.adoc#modify-the-schema,Modify the Schema>>.

More info: http://asciidoctor.org/docs/user-manual/#inter-document-cross-references

Ordered and Unordered Lists

AsciiDoc supports three types of lists:

Unordered lists
Ordered lists
Labeled lists

Each type of list can be mixed with the other types. So, you could have an ordered list inside a labeled list if necessary.

Unordered Lists

Simple bulleted lists need each line to start with an asterisk (*). It should be the first character of the line, and be followed by a space.

These lists also need to be separated from the

More info: http://asciidoctor.org/docs/user-manual/#unordered-lists

Ordered Lists

Numbered lists need each line to start with a period (.). It should be the first character of the line, and be followed by a space.

This style is preferred over manually numbering your list.

More info: http://asciidoctor.org/docs/user-manual/#ordered-lists

Labeled Lists

These are like question & answer lists or glossary definitions. Each line should start with the list item followed by double colons (::), then a space or new line.

Labeled lists can be nested by adding an additional colon (such as :::, etc.).

If your content will span multiple paragraphs or include source blocks, etc., you will want to add a plus sign (+) to keep the sections together for your reader.

We prefer this style of list for parameters because it allows more freedom in how you present the details for each parameter. For example, it supports ordered or unordered lists inside it automatically, and you can include multiple paragraphs and source blocks without trying to cram them into a smaller table cell.

More info: http://asciidoctor.org/docs/user-manual/#labeled-list

Images

There are two ways to include an image: inline or as a block.

Inline images are those where text will flow around the image. Block images are those that appear on their own line, set off from any other text on the page.

Both approaches use the image tag before the image filename, but the number of colons after image define if it is inline or a block. Inline images use one colon (image:), while block images use two colons (image::).

Block images automatically include a caption label and a number (such as Figure 1). If a block image includes a title, it will be included as the text of the caption.

Optional attributes allow you to set the alt text, the size of the image, if it should be a link, float and alignment.

More info: http://asciidoctor.org/docs/user-manual/#images

Tables

Tables can be complex, but it is pretty easy to make a basic table that fits most needs.

Basic Tables

The basic structure of a table is similar to Markdown, with pipes (|) delimiting columns between rows:

|===
| col 1 row 1 | col 2 row 1|
| col 1 row 2 | col 2 row 2|
|===

Note the use of |=== at the start and end. For basic tables that’s not exactly required, but it does help to delimit the start and end of the table in case you accidentally introduce (or maybe prefer) spaces between the rows.

Header Rows

To add a header to a table, you need only set the header attribute at the start of the table:

[options="header"]
|===
| header col 1 | header col 2|
| col 1 row 1 | col 2 row 1|
| col 1 row 2 | col 2 row 2|
|===

Defining Column Styles

If you need to define specific styles to all rows in a column, you can do so with the attributes.

This example will center all content in all rows:

[cols="2*^" options="header"]
|===
| header col 1 | header col 2|
| col 1 row 1 | col 2 row 1|
| col 1 row 2 | col 2 row 2|
|===

Alignments or any other styles can be applied only to a specific column. For example, this would only center the last column of the table:

[cols="2*,^" options="header"]
|===
| header col 1 | header col 2|
| col 1 row 1 | col 2 row 1|
| col 1 row 2 | col 2 row 2|
|===

Many more examples of formatting:

Columns: http://asciidoctor.org/docs/user-manual/#cols-format
Cells: http://asciidoctor.org/docs/user-manual/#cell

More Options

Tables can also be given footer rows, borders, and captions. You can determine the width of columns, or the width of the table as a whole.

CSV or DSV can also be used instead of formatting the data in pipes.

More info: http://asciidoctor.org/docs/user-manual/#tables

Admonitions (Notes, Warnings)

AsciiDoc supports several types of callout boxes, called "admonitions":

NOTE
TIP
IMPORTANT
CAUTION
WARNING

It is enough to start a paragraph with one of these words followed by a colon (such as NOTE:). When it is converted to HTML or PDF, those sections will be formatted properly - indented from the main text and showing an icon inline.

You can add titles to admonitions by making it an admonition block. The structure of an admonition block is like this:

.Title of Note
[NOTE]
====
Text of note
====

In this example, the type of admonition is included in square brackets ([NOTE]), and the title is prefixed with a period. Four equal signs give the start and end points of the note text (which can include new lines, lists, code examples, etc.).

More info: http://asciidoctor.org/docs/user-manual/#admonition

Tools for Working with AsciiDoc Files

AsciiDoc vs Asciidoctor

The Solr Ref Guide is written in AsciiDoc format. This format is generally considered an extension of Markdown, because it has support for tables of contents, better table support, and other features that make it more appropriate for writing technical documentation.

We are using a version of the AsciiDoc syntax along with tools from an open source project called Asciidoctor. This provides full support for the AsciiDoc syntax, but replaces the original Python processor with one written in Ruby. There is a Java implementation, known as AsciidoctorJ. Further extensions from the original AsciiDoc project include support for font-based icons and UI elements.

Helpful Tools

You can write AsciiDoc without any special tools. It’s simply text, with familiar syntax for bold (*) and italics (_).

Having some tools in your editor is helpful, though.

Doc Preview

This allows you to see your document in something close to what it might appear like when output to HTML.

The following information is from http://asciidoctor.org/docs/editing-asciidoc-with-live-preview.

Atom has AsciiDoc Preview, which gives you a panel that updates as you type. There are also a couple of other plugins to support AsciiDoc format and auto-complete.
There is a Live Preview browser plugin for Chrome, Firefox and Opera which allow you to open your AsciiDoc page in the browser. It will also update as you type.
There is also an Intellij IDEA plugin to support AsciiDoc format.

Modifying the Publication Formats

The Solr Reference Guide is published in two formats: HTML and PDF. Different tools are used for each.

Making Changes to HTML Version

The Solr Ref Guide uses Jekyll to build the HTML version of the site.

What is Jekyll?

Jekyll is a static site generator, meaning that it takes some set of documents and produces HTML pages. It allows for templating of the pages, so each page has the same look and feel without having to code headers, footers, logos, etc., into every page.

Jekyll is an open source project written in Ruby, online at https://jekyllrb.com/.

How We Use Jekyll

The following sections describe the main features of Jekyll that you will encounter while working with the Solr Ref Guide.

Jekyll-Asciidoctor Plugin

We use a plugin for Jekyll from the Asciidoctor project to integrate Jekyll with Asciidoc formatted content. The source for the plugin is available at https://github.com/asciidoctor/jekyll-asciidoc.

This plugin allows us to use Asciidoctor-style variables with Jekyll, instead of having to maintain two sets of the same variables (one for HTML version and another for PDF version).

_config.yml

The _config.yml is a global configuration file that drives many of the options used when building the site (particularly in our use of Jekyll).

We have template-ized _config.yml, so in our use of Jekyll you will find it in solr/solr-ref-guide/src as _config.yml.template. This allows us to define some variables during the build and use common Lucene/Solr build parameters (such as project versions).

Front Matter

Front matter for Jekyll is similar to a header that defines the title of the page, and other variables that may be helpful (or even required) when rendering the page.

Every document that will be converted to HTML must include at least the page title at the top of the page. Page titles are defined with a single equal sign (=) followed by the title that will appear at the top of the page (such as = Topic of the Page).

Many guides to Jekyll also say that defining the layout in the front matter is required. However, since we only use one layout for all pages, we have defined this as a default.

The Solr Ref Guide uses the front matter to define some custom attributes on a per page basis:

page-children - ordered list of child pages, this is used to build the site navigation menu that appears to the left of each page’s content (and to order the pages in the PDF)

Other page-level elements can also be defined, such as an Asciidoctor attribute that should apply only to that page, but are not needed on a regular basis.

The format for adding any parameter to the front matter is to use colons on both sides of the parameter, followed by the value for the parameter (such as :page-toc: false).

There are some optional custom attributes that can be defined in pages to affect the Table of Contents presentation in Jekyll:

page-toclevels

Changes how "deep" the TOC will be in terms of nested section/sub-section titles (default = 2).

page-tocclass

Changes the CSS class applied to the TOC. The options are:

normal, the default, resulting in the class name toc-normal being used in the HTML.
right, to put the TOC on the right side of the page.
column, to make a very long TOC span 3 columns at the top of the page.

page-toc

If this is false, then no TOCs will be generated for the page at all. The default is true, so can usually be left undefined.

{section-toc}

A special macro that can be used anywhere in a page to create an "In this Section" TOC covering only the sub-headings in the same section.

Setting :page-toc: false will prevent the section-toc macro from working, so if you want no "top level" TOC, but you do want section TOCs, use :page-toclevels: 0.

Layouts

Layouts define the "look and feel" of each page. Jekyll uses Liquid for page templates.

For our implementation of Jekyll, layouts are found in solr-ref-guide/src/_layouts.

We currently use only the _layouts/default.html layout for overall page structure, and _layouts/page.html for the page-level content. The page.html layout is inserted into the default.html layout.

Includes

Include files are (usually) small files that are pulled into a layout when a page is being built. They are Liquid templates that define an area of the page. This allows flexibility across layouts - all pages can have the same header without duplicating code, but different pages could have different menu options.

Include files that we use define the top navigation, the page header, the page footer, and tables of contents.

For our implementation of Jekyll, include files are found in solr-ref-guide/src/_includes.

Data Files

Data files include data such as lists that should be included in each page. The left-hand navigation menu is an example of a data file. However, in our build, the navigation is built from the page-children hierarchies.

For our implementation of Jekyll, data files are found in solr-ref-guide/src/_data.

Using Bootstrap Components

The HTML files include Bootstrap (v3.3.4 as of October 2017, see _includes/head.html to confirm the Bootstrap version currently being used), so all of the components of Bootstrap are available.

However, before trying to use these components, it’s important to understand how Jekyll and Asciidoctor relate to one another.

How Jekyll Consumes .adoc Files

Jekyll creates all of the page elements we do not define in an .adoc file: the header, footer, navigation elements, comments, and other parts of the page that we don’t worry about as we write the content of a page. It produces a template that our content is inserted into.

Asciidoctor converts our content to HTML - defining its own classes to define default styling - and inserts it into the template Jekyll has provided. While we have a lot of control over the CSS for styling, without creating custom plugins for Asciidoctor, we don’t control the HTML elements produced by the plugin.

This is important because in order to use Bootstrap components, we need to insert div classes into the HTML produced by Asciidoctor.

Asciidoctor Roles

Asciidoctor helpfully provides a way to define custom <div> classes in .adoc files, as long as we understand how to use it.

Asciidoctor does not call these "divs" or "classes", but instead "roles". We can give any content a role - to images, content blocks (such as [source] or [NOTE], etc.), even a word in the middle of a sentence.

Because roles are so flexible, they only apply to the thing - the word, content block, image, etc. - they are directly applied to. This means that if we want an entire section of content to be given a specific role in the HTML (i.e., enclosed in a <div>), then we need to put the content in a block.

Creating Tabbed Sections

Hopefully a little bit of background on roles is helpful to understanding the rest of what we’ll do to create a tabbed section in a page.

See the Bootstrap docs on tabs for details on how to use tabs and pills with Bootstrap. As a quick overview, tabs in Bootstrap are defined like this:

<ul class="nav nav-pills"> (1)
  <li class="active"><a data-toggle="pill" href="#sec1">Section 1</a></li>
  <li><a data-toggle="pill" href="#sect2">Section 2</a></li>
</ul>

<div class="tab-content"> (2)
  <div id="sect1" class="tab-pane active"> (3)
    <h3>Section 1</h3>
    <p>Some content.</p>
  </div>
  <div id="sect2" class="tab-pane">
    <h3>Section 2</h3>
    <p>Some other content.</p>
  </div>
</div>

1	This section creates an unordered list with a line item for each tab. The `data-toggle` and `class` parameters are what tell Bootstrap how to render the content.
2	Note the class defined here: `<div class="tab-content">`. This defines that what follows is the content that will make up the panes of our tabs. We will need to define these in our document.
3	In our document, we need to delineate the separate sections of content that will make up each pane.

We have created some custom JavaScript that will do part of the above for us if we assign the proper roles to the blocks of content that we want to appear in the tab panes. To do this, we can use Asciidoctor’s block delimiters to define the tabbed content, and the content between the tab.

Define an "open block" (an unformatted content block), and give it the role .dynamic-tabs. An open block is defined by two hyphens on a line before the content that goes in the block, and two hyphens on a line after the content to end the block. We give a block a role by adding a period before the role name, like this:
```
[.dynamic-tabs]
--
The stuff we'll put in the tabs will go here.
--
```

Next we need to define the content for the tabs between the open block delimiters.

We enclose each tab pane in another type of block, and "example" block. This allows us to include any kind of content in the block and be sure all of the various types of elements (heading, text, examples, etc.) are included in the pane.
We give the example block another role, tab-pane, and we must make sure that each pane has a unique ID. We assign IDs with a hash mark (\#) followed by the ID value (#sect1).
We also need to define a label for each tab. We do this by adding another role, tab-label to the content we want to appear as the name of the tab.

In the end one pane will look like this:

[example.tab-pane#sect1] (1)
==== (2)
[.tab-label]*Section 1*  (3)
My content...
====

1	When we define the example block with `[example]`, it’s followed by `.tab-pane#sect1` as the class (each class separated by a period `.`) and the ID defined in the tab definition earlier. Those will become the classes (`class="tab-pane active"`) and ID (`id="sect1"`) in the resulting HTML.
2	Example blocks are delimited by 4 equal signs (`====`) before and after the enclosed content.
3	The words "Section 1" will appear in the HTML page as the label for this tab.

Create [example.tab-pane#id] sections for each tab, until you finally end up with something that looks like this:

[.dynamic-tabs]
--
[example.tab-pane#sect1]
====
[.tab-label]*Section 1*
My content...
====

[example.tab-pane#sect2]
====
[.tab-label]*Section 2*
My content...
====
--

Because these tabbed sections are created when the HTML and related JavaScript is loaded, when the PDF is generated it will ignore all the class and ID information in the page because it is meaningless to it (asciidoctor-pdf does not recognize roles/custom classes). In the PDF, this content will be displayed in Example Blocks, which have some formatting rules, but will not be shown as tabs and shouldn’t show any other oddities.

Building the HTML Site

An Ant target build-site will build the full HTML site. This target builds the navigation for the left-hand menu, and converts all .adoc files to .html, including navigation and inter-document links.

Building the HTML has several dependencies that will need to be installed on your local machine. Review the README.adoc file in the solr/solr-ref-guide directory for specific details.

Build Validation

When you run ant build-site to build the HTML, several additional validations occur during that process. See solr-ref-guide/tools/CheckLinksAndAnchors.java for details of what that tool does to validate content.

Making Changes to the PDF Version

The main published artifact of the Solr Reference Guide is a PDF document.

The PDF version of the Ref Guide uses asciidoctor-pdf (indirectly) to convert the .adoc files to PDF.

In order for all of the various files to be converted into a single, large, PDF file, another .adoc file must be created.

About asciidoctor-pdf

Our PDF build process uses the asciidoctor-ant project to integrate Asciidoctor.

The asciidoctor-ant project is really an Ant wrapper around asciidoctorj. This is a Java binding of several Asciidoctor projects that produces a .jar that includes Asciidoctor, Asciidoctor PDF and Asciidoctor EPub.

Since it is a binding for the Ruby implementations of these three projects, they each work the same way they would if we were using the Ruby versions.

Configuring the PDF Theme

Nearly all of the styling and layout of the PDF is controlled by a single YAML file that is called a theme.

See the section Modifying the Theme for information on how to update the theme.

Declaring the Theme to Use

Our theme is refguide-theme.yml, found in solr/solr-ref-guide/pdf/themes.

New themes must be named to include -theme.yml at the end. They must also be in proper YAML format.

The theme to use when generating the PDF is defined with the pdf-style attribute. Only the first part of the file name is used. So, a theme file named refguide-theme.yml is selected by defining pdf-style=refguide.

In the section Asciidoctor PDF Attributes below, we can see how we’ve declared this in our Ant target.

Modifying the Theme

All of the styling capabilities are described in the Asciidoctor PDF Theming Guide.

About the Uber-Document

In order for all of the files in .adoc format to be compiled into a single PDF, we need to give the process one .adoc file (an "uber-document") that includes all of the content we want in the PDF. In other words, there is no command that says "take all the files in this directory and make one PDF".

Asciidoctor has a nice feature called the include directive, which allows you to insert content from another file into the current file (more details in the Asciidoctor documentation).

We wanted to make sure we didn’t add to the burden of maintaining the PDF by asking everyone to update the uber-document or trying to institute some sort of check on if all the pages are included and in the right place in the hierarchy. Instead, the uber-document is build programmatically at build time.

The uber-document is found in solr/solr-ref-guide/src/pdf/SolrRefGuide-all.adoc. This document is very simple - it includes only the Apache license statement and a single include directive to another file:

include::_data/pdf-main-body.adoc[]

The Ant target build-pdf includes a dependency on another target, build-nav-data-files. This target looks at each document and builds the pdf-main-body.adoc file with the proper hierarchy of all pages.

You will not see the pdf-main-body.adoc in solr/solr-ref-guide/src/_data directory. This file exists only once it has been built, and it is placed in the build directory, at solr/solr-ref-guide/build/content_data.

Configuring the build-pdf Ant Target

Since most of the styling and layout is defined by the theme, the options that must be configured in the Ant target relate to where to find referenced files, font directories, image directories, and similar top-level document settings.

There are two types of settings at work in our Ant target. First are the settings which are part of the asciidoctor-ant jar. Second, we are able to declare any Asciidoctor PDF attribute (setting) that we want to apply to the entire PDF.

Our Ant target (build-pdf) uses the following settings:

<asciidoctor:convert
             sourceDirectory="${build.content.dir}/pdf"
             sourceDocumentName="SolrRefGuide-all.adoc"
             baseDir="${build.content.dir}"
             outputDirectory="${build.dir}"
             backend="pdf"
             extensions="adoc"
             sourceHighlighter="coderay"
             embedAssets="true"
             imagesDir="${build.content.dir}"
             doctype="book"
             safemode="unsafe">
  <attribute key="icons" value="font" />
  <attribute key="icon-set" value="fa" />
  <attribute key="pdf-stylesDir" value="./pdf/themes"/>
  <attribute key="pdf-style" value="refguide"/>
  <attribute key="pdf-fontsDir" value="./pdf/fonts"/>
  <attribute key="pagenums" value='' />
  <attribute key="figure-caption!" value='' />
  <attribute key="idprefix" value='' />
  <attribute key="idseparator" value='-' />
  <!-- attributes used in adoc files -->
  <!-- NOTE: If you add any attributes here for use in adoc files, you almost certainly need to also add
      them to the _config.yml.template file for building the jekyll site as well
  -->
  <!-- ... -->
</asciidoctor:convert>

There are a lot of options here. Note that some include the <attribute> tag and some do not. Those that do not are options provided by asciidoctor-ant so don’t need any special syntax. The options with the <attribute> tag are not provided by asciidoctor-ant but are supported by asciidoctor-pdf so will be applied when the target is run. A few of these are custom variables that we have defined ourselves.

Asciidoctor Ant Attributes

sourceDirectory="${build.content.dir}/pdf": Defines where to find the source file to build the PDF. The variable is declared earlier in build.xml.
sourceDocumentName="SolrRefGuide-all.adoc": The source file name, which in our case will be built as described in the section About the Uber-Document.
baseDir="${build.content.dir}": The root path for any included files.
outputDirectory="${build.dir}": Defines where the resulting PDF file should go. In this case, the
backend="pdf": The output format. The asciidoctor-ant jar is also capable of outputting DocBook format, so we must declare we want a PDF.
extensions="adoc": The file extensions to allow for the source document.
sourceHighlighter="coderay": The library to use for syntax highlighting source code.
imagesDir="${build.content.dir}": The directory to use to find images referenced in the documents.
doctype="book": Adds support for book-style format and sections, such as a preface, colophon, glossary, index, etc.
safemode="unsafe">: Allows including resources that are external to the parent directory of the source file. For example, source examples could be pulled from Solr’s source code instead of copied to documentation. This setting allows that to happen.

Asciidoctor PDF Attributes

<attribute key="icons" value="font" />: The style of icons.
<attribute key="icon-set" value="fa" />: The icon set to use. We use the Font Awesome font set.
<attribute key="pdf-stylesDir" value="./pdf/themes"/>: The directory to find PDF themes. See the section Configuring the PDF Theme for more details on themes.
<attribute key="pdf-style" value="refguide"/>: The theme to use. The theme must be saved in the directory referenced with the pdf-stylesDir attribute, and must be named <pdf-style>-theme.yml.
<attribute key="pdf-fontsDir" value="./pdf/fonts"/>: The directory where to find fonts declared in the theme.
<attribute key="figure-caption!" value='' />: Sets caption labels and numbers (such as "Figure 1") to block images. The exclamation at the end of this setting in our config disables figure captioning.
<attribute key="idprefix" value='' />: Sets the prefix for auto-generated section IDs, such as those from headings in a page. In our config, this is effectively "null", so auto-generated section IDs do not have any prefix.
<attribute key="idseparator" value='-' />: Sets the separator between words in auto-generated section IDs to a hyphen (-).

Custom Attributes

These attributes use variables that are inserted by Ant during the PDF creation process. This allows us to pull from standard Lucene/Solr build files, and not have to update several places for any release. The Ant build process updates the _config.yml file from the _config.yml.template, then these attributes pull the proper value from that file.

<attribute key="solr-guide-draft-status" value="$" />: Indicates if this is a DRAFT PDF or not.
<attribute key="solr-guide-version" value="$7.5" />: The version of the Guide itself.
<attribute key="solr-docs-version" value="$7.5" />: The version of Solr covered by this guide.
<attribute key="solr-javadocs" value="$https://lucene.apache.org/solr/7_5_0/" />: Sets the path for Solr javadoc links to include the right path for the current release version.
<attribute key="lucene-javadocs" value="$https://lucene.apache.org/core/7_5_0/" />: Sets the path for Lucene javadoc links to the right path for the current release version.
<attribute key="build-date" value="${DSTAMP}" />: Sets the date of the build to add the date to the footer of each page of the PDF.
<attribute key="build-year" value="${current.year}" />: Sets the year of the build to add the date to the copyright notice.

Ref Guide Publication Process

The PDF version of the Guide is the official release, and requires a vote before release.

The HTML version is considered a "convenience" version, and does not require a vote.

It’s strongly preferred that both PDF and HTML versions are available during the vote for the PDF. However, since the HTML version is not an official release, it is more of an unwritten rule to publish the HTML at the same time as producing a release candidate for the PDF.

These instructions assume you have the proper dependencies for building HTML and are publishing both versions simultaneously. See Additional Pre-Requisites for HTML for details.

Pre-Requisites

The PDF version of the Solr Reference Guide is the official version. As such, it is voted on by the community before release, and is treated as an official artifact of the Lucene/Solr project.

Pre-Requisites for PDF

The build process downloads the required .jar to build the PDF version, so the only pre-requisites are for the publication process:

You have checked out the Lucene/Solr source code on the machine you will be doing the release from. You will need scripts in the dev-tools directory.
You have generated a GPG key. See the Apache documentation on generating a code signing key.
You have Python 3 installed. This is needed to poll the mirrors after release to be sure it’s propagated enough to make the announcement.
You have Subversion installed. This is needed for committing the PDF to dist/dev and dist/releases repos.

Additional Pre-Requisites for HTML

To build the PDF locally, you need several required dependencies, listed in solr-ref-guide/README.adoc. If you do not have these dependencies locally, you cannot build and publish the Ref Guide.

Builds are available via Jenkins for several branches. However, these HTML pages will have the DRAFT status noted in them and will not be suitable for publishing.

Generate the PDF & HTML

No local dependencies are required to build the PDF. The Ant target will download the jars and other items it requires.

The build process generates the PDF, including the page hierarchy, and then runs an optimization script on the PDF to make it smaller.

To build the PDF:

Run ant clean default -Dsolr-guide-version=X.Y, where X.Y is the version you want to publish (i.e., 7.4).
The resulting PDF will be in solr/build/solr-ref-guide. The HTML files will be in solr/build/solr-ref-guide/html-site.

The -Dsolr-guide-version system property is optional. By default the build system uses the version.properties of the current branch and assumes this is a DRAFT build. Including this system property ensures the DRAFT watermark and labels are removed from the HTML files.

Prepare & Upload Release Candidate

Upload PDF to dist/dev

The dist/dev Subversion repository includes a directory for the Solr Ref Guide at lucene/solr/ref-guide which can host the release candidate (RC) during the VOTE stage of the process.

These steps walk through checking out this directory and uploading the Guide to it.

Checkout the directory: svn co https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide solr-ref-guide-rc
- If you have already checked out this directory, you can simply update it: svn update solr-ref-guide-rc
Change directories so solr-ref-guide-rc is your working directory (cd solr-ref-guide-rc).

The next step requires that you have already generated your GPG keys. Your GPG passphrase will be required.

Run the Prep Ref Guide script to prepare the RC. This script ensures proper naming of the PDF file, generates .sha1, .sha512 and .asc files and creates the proper RC sub-directories under solr-ref-guide-rc.

The structure of the input is: prep-solr-ref-guide-rc.sh <path/PDFfilename> <Solrversion-RC#> GPGkey.

From the solr-ref-guide-rc directory, it will look something like this:

$ ~/lucene-source/dev-tools/scripts/prep-solr-ref-guide-rc.sh apache-solr-ref-guide-7.0.pdf 7.0-RC0

+ mkdir apache-solr-ref-guide-7.0-RC0
+ mv apache-solr-ref-guide-7.0.pdf apache-solr-ref-guide-7.0-RC0/apache-solr-ref-guide-7.0.pdf
+ cd apache-solr-ref-guide-7.0-RC0
+ shasum apache-solr-ref-guide-7.0.pdf
+ shasum -a 512 apache-solr-ref-guide-7.0.pdf
+ gpg -u DEADBEEF --armor --output apache-solr-ref-guide-7.0.pdf.asc --detach-sig apache-solr-ref-guide-7.0.pdf

You need a passphrase to unlock the secret key for
user: "Your Name <you@apache.org>"
4096-bit RSA key, ID DEADBEEF, created 1969-07-04

Add and commit the new release candidate to the dist/dev with these steps:
1. svn add apache-solr-ref-guide-7.0-RC0
2. svn commit -m "7.0 ref guide RC0"

Upload HTML Files

Step 1: Update extpaths.txt in CMS Staging

Checkout CMS trunk:
```
svn co --depth=immediates https://svn.apache.org/repos/asf/lucene/cms/trunk/content website-source
```
- If you already have this repo checked out, you can simply svn up website-source to update to the latest revision.
cd website-source
Add Guide branch dir: echo solr/guide/X_Y >> extpaths.txt

Commit changes:

svn commit -m "Update CMS production sync exceptions for X_Y_Z Guide" extpaths.txt

Step 2: Push Guide to Website Production

Go to the checkout directory where you have built the Guide and push the documentation via subversion import. You must push it to the path you just added to extpaths.txt, so if the path you added was solr/guide/6_5, you’ll use the path as shown in the below example:

svn -m "Add Ref Guide for Solr 6.5" import <checkoutroot>/solr/build/solr-ref-guide/html-site https://svn.apache.org/repos/infra/websites/production/lucene/content/solr/guide/6_5

Confirm you can browse to these URLs manually, and especially that Solr javadocs link back to lucene’s correctly. Example: https://lucene.apache.org/solr/guide/6_5

Step 3: Push Staging extpaths.txt to Production

The extpaths.txt works by listing paths that should be ignored when the CMS syncs the staging and production repositories. Publishing staging to production will only succeed if the paths listed in extpaths.txt exist in production. At the same time, if a path exists in production but not in staging it will be deleted unless it is defined in extpaths.txt.

After pushing the content to production, check that the extpaths.txt in production includes the proper path to ensure that the Guide is not deleted incorrectly. If it does not exist in production, try to publish the site again to make sure it is updated.

Production URL: https://lucene.apache.org/extpaths.txt

Hold a VOTE

Votes must be sent to the lucene-dev mailing list (dev@lucene.apache.org).

Send an email to dev@lucene.apache.org with subject, "VOTE: Release Apache Solr Ref Guide for Solr X.Y".
The body of the email should include the full URL of the RC directory in the dist/dev repo. Such as: https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-7.0-RC0
You can add your own +1 to the vote announcement email.
If there are issues that need to be resolved, you can start the process over, using RC1, RC2, etc., as needed.

Ideally, the HTML version will also be available for voters to review.

Publish the Guide

Move the PDF to dist/releases

Once at least three PMC members have voted for release (see Apache Voting Process for details on the rules for votes), the release candidate can be released.

Use the Publish Solr Ref Guide script (publish-solr-ref-guide.sh) to generate the proper SVN commands to be run to execute a remote move of the RC files to the final dist/releases repository.

The script takes only the version and RC number that passed the vote as inputs, such as 7.0-RC0.

The input and output of the script will look like this:

$ ~/lucene-source/dev-tools/scripts/publish-solr-ref-guide-rc.sh X.Y-RCZ

## Run the following commands when ready...
svn move -m 'publishing apache-solr-ref-guide-X.Y-RCZ' https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RCZ/apache-solr-ref-guide-X.Y.pdf https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RCZ/apache-solr-ref-guide-X.Y.pdf.asc https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RCZ/apache-solr-ref-guide-X.Y.pdf.sha1 https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RCZ/apache-solr-ref-guide-X.Y.pdf.sha512 https://dist.apache.org/repos/dist/release/lucene/solr/ref-guide/

svn rm -m 'cleaning up apache-solr-ref-guide-X.Y-RCZ' https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RCZ

The release should propagate to as many mirrors as possible before announcing the release, generally 24 hours is long enough. Use the Poll Mirrors script (poll-mirrors.py) to check the status:
```
python3 -u ~/lucene-source/dev-tools/scripts/poll-mirrors.py -details -p lucene/solr/ref-guide/apache-solr-ref-guide-X.Y.pdf
```
- This script requires Python 3 to be installed on your machine.
- If you have over 85% of the mirrors with the file, it’s OK to go ahead with the announcement.
You may get an automated email about updating the ASF release repository; you can safely ignore this email.

Archive Old PDF Versions

The dist/releases repo is only meant to keep the latest releases. Shortly after new releases are mirrored, they are copied to archive.apache.org, so older releases can safely be deleted from dist/releases since they have been backed up in the archives.

Run the Archive Ref Guide script (archive-solr-ref-guide.sh) using the X.Y version of the Ref Guide that has just been published. Older RCs will also be removed.

Again, this script doesn’t do any direct removal of files, it only outputs SVN commands for you to copy and paste:

$ ~/lucene-source/dev-tools/scripts/archive-solr-ref-guide.sh X.Y
## Run the following commands when ready...

# Delete old releases
svn rm -m 'removing archived ref guide files prior to X.Y' https://dist.apache.org/repos/dist/release/lucene/solr/ref-guide/apache-solr-ref-guide-A.B.pdf https://dist.apache.org/repos/dist/release/lucene/solr/ref-guide/apache-solr-ref-guide-A.B.pdf.asc https://dist.apache.org/repos/dist/release/lucene/solr/ref-guide/apache-solr-ref-guide-A.B.pdf.sha1

# Delete old RC files
svn rm -m 'cleaning up old RCs now that X.Y has been released' https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RC0/ https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y-RC1/

Make New HTML Version the Default

When the PDF is announced as available, the HTML version should already be available on the Solr website. There are a few steps to take to make the new HTML version the default.

You can use the CMS system for these changes, or you can edit the file locally and commit it to the staging repo.

Update the landing page at https://lucene.apache.org/solr/guide (the file is at content/solr/guide/index.mdtext in SVN) to link to the newest version.
Update the Guide redirect rule that looks like the below in content/.htaccess so URLs without a version in the path are redirected to the latest available version.
```
RedirectMatch temp /solr/guide/(?!index.html)([a-z].*) /solr/guide/7_0/$1
```
In the above example, you would change the 7_0 part of the path to the right version (7_1, etc.).

Announce the Release

Announce the availability of the new Ref Guide on solr-user@lucene.apache.org and CC general@lucene.apache.org and announce@apache.org.

You must send the announcement email from your @apache.org email address or announce@apache will reject it.

Always use the link to the download redirector for the announcement, as it will automatically direct users to the closest mirror for download: https://www.apache.org/dyn/closer.cgi/lucene/solr/ref-guide/apache-solr-ref-guide-X.Y.pdf.

You should include a link to the HTML version in your announcement.

Sample announcement

The Lucene PMC is pleased to announce that the Solr Reference Guide
for 7.0 is now available.

This 1,035-page PDF is the definitive guide to using Apache Solr, the
search server built on Lucene.

The PDF Guide can be downloaded from:
https://www.apache.org/dyn/closer.cgi/lucene/solr/ref-guide/apache-solr-ref-guide-7.0.pdf.

It is also available online at https://lucene.apache.org/solr/guide/7_0.

If the Guide is being published more than a day or two after the application itself, you should update the Solr website news page with the announcement (https://lucene.apache.org/solr/news.html).

Updating HTML Files after Publication

If you need to re-publish an existing online copy of the Guide, you will need to checkout the directory in production website repository and overwrite the existing files:

Build the new HTML files locally (ant clean build-site), or download them from Jenkins.
Checkout the directory you need to update from the production repo: svn co https://svn.apache.org/repos/infra/websites/production/lucene/content/solr/guide/<dir>;.
- This command checks out the Guide version directory into a local subdirectory with the same name as the version (such as "6_5"). You can provide a better name locally if you prefer by adding it to the end of the command shown above.
- Don’t shortcut this and download the whole production website. It will take an incredibly long time and that will feel like forever.
Copy the files from the build location to the checked out Guide directory. For example, if we needed to replace the current Guide for Solr 6.5, we’d do cp -r <checkoutroot>/solr/build/html-site 6_5/.
Use svn status to see the files modified.
If there are any pages added or deleted, use svn add <file> or svn rm <file> as needed.
Commit the changes: svn commit -m "Update production 6.5 Ref Guide"

Errata