.. Instructions to contribute to Byteflies Documentation
.. Author: Jonathan Dan, Benjamin Vandendriessche

.. STATUS: production

###############
Getting Started
###############

This document consists of two parts:

1. A guide for *Users* on how to :ref:`contribute <contribute>` to the documentation
2. A guide for *Documentation Managers* on how to :ref:`moderate <moderate>` the documentation

User
    Anybody with read access to the documentation and source code repository. Users can make changes to existing documents and add new documents in their personal documentation *fork* only.

Doc(umentation) Manager
    A moderator assigned to a specific *Tech Team* who needs to sign off on all *User* suggested changes before they can be merged into the production copy of the documentation.

.. attention:: **PLEASE READ**: The information below is useless without adhering to this simple concept: *before* making any changes to your local documentation fork, you **need to make sure it is up-to-date** with the production version. Failing to do so will make the *Doc Managers* job harder (as well as yours), and something bad may (or may not) happen to a puppy. This is the single most important piece of information in this guide for which the step-by-step instructions can be found :ref:`here <upstream_update>`\ [#]_.

.. [#] You should still read the entire `User Guide`_ because of the puppies.

.. rubric:: Quick Navigation
.. contents::
   :local:
   :depth: 2

----

**********
User Guide
**********

The documentation is generated using `Sphinx <http://sphinx-doc.org>`_. Sphinx uses the `reStructuredText <http://docutils.sourceforge.net/rst.html>`_ markup language for writing documents. A brief tutorial can be found :doc:`here <reST_tutorial>`.

The documentation is hosted on `Github <https://github.com/Byteflies/documentation>`__. Only *Doc Managers* have write access to this repository. All other *Users* have read access and should contribute to the documentation by :ref:`submitting pull requests <contribute>`. It is the responsibility of the *Doc Managers* to review, revise, and approve/decline the proposed changes before merging them into the production copy of the documentation.

.. _sphinx_install:

Installing documentation build tools
====================================

Sphinx is a Python program. In order to use it you will first need a working Python 3 installation. To verify, run ``python3 --version`` from a terminal. If you receive an error, do the following (macOS):

1. Install brew (instructions on https://brew.sh/)
2. Install Xcode: ``xcode-select --install``
3. Install Python 3: ``brew install python3``

Next, install the following python packages (available on `PyPi <https://pypi.python.org/pypi/Sphinx>`_): ``pip3 install sphinx sphinx_rtd_theme nbsphinx sphinxcontrib-httpdomain``

Building the documentation is as simple as running ``make html`` from a terminal in the documentation root directory.

.. hint:: It is good practice to occasionally run ``make clean`` because Sphinx only rebuilds files that have been changed. Occasionally, this could lead to some minor issues with the file structure.

.. _contribute:

Contributing to the documentation
=================================

We use the git *fork* and *pull request* model for contributing to the documentation. **git** is a version control system for tracking changes in files and coordinating work on those files among multiple people.

`fork <https://help.github.com/articles/fork-a-repo>`_
  A fork is a copy of a repository. Forking a repository allows you to freely experiment with changes without affecting the original project.

`pull-request <https://help.github.com/articles/about-pull-requests>`_
  Pull requests let you tell others about changes you have pushed to a repository on GitHub. Once a pull request is opened, you can discuss and review the potential changes with collaborators and add follow-up commits before the changes are merged into the production repository.

.. hint:: If you are not familiar with the git *fork* and *pull request* model, this tutorial is a great reference: `How To Create a Pull Request on GitHub  <https://www.digitalocean.com/community/tutorials/how-to-create-a-pull-request-on-github>`_. The instructions below are mostly based on command line instructions. If you feel more comfortable using a UI application, `Github Desktop <https://desktop.github.com/>`_ can perform most (although not all) the tasks explained here.

Setting up your documentation repository
----------------------------------------

Follow these steps to set up your personal copy of the documentation (these steps should ony be performed once):

1. Make a duplicate (*fork*) of the `documentation master <https://github.com/Byteflies/documentation>`__ (i.e. the production version):

.. image:: images/github_fork.gif
   :scale: 40 %
   :align: center
   :alt: `source <https://www.digitalocean.com/community/tutorials/how-to-create-a-pull-request-on-github>`__

2. Copy (*clone*) that duplicate to your local system: ``git clone git@github.com:USERNAME/documentation.git``

.. hint:: This step assumes you already set up an RSA key, and the public key was added to your Github profile. If you haven't yet, follow :ref:`these instructions <rsa_gen>`. Alternatively, use the ``https://github.com/username/documentation.git`` format, and authenticate with your Github username and password. If you have :ref:`2FA <2FA>` activated on your Github account, you will have to generate a `personal access token <https://github.com/settings/tokens>`_ first and use that as your password. To remember your credentials on your personal device, follow `these instructions <https://help.github.com/articles/updating-credentials-from-the-osx-keychain/>`_ (macOS).

3. You are now ready to start making modifications or add documents in your local copy. However, your changes need to be compared (*diff*) to the documentation master. To do so, instruct your local copy to use the documentation master (*remote*) as the `equivalent of an earlier commit <https://help.github.com/articles/configuring-a-remote-for-a-fork>`_, like so: ``git remote add upstream git@github.com:Byteflies/documentation.git``. Note that we refer to the master documentation as *upstream*, this is arbitrary.

.. _pullreq:

Making changes to the documentation
-----------------------------------

.. _upstream_update:

.. attention:: **Before starting any new modifications**, always ensure your forked repository is up to date with the *upstream* (production) repository like so:

    1. **Checkout** to *your* master branch: ``git checkout master``
    2. **Pull** changes from *upstream* (production): ``git fetch upstream``
    3. **Update** your master branch: ``git rebase upstream/master``
    4. **Push** your local master branch to *your* online copy: ``git push``

1. Make a new **branch** for your changes: ``git checkout -b NEW_BRANCH_NAME``
2. Add your changes, files, or generally make the documentation a better place :sup:`TM JD`
3. **Commit** your changes: ``git add .``, followed by ``git commit``
4. **Add** your branch to Github: ``git push --set-upstream origin NEW_BRANCH_NAME``
5. **Push** your changes: ``git push``
6. Create a **pull request** from your new branch on `Github <https://help.github.com/articles/creating-a-pull-request>`_

.. image:: images/pull-request.png
   :scale: 75 %
   :align: center
   :alt: `source <https://www.digitalocean.com/community/tutorials/how-to-create-a-pull-request-on-github>`__

.. attention:: Before submitting a :ref:`pull request <contribute>`, first check if the changes you introduced do not produce build errors. Always build the documentation locally and verify the build log is error free before pushing it online.

7. Assign an appropriate category tag: ``tools``, ``tech_hw``, ``tech_sw``, or ``tech_sp``
8. Assign an appropriate status tag: ``minor update``, ``major update``, ``draft``, or ``production``

In case of issues
-----------------

The following procedure can be used to reset the local copy of your fork to the same state as the production master:

.. danger:: You *will* loose any changes that are newer than the production version. Use with caution!

::

    git remote add upstream git@github.com:Byteflies/documentation.git
    git fetch upstream
    git checkout master
    git reset --hard upstream/master
    git push origin master --force

The first command may not be necessary if *upstream* was already defined.

----

.. _moderate:

***************************
Documentation Manager Guide
***************************

*Doc Managers* review and sign-off on documentation changes submitted by *Users*. They should ensure consistency of the documentation, enforce proper template usage, and include new documents at a relevant place in the documentation structure. Consequently, they need write access both to the Github repository and to the *Local Server* repository (not to be confused with your local git copy).

.. hint:: Use the Github Issue tracker to assign documentation tasks to other users, using the ``request`` tag.

Review process
==============

If this is your first time, make sure you reviewed `Github & Local Server`_ first before proceeding!

Building documentation
----------------------

.. hint:: As a *Doc Manager*, there is no need to use :ref:`pull requests <pullreq>` to make your own changes. Simply branch off the master, make changes, and merge them with the master when ready. However, it does make sense to work on your own :ref:`fork <contribute>` for larger changes to keep things organized.

If you followed the  initial setup :ref:`instructions <gitsetup>` correctly, pushing changes to the *origin* will update both the Github repository and the copy on the *Local Server*. The latter will trigger the build process.

.. attention:: When working remotely, a VPN connection will be needed to update the *Local Server*. If not available, simply push again when connected to the local network.

When reviewing :ref:`pull requests <pullreq>` made by other *Users*, you have multiple options available:

1. Use the Github commenting system to ask the submitter for additional information or changes
2. Make changes yourself, or simply copy new files to your working branch
3. If there are potential conflicts with production documentation: ``merge & commit`` the :ref:`pull request <pullreq>` to approve it and add it as a new commit to the *origin*
4. If the changes can be accepted as is: ``rebase & commit`` the :ref:`pull request <pullreq>` to approve it and merge it with the *origin* without generating a separate commit

Document consistency guidelines
===============================

The information in this section is specific to *Doc Managers*. A summary of relevant reStructuredText and Byteflies Documentation-specific syntax is summarized :doc:`elsewhere <reST_tutorial>`. *Doc Managers* should familiarize themselves with the syntax and verify its correct usage.

Document organization
---------------------

.. figure:: images/file-structure.png
   :scale: 50 %
   :align: left

   Documentation file structure.

* The root contains the main ``index.rst`` master file, as well as make, configuration, README, and LICENSE files; no other files should be placed in the root
* ``_build`` contains the built files, e.g. after running ``make html``
* ``_static`` is used for custom CSS specifications
* ``_templates`` is used to override theme settings
* All other directories are specific to the Byteflies Documentation structure (*see below*): ``tech_xx`` are the respective team root directories with ``manuals``, ``reference``, and ``tutorials`` subdirectories
* Every category directory can contain a subdirectory ``images`` which is primarily used to store images as well as other linked files (if it was not generated yet, add one to your directory structure when needed)

.. Text block spacers to move next block underneath image

|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|

.. attention:: All directories need to have a *master file* that allows Sphinx to read the file hierarchy. For instance, ``tech_hw`` contains ``tech_hw.rst``, and ``tech_hw/manuals`` contains ``manuals_hw.rst``. Do *not* change these files as it can lead to build errors that orphan deeper files. If you do need to test a file that is not yet connected to the file structure, add the ``:orphan:`` tag above the main title.

.. include:: ../snippets/doc_categories.rst

.. hint:: If a document contains information that is relevant to more than one team, choose the most relevant one and add appropriate links and references to other teams.

Document header
---------------

Every file should include the following header lines (preceded by ``..`` to mark it as a comment):

1. Brief (< 80 char) description of document purpose: use capitalization to refer to documentation-specific nomenclature
2. ``Author: First Last``; can be more than one, separated by commas OR ``Documentation Manager: First Last``; this is reserved for (sub)root files that should rarely  -if ever- be updated
3. ``Version: x.y [Month Year]``; (**Optional: only for documents for which multiple versions may have to be maintained**)
4. Blank line
5. ``STATUS: label``; label can be ``draft``, ``production``, or ``snippet``\ [#]_

For example: ::

    .. Instructions about the most amazing Byteflies device
    .. Author: Byte Fly
    .. Version: 1.0 [January 2017]

    .. STATUS: draft

.. [#] The production and snippet labels are final, draft documents should graduate to production or snippet eventually. If a document is marked as a draft, add a ".. note:: This is a DRAFT" directive.

Additionally, the header can contain custom CSS specifications, e.g. to allow *Users* to colorize text red, add ``.. role:: red`` to the start of the file (just before the main title, separated by a blank line). In the document, words can be highlighted in the specified color using ``:red:`text```. Refer to the ``_static/css`` directory to see available options, or add new ones.

.. note:: Snippets are short blurbs that need to be repeated in multiple locations. To avoid having to maintain the same information in more than one place, split off the text in a separate file in the ``snippets`` directory and import it into another file using the ``.. include:: ../snippets/snippet.rst`` directive.

Document sections
-----------------

By convention, a maximum of 5 section levels are allowed. Each document should only have one level 1 section (main title). Level 1 and level 2 section titles should use title case. Lower level headings should only have the first word capitalized. To generate a heading that is separate from the main section levels, use the ``.. rubric::`` directive (see also :doc:`here <reST_tutorial>`).

.. _gitsetup:

Github & Local Server
=====================

.. attention:: The following steps should only be performed once.

The documentation repository on Github is duplicated on the *Local Server*, and built into HTML there. *Doc Managers* should:

* Request to be added to the ``Documentation Managers`` team on Github
* Set up an RSA key and send their public key to a server admin

.. note::

  **Generating an RSA key**

  .. include:: ../snippets/rsa_gen.rst

One additional step is needed to configure the *remotes* so that a *push* command updates both Github and the *Local Server*:

1. ``git clone git@github.com:Byteflies/documentation.git``
2. ``git remote set-url --add --push origin git@github.com:Byteflies/documentation.git``
3. ``git remote set-url --add --push origin git@flyswatter.local:documentation``

You can check everything is set up correctly by checking the output of ``git remote -v``, which should look as follows:

.. code:: bash

  origin	git@github.com:Byteflies/documentation.git (fetch)
  origin	git@flyswatter.local:documentation (push)
  origin	git@github.com:Byteflies/documentation.git (push)

Advanced Sphinx setup
---------------------

The following assumes ``sphinx_rtd_theme`` is installed.

1. To configure Sphinx for first-time use, do ``sphinx-quickstart`` and follow the instructions. Make sure ``autodoc``, ``doctest``, ``intersphinx``, ``todo``, ``mathjax``, ``viewcode``, and ``githubpages`` are enabled

2. To build HTML or LaTeX (assuming a Makefile was properly configured in the previous step), do ``make html`` or ``make latex``, respectively

3. To generate a PDF from LaTex, the following packages are needed (only tested on Ubuntu 14.04):

    ``sudo apt install texlive-latex-base texlive-latex-extra texlive-fonts-recommended``

4. To render a PDF from LaTeX, do ``latexpdf file.tex``

5. To change the default theme to the "Read the Docs" theme, replace ``html_theme = alabaster`` in ``conf.py`` with:

.. code-block:: python

   import sphinx_rtd_theme

   html_theme = "sphinx_rtd_theme"
   html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]

Jupyter Notebook integration
----------------------------

Sphinx can integrate `Jupyter notebooks <http://jupyter.org/>`_ in the documentation as a static printout or a file attachment. These instructions assume ``nbsphinx`` is installed.

1. To activate the extension, add the following in ``conf.py``:

.. code-block:: python

   extensions = [
    'nbsphinx',
    <other extensions>
   ]
   exclude_patterns = [<other exclusion patterns>, '**.ipynb_checkpoints']

2. To convert a Jupyter notebook to a reST file, do ``jupyter nbconvert *.ipynb --to rst`` (where * is the filename)

3. To insert a link to a Jupyter notebook, do ``:download:`ipynb </path/*.ipynb>``` (where * is the filename)