========================================
Building, collecting, and serving assets
========================================
MDN requires static assets (images, JavaScript, CSS, and other files) to
augment the HTML generated by Django. Kuma uses a combination of
technologies and techniques to build and deliver static assets. Many of the
build and development processes are documented on the
:doc:`Installation </installation>` and :doc:`Development </development>`
documents. This document goes into the details.

The three phases of an asset's life are **Building**, **Collecting**, and
**Serving**.

* **Building** - Source files are processed to build intermediate and final assets
    - :ref:`Locale files <build-locale-files>`  contain translatable strings
      extracted from source files, are translated by humans, and are compiled
      to a binary representation for runtime use.
    - :ref:`Localization JavaScript files <build-locale-js>` contain
      translated strings in an executable format to allow in-browser translation
      of UI strings.
    - :ref:`CKEditor <build-ckeditor>` is packaged with custom plugins for
      MDN's use cases.
    - Many :ref:`JavaScript libraries <include-js-libraries>` that are used on
      MDN are included in Kuma's repository for version control.
    - :ref:`JavaScript bundles <pipeline-js-bundles>`, assembled by
      ``django-pipeline``, combine several source files into a
      single minified JavaScript file.
    - :ref:`CSS bundles <pipeline-css-bundles>`, assembled by
      ``django-pipeline`` or ``process:sass``, combine several source Sass files into a
      single minified CSS file.
* **Collecting** - Built assets are collected to the ``/static`` folder.
    - Django's :ref:`staticfiles <django-staticfiles>` provides a framework
      of :ref:`Finder <staticfiles-finder>` and
      :ref:`Storage <staticfiles-storage>` classes for collecting files.
    - django-pipeline_ augments ``staticfiles``, to allow compiling, bundling,
      and minifiying JS and CSS assets in the collection phase.
* **Serving** - Collected assets are served to visitors in development and production
    - In `Django and Jinja templates`_, the :ref:`static tag <static-tag>` returns
      the URL of assets collected by ``staticfiles``.
    - The :ref:`statici18n tag <statici18n-tag>` returns the URL of localization
      JavaScript collected by ``staticfiles``.
    - The :ref:`pipeline tags <pipeline-tags>` ``javascript`` and
      ``stylesheet`` return the HTML markup for assets processed and collected
      by ``django-pipeline``.
    - :ref:`WhiteNoise <serve-whitenoise>` serves static assets as part of the
      Django process.


.. _`Django and Jinja templates`: http://jinja.pocoo.org/docs/2.10/templates/#

.. _build-locale-files:

Extracting and building locale files
====================================
Kuma uses Pontoon_ to translate strings in the user interface, in error
messages, and in emails. These are stored in the mdn-l10n_ repository,
and included as a `git submodule`_ at ``locale/``.  See the
:doc:`localization document </localization>` for more details about locales.

Puente_ extracts strings to the Portable Object Template (``.pot``) files,
as specified in the `PUENTE configuration`_. The
file ``locale/templates/LC_MESSAGES/django.pot`` contains strings from template
files and Python code. The file ``javascript.pot`` contains strings from
JavaScript files. Puente looks for the string parameters of ``gettext``
functions, such as ``gettext()``, the common alias ``_()``, and ``ngettext()``.
It also parses longer strings in the template tag ``trans``.

Next the changes are merged into the existing Portable Object (``.po``) files,
such as ``locale/fr/LC_MESSAGES/django.po``, to add new strings and comment out
removed strings.

Extracting and merging is done with ``make localeextract``, usually
during :doc:`deployment </deploy>`, when UI strings change.  This uses the
`extract management command` provided by Puente, which uses Babel_ to extract
strings and update the catalog. A maintainer pushes the updated catalogs as a
new commit to the mdn-l10n_ repository.

Pontoon detects that the repository has changed, and notifies localization
teams that there are new strings. In about 48 hours, the most active teams will
translate strings into the top 10 MDN languages. These are applied by updating
the ``locale`` submodule during the :doc:`deployment process </deploy>`.

At run time, Machine Object (``.mo``) files, such as
``locale/fr/LC_MESSAGES/django.mo``, are used by gettext functions, like
``gettext()`` and ``_()``, to display the localized strings.
These are built with ``make localecompile`` when creating the production
images or when a developer wants to see updated translations.

.. _Pontoon: https://pontoon.mozilla.org/projects/mdn/
.. _mdn-l10n: https://github.com/mozilla-l10n/mdn-l10n
.. _`git submodule`: https://github.blog/2016-02-01-working-with-submodules/
.. _gettext: https://en.wikipedia.org/wiki/Gettext
.. _Puente: https://puente.readthedocs.io/en/latest/
.. _`PUENTE configuration`: https://github.com/mdn/kuma/blob/master/kuma/settings/common.py#L645-L669
.. _`extract management command`: https://github.com/mozilla/puente/blob/master/puente/management/commands/extract.py
.. _Babel: http://babel.pocoo.org/en/latest/messages.html

.. _build-locale-js:

Building localization JavaScript
================================
Django includes a `JavaScriptCatalog view`_ that provides JavaScript
implementations of gettext_ functions, as well as translations for each
locale. It is ineffecient to use this view directly, since it is generated
on access. For efficiency, django-statici18n_ generates files for each locale
from the ``JavaScriptCatalog`` output, so they can be served as static assets.

The translation catalog files are created with ``make compilejsi18n``
from the locale Machine Object ``.mo`` files.  This calls the
`compilejsi18n management command`_ provided by `django-statici18n`.
Kuma sets ``STATICI18N_ROOT`` to ``build/locale``, and the output files have
names like ``build/locale/jsi18n/de/javascript.js``.

.. _`JavaScriptCatalog view`: https://docs.djangoproject.com/en/1.11/topics/i18n/translation/#module-django.views.i18n
.. _django-statici18n: https://django-statici18n.readthedocs.io/en/latest/
.. _`compilejsi18n management command`: https://github.com/zyegfryed/django-statici18n/blob/master/src/statici18n/management/commands/compilejsi18n.py

.. _build-ckeditor:

Building CKEditor
=================
CKEditor is a complex JavaScript application that provides a WYSIWYG editor
for MDN wiki pages. It is packaged with plugins, some from third parties,
and some custom to MDN.

The CKEditor build process is documented on the
:doc:`CKEditor document</ckeditor>`. The built files are checked into the
Kuma repository.

.. _include-js-libraries:

Including JS libraries
======================
Third-party JavaScript libraries are included in the Kuma repository, to
avoid ambiguity about what versions of libraries are used.
Some libraries were added manually, and others with Bower_. See
:ref:`front-end-asset-dependencies` for more details about these
libraries.

Some of these libraries are served directly to visitors, while others are
included in pipleline JavaScript bundles.

.. _Bower: http://bower.io

.. _pipeline-js-bundles:

Building pipeline JavaScript bundles
====================================
Pipeline JavaScript bundles combine several JavaScript files into a single
file, with optional minimization. For example, the file
`static/build/js/main.js`_ is the combination of 10 JavaScript files:

* `kuma/static/js/libs/jquery/jquery.js`_ (JQuery 2.2.0)
* `kuma/static/js/libs/icons.js`_
* `kuma/static/js/components.js`_
* `kuma/static/js/analytics.js`_
* `kuma/static/js/main.js`_
* `kuma/static/js/components/nav-main-search.js`_
* `kuma/static/js/auth.js`_
* `kuma/static/js/highlight.js`_
* `kuma/static/js/wiki-compat-trigger.js`_
* `kuma/static/js/lang-switcher.js`_

The JS bundles are specified in PIPELINE_JS_ in the Django settings.
The bundles are served differently in "development" and "production" modes.
This is roughly controlled by the Django setting ``DEBUG``, which sets further
parameters like ``PIPELINE[PIPEINE_ENABLED]``, and the environment
setting ``DJANGO_SETTINGS_MODULE``, which switches the Django settings
file. See django-pipeline_ as well as the :ref:`pipeline tags <pipeline-tags>`
section for details.

In development, the source files (10 for ``main.js``) are served, so there are
10 ``<script>`` elements in the HTML when ``{{javascript('main')}}`` is
used in a template.  In production, the output bundle is used, so a single
``<script>`` tag appears in the HTML. The single bundle is also processed
with UglifyJS_, which removes whitespace, replaces variable names with
shorter names, and performs other transformations to make the file smaller.

.. _`static/build/js/main.js`: https://developer.mozilla.org/static/build/js/main.js
.. _`kuma/static/js/libs/jquery/jquery.js`: https://github.com/mdn/kuma/blob/master/kuma/static/js/libs/jquery/jquery.js
.. _`kuma/static/js/libs/icons.js`: https://github.com/mdn/kuma/blob/master/kuma/static/js/libs/icons.js
.. _`kuma/static/js/components.js`: https://github.com/mdn/kuma/blob/master/kuma/static/js/components.js
.. _`kuma/static/js/analytics.js`: https://github.com/mdn/kuma/blob/master/kuma/static/js/analytics.js
.. _`kuma/static/js/main.js`: https://github.com/mdn/kuma/blob/master/kuma/static/js/main.js
.. _`kuma/static/js/components/nav-main-search.js`: https://github.com/mdn/kuma/blob/master/kuma/static/js/components/nav-main-search.js
.. _`kuma/static/js/auth.js`: https://github.com/mdn/kuma/blob/master/kuma/static/js/auth.js
.. _`kuma/static/js/highlight.js`: https://github.com/mdn/kuma/blob/master/kuma/static/js/highlight.js
.. _`kuma/static/js/wiki-compat-trigger.js`: https://github.com/mdn/kuma/blob/master/kuma/static/js/wiki-compat-trigger.js
.. _`kuma/static/js/lang-switcher.js`: https://github.com/mdn/kuma/blob/master/kuma/static/js/lang-switcher.js
.. _PIPELINE_JS: https://github.com/mdn/kuma/blob/master/kuma/settings/common.py#L909-L923
.. _UglifyJS: https://github.com/mishoo/UglifyJS2/tree/v2.x

.. _pipeline-css-bundles:

Building pipeline CSS bundles
=============================
Pipeline CSS bundles are conceptually similar to
:ref:`Pipeline JS Bundles <pipeline-js-bundles>`. Some contain multiple
source files, such as `static/build/styles/dashboards.css`_,
which combines:

* `kuma/static/styles/dashboards.scss`_
* `kuma/static/styles/diff.scss`_

Source styles are written in Sass_, and compiled to CSS with node-sass_. These
must be compiled to CSS in both development and production modes. Backend
developers tend to use ``make build-static`` to build and collect these files,
and front-end developers tend to use ``nom run process:sass`` to directly compile them.

The CSS bundles are specified in PIPELINE_CSS_ in the Django settings.
The bundles are served differently in "development" and "production" modes.
This is roughly controlled by the Django setting ``DEBUG``, which sets further
parameters like ``PIPELINE[PIPEINE_ENABLED]``, and the environment
setting ``DJANGO_SETTINGS_MODULE``, which switches the Django settings
file. See django-pipeline_ as well as the :ref:`pipeline tags <pipeline-tags>`
section for details.

In development, the source files (2 for ``dashboards.css``) are used, so there are
2 ``<link>`` elements in the HTML when when ``{{stylesheet('dashboards')}}`` is
used in a template.  In production, the output bundle is used, so a single
``<link>`` tag appears in the HTML. When bundled, CSS is also processed by
clean-css_, which transforms the CSS to make the output files smaller.

.. _`static/build/styles/dashboards.css`: https://developer.mozilla.org/static/build/styles/dashboards.css
.. _`kuma/static/styles/dashboards.scss`:  https://github.com/mdn/kuma/blob/master/kuma/static/styles/dashboards.scss
.. _`kuma/static/styles/diff.scss`: https://github.com/mdn/kuma/blob/master/kuma/static/diff.scss
.. _PIPELINE_CSS: https://github.com/mdn/kuma/blob/master/kuma/settings/common.py#L787-L793
.. _clean-css: https://github.com/jakubpawlowicz/clean-css

.. _django-staticfiles:

Collecting asset files with staticfiles
=======================================
Django provides the django.contrib.staticfiles_ app, widely used in Django
projects to standardize where assets are stored, to collect them for
development and production, and to use different asset URLs in different
environments.

In development mode, the ``staticfiles`` app helps identify assets spread
across the project, and often allows a rapid development cycle (for example,
change a file, refresh the browser, and see the effects of the changed file).
For production, the ``staticfiles`` app provides the management command
``collectstatic``, which gathers files to the ``/static`` folder for efficent
file serving.

The Django documents for ``staticfiles`` are mostly focused on usage.
Additional details are needed to understand how ``django-pipeline``
customizes ``staticfiles``.

Configuration
*************

The ``staticfiles`` app is configured by Django settings:

``STATIC_ROOT``
   The folder on the file system where assets are collected. For MDN, this is
   the ``static`` folder in the ``kuma`` directory.
``STATIC_URL``
   The base URL for static assets. In development, this is
   ``http://localhost:8000/static/``, and in production it is
   ``https://developer.mozilla.org/static/``.
``STATICFILES_FINDERS``
   The dotted path to classes implementing ``staticfiles`` :ref:`Finder <staticfiles-finder>`.
   These determine what files will be collected and served. Kuma uses four finders:

   * `django.contrib.staticfiles.finders.FileSystemFinder`_: Finds files
     in folders specified by ``STATICFILES_DIRS``
   * `django.contrib.staticfiles.finders.AppDirectoriesFinder`_: Finds
     files in the ``static`` subfolder of any installed apps
   * `pipeline.finders.CachedFileFinder`_: Strips hashes from filenames to
     identify the "pre-cached" names for files.
   * `pipeline.finders.PipelineFinder`_: When combined assets are not enabled
     (``PIPELINE['PIPELINE_ENABLED'] == False``), returns the source
     files instead of the combined bundle file.

``STATICFILES_DIRS``
   A list of folders in the ``kuma`` directory that the ``FileSystemFinder``
   will scan for static assets. For MDN, this includes:

   * ``assets/static``
   * ``assets/ckeditor4/build`` (to ``/static/js/libs/ckeditor4/build``)
   * ``kuma/static``
   * ``kuma/javascript/dist``
   * ``build/locale``
   * ``jinja2/includes/icons``

   For example, the :ref:`localization JavaScript <build-locale-js>`
   ``build/locale/jsi18n/fr/javascript.js`` will be collected to
   ``static/jsi18n/fr/javascript.js``.
``STATICFILES_STORAGE``
   The dotted path to a class implementing ``staticfiles``
   :ref:`Storage <staticfiles-storage>`.  Storage determines where files
   are stored, what URLs they have, and provides hooks for modifying files when
   copying them.  Kuma uses three different storages, depending on the context:

   * Development server (``DEBUG=True``): pipeline.storage.NonPackagingPipelineStorage_,
     which avoids combining files when collecting them.
   * Production server (``DEBUG=False``): kuma.core.pipeline.storage.ManifestPipelineStorage_,
     which combines packaged files, hashes the names, and creates a manifest.
   * Testing (``pytest``, TravisCI, etc.) and ``make`` commands: pipeline.storage.PipelineStorage_,
     which combines packaged files but does not hash the names.

.. _django.contrib.staticfiles: cs.djangoproject.com/en/1.11/ref/contrib/staticfiles/

.. _staticfiles-finder:

Finder classes
**************
The ``staticfiles`` app uses Finders to locate asset files. Django considers
this a private API, so it may change in the future. There are two methods the
``BaseFinder`` class expects to be implemented:

* ``find(path)``: Given a short path like ``css/wiki.css``, return
  the absolute path to the file. This is used by the ``findstatic`` management
  command, and to find files when serving assets in development mode.
* ``list(ignore_patterns)``: Return a list of the files this Finder can
  find, along with a :ref:`storage instance <staticfiles-storage>` for each.
  The ``collectstatic`` management command uses this to gather files.

.. _django.contrib.staticfiles.finders.FileSystemFinder:
.. _FileSystemFinder:
.. _django.contrib.staticfiles.finders.AppDirectoriesFinder:

The ``staticfiles`` app provides two finders used by Kuma:

 * The ``FileSystemFinder`` collects files under the folders specified in the
   ``STATICFILES_DIRS`` setting.
 * The ``AppDirectoriesFinder`` collects files in the (optional) ``static``
   subfolder of any installed app listed in ``INSTALLED_APPS``. This is how
   Django applications, including ones bundled with Django, distribute
   JavaScript, CSS, images, and other assets. It isn't used for Kuma's apps.
   Instead, we've standardized on ``kuma/static`` and other named paths.

The Finders are used by WhiteNoise_ to determine which file to serve in
development mode. The management command ``findstatic`` can be used to
determine which file is served, such as::

    $ ./manage.py findstatic -v2 js/main.js

    Found 'js/main.js' here:
      /app/kuma/static/js/main.js
      /app/static/js/main.js
    Looking in the following locations:
      /app/kuma/static
      /app/build/locale
      /app/jinja2/includes/icons
      /usr/local/lib/python2.7/site-packages/flat/static
      /usr/local/lib/python2.7/site-packages/django/contrib/admin/static
      /usr/local/lib/python2.7/site-packages/constance/static
      /usr/local/lib/python2.7/site-packages/djcelery/static
      /usr/local/lib/python2.7/site-packages/django_extensions/static
      /usr/local/lib/python2.7/site-packages/rest_framework/static
      /usr/local/lib/python2.7/site-packages/debug_toolbar/static
      /app/static

When multiple files are found, the first is used. In the above example,
``/app/kuma/static/js/main.js`` will be served in development for
``/static/js/main.js``.

.. _staticfiles-storage:

Storage classes
***************

The ``staticfiles`` app uses a ``Storage`` class, which extends
`Django's Storage class`_ for asset workflows.  Django documents
`how to write a custom storage system`_, and there are many
`3rd-party storage packages`_ for using various cloud providers for file
hosting. The configured ``STATICFILES_STORAGE`` class is used when collecting
files with ``./manage.py collectstatic``.

Django's standard ``Storage`` classes provide methods like ``delete()``,
``exists()``, and ``size()`` for implementing file methods, and methods like
``listdir()`` for getting lists of files. There is a wide variety of storage
backends with different capabilities, and Django allows most methods to raise
``NotImplementedErrror`` if an operation is not supported or is too expensive.

A ``staticfiles`` ``Storage`` class extends the standard ``Storage`` classes and
requires a few more methods, although the exact methods are undocumented. Some
are ``path(name)``, to turn a relative path to a full path, and ``url(path)``,
to get the external URL of the file.  An optional method, ``post_process()``, can
be defined to further process the files, and returns a map of the old paths to
the new paths.

The default storage, StaticFilesStorage_, is based on the standard
FileSystemStorage_, and copies static files to ``STATIC_ROOT`` (the ``static``
folder). For the ``url()`` method, it prepends the ``STATIC_URL`` to the path.

ManifestStaticFilesStorage_ implements the ``post_process()`` method to add the
MD5_ hash of the file's contents to the filename. This allows these files to be
served with very long cache times, since changes will also change the filename.
It also requires manipulating the contents so that references to assets within
other files, such as a CSS `@import statement`_, are updated to the hashed
names. This often requires source files use relative paths like
``../img/logo.svg``, so that the tool can find the destination file.

Because of the intense file processing, ``ManifestStaticFilesStorage`` doesn't
support the live updates of development mode. It requires ``DEBUG=False``, and
that ``./manage.py collectstatic`` is run before running the server, or before
a server restart. A map of original to hashed names is stored in
`staticfiles.json`_, and is read at server startup to determine the hashed
names.

CachedStaticFilesStorage_ is similar to ``ManifestStaticFilesStorage``, but
stores the filename mapping in the cache. It is slower than
``staticfiles.json``, and is used when write access to the filesystem is
forbidden.

.. _`Django's Storage class`: https://docs.djangoproject.com/en/1.11/ref/files/storage/#the-storage-class
.. _`how to write a custom storage system`: https://docs.djangoproject.com/en/1.11/howto/custom-file-storage/
.. _`3rd-party storage packages`: https://djangopackages.org/grids/g/storage-backends/
.. _StaticFilesStorage: https://docs.djangoproject.com/en/1.11/ref/contrib/staticfiles/#staticfilesstorage
.. _FileSystemStorage: https://docs.djangoproject.com/en/1.11/ref/files/storage/#the-filesystemstorage-class
.. _ManifestStaticFilesStorage: https://docs.djangoproject.com/en/1.11/ref/contrib/staticfiles/#manifeststaticfilesstorage
.. _MD5: https://en.wikipedia.org/wiki/MD5
.. _`@import statement`: https://developer.mozilla.org/en-US/docs/Web/CSS/@import
.. _`staticfiles.json`: https://developer.mozilla.org/static/staticfiles.json
.. _CachedStaticFilesStorage: https://docs.djangoproject.com/en/1.11/ref/contrib/staticfiles/#cachedstaticfilesstorage


.. _django-pipeline:

django-pipeline
===============

The `django-pipeline library`_ is used for packing assets.  It provides CSS and
JavaScript concatenation and compression, built-in JavaScript template support,
and optional data-URI image and font embedding. It does this by extending and
overriding the django-staticfiles_ app, so that assets are processed with the
standard ``./manage.py collectstatic`` command.

Kuma uses ``django-pipeline`` to:

* Compile Sass_ .sccs files plain CSS with node-sass_
* Combine multiple JS and CSS files into a single file ("bundle") in production
* Compress CSS files with cleancss_
* Compress JS files with UglifyJS_

Configuration
*************

The ``django-pipeline`` app is configured with the dictionary ``PIPELINE``.
There are many `configuration items`_, some of which are:

* ``PIPELINE_ENABLED``: ``True`` to concatenate and compress assets
  (testing and production), and ``False`` to skip concatenation and
  compression.
* ``PIPELINE_COLLECTOR_ENABLED``: ``True`` to collect assets (testing and
  production), and ``False`` to skip collection and leave them in the
  source locations.
* ``COMPILERS``: A list of CSS compilers. ``pipeline``'s ``SASSCompiler`` in
  testing and production, and ``kuma.core.pipeline.sass.DebugSassCompiler``
  (which does nothing, but instead defers to ``node-sass``) in development.

The ``Makefile`` specifies the testing configuration, so commands like
``make collectstatic`` run with ``PIPELINE_ENABLED`` and
``PIPELINE_COLLECTOR_ENABLED``. However, they are disabled when running the
development server.

``django-pipeline`` specifies outputs as a "package", which specifies one or
more inputs, one output, and some optional settings and overrides.
``PIPELINE['JAVASCRIPT']`` specifies the JavaScript packages, and
``PIPELINE['STYLESHEETS']`` specifies the Sass_/CSS packages.

Finders
*******

Kuma uses two :ref:`Finders <staticfiles-finder>` from ``django-pipeline``.

.. _pipeline.finders.CachedFileFinder:

``CachedFileFinder`` strips hashes from filenames to identify the
"pre-cached" names for files, by removing the middle element of filenames
with three dots. This may have been useful in django-pipeline 1.3 or earlier,
but it appears to do nothing now, or could potentially do the wrong thing
such as resolving ``bootstrap.min.js`` as ``bootstrap.js``.

.. _pipeline.finders.PipelineFinder:

``PipelineFinder`` does nothing if ``PIPELINE['PIPELINE_ENABLED']`` if
``True`` (testing and production), and uses the Storage to find files if it
is disabled. For Kuma, this means it may find files in the ``STATIC_ROOT``
directory. However, since the FileSystemFinder_ finds most files in
``kuma/static`` first, it is doubtful if this Finder ever applies.

Storage
*******

Most of the functionality of ``django-pipeline`` is implemented as a
:ref:`Storage class <staticfiles-storage>`, and Kuma uses three different
implementations depending on the environment.

.. _pipeline.storage.PipelineStorage:

The simplest storage, used during testing and in the ``Makefile``, is
``pipeline.storage.PipelineStorage``, which extends
the :ref:`staticfiles Storage class <staticfiles-storage>`
``StaticFilesStorage``, with a ``post_process`` step that packages JS and CSS
into one-file bundles, according to the ``PIPELINE`` configuration.

.. _pipeline.storage.NonPackagingPipelineStorage:

Development uses ``pipeline.storage.NonPackagingPipelineStorage``.
This works the same way as ``PipelineStorage``, but avoids creating packages,
where several files are combined into one. JavaScript files are
served from the source folders, but CSS files need to be compiled from Sass_,
and are served from the ``/static`` folder after collection. When developing
style files, a developer needs to run ``./manage.py collectstatic`` to see changes.

.. _kuma.core.pipeline.storage.ManifestPipelineStorage:

In production, ``kuma.core.pipeline.storage.ManifestPipelineStorage`` is used.
This combines the package processing of ``PipelineStorage`` with the hashed
assets and ``staticfiles.json`` of ``ManifestStaticFilesStorage``. These are
generated when the production Docker containers are created.

.. _`django-pipeline library`: https://django-pipeline.readthedocs.io/en/latest/
.. _node-sass: https://github.com/sass/node-sass
.. _cleancss: https://github.com/jakubpawlowicz/clean-css-cli
.. _`configuration items`: https://django-pipeline.readthedocs.io/en/latest/configuration.html
.. _SASS: https://sass-lang.com/

.. _PostCSS: https://postcss.org

.. _static-tag:

Template tag static
===================
Django provides a template tag static_ that outputs the URL of the static
asset for HTML. Without ``staticfiles`` installed, it just adds ``STATIC_URL``
to the start of the path. With ``staticfiles``, it calls the ``url(path)``
method of the :ref:`Storage class <staticfiles-storage>`. In production, with
``ManifestStaticFilesStorage``, it uses ``staticfiles.json`` to return a
URLs with hashes in the name.

For example, here is the HTML that includes the Tumbeast_ in `the 404 page`_::

    <div id="beastainer">
      <img id="beast404le" src="{{ static('img/beast-404_LE.png') }}" alt="">
      <img id="beast404re" src="{{ static('img/beast-404_RE.png') }}" alt="">
      <img class="beast 404" src="{{ static('img/beast-404.png') }}" alt="">
    </div>

.. _static: https://docs.djangoproject.com/en/1.11/ref/templates/builtins/#std:templatetag-static
.. _Tumbeast: https://theoatmeal.com/static/state_web_summer.html#tumblr
.. _`the 404 page`: https://github.com/mdn/kuma/blob/master/jinja2/404.html

.. _statici18n-tag:

Template tag statici18n
=======================
The tag ``statici18n`` is provided by django-statici18n_. It works like the
``static`` tag, outputing the URL of the
:ref:`localization JavaScript <build-locale-js>`. This is included in
``<body>`` of all page via `the base template`_, near the bottom::

   <script src="{{ statici18n(request.LANGUAGE_CODE) }}"></script>

.. _`the base template`: https://github.com/mdn/kuma/blob/master/jinja2/base.html

.. _pipeline-tags:

Template tags javascript and stylesheet
=======================================
:ref:`django-pipeline <django-pipeline>` provides two template tags,
``{% javascript('bundle') %}`` and ``{% stylesheet('bundle') %}``, that
can inject the ``<script>`` and ``<link>`` elements into a template.

Bundling is controlled by the setting ``PIPELINE['PIPELINE_ENABLED']``
(``False`` for development, ``True`` for production). When bundled, the assets
are assumed to be processed and collected, so a single element representing
the final asset URL is inserted. When bundling is off, the assets are assumed
to still be in the source form, and multiple HTML elements are inserted into
the document. These tags look more like Jinja2 calls then HTML, like these
tags from `the revision dashboard`_::

    {% block js %}
    {% javascript 'jquery-ui' %}
    {% javascript 'dashboard' %}
    {% endblock %}

``django-pipeline`` supports other output formats. For example, the
``editor-content`` bundle is processed with the javascript-array_ template,
which converts the URLs to a format that can be injected into a JavaScript
array, such as `the configuration script`_::

   win.mdn.assets = {
        css: {
            'editor-content': [
                {%- stylesheet 'editor-content' %}
                {%- stylesheet 'editor-locale-%s' % LANG %}
            ],
            'wiki-compat-tables': [{% stylesheet 'wiki-compat-tables' %}]
        },
        js: {
            'syntax-prism': [{% javascript 'syntax-prism' %}],
            'wiki-compat-tables': [{% javascript 'wiki-compat-tables' %}]
        }
    };

.. _`the revision dashboard`: https://github.com/mdn/kuma/blob/master/kuma/dashboards/jinja2/dashboards/revisions.html
.. _javascript-array: https://github.com/mdn/kuma/blob/master/jinja2/pipeline/javascript-array.jinja
.. _`the configuration script`: https://github.com/mdn/kuma/blob/master/jinja2/includes/config.html

.. _serve-whitenoise:

Serving assets with WhiteNoise
==============================
WhiteNoise_ is a static file serving application, and is an alternative to
serving static assets with nginx_, Apache_, or from `Amazon S3`_. On Kuma,
it is used to serve static assets in development as well as production. It
made it easy to serve HTML and related assets on the same `HTTP/2`_
connection.

In development (``DEBUG`` = ``True``) and testing, WhiteNoise is in
"autorefresh" mode, and uses the staticfiles-finder_. Each web request to
``/static`` scans for the file to use, which can be slow, but will catch
any changes made to the files.

In production (``DEBUG`` = ``False``), the files in ``STATIC_ROOT``
(``/static``) are indexed when the web server starts up. It also
determine headers, such as caching headers and the CORS_ header, that will be
sent with the file. This makes it very fast to serve static files, but changes
after the web server starts will not be noticed.

WhiteNoise provides its own :ref:`Storage classes <staticfiles-storage>`, that
can compress and cache static asset files. These are currently unused by Kuma,
which uses classes based on those provided by django-pipeline_.

.. _WhiteNoise: http://whitenoise.evans.io/en/stable/
.. _nginx: https://en.wikipedia.org/wiki/Nginx
.. _Apache: https://en.wikipedia.org/wiki/Apache_HTTP_Server
.. _`Amazon S3`: https://en.wikipedia.org/wiki/Amazon_S3
.. _`HTTP/2`: https://en.wikipedia.org/wiki/HTTP/2
.. _`CORS`: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS

Future
======

* Ensure files that are not meant for visitors are not collected, to speed
  up development, collecting, and preparing production images.
* Remove the ``CachedFileFinder`` and ``PipelineFinder``.
* Remove ``django-pipeline``, using ``webpack`` on the server as well before
  running ``./manage.py collectstatic``.
* Add ``django-webpack-loader`` or similar to integrate React assets

History
=======

The ``staticfiles`` application was probably part of the Kuma project from the
beginning in 2011. In the SCL3 datacenter, one of the first steps of a
production push was collecting the static files to a directory on a network
drive. This was shared between web servers, so that the new assets were
immediately avaiable as the new code was deployed. Because of file hashing, it
was possible to keep old versions of assets along with new versions. These
files were served by Apache.

In 2013, ``staticfiles`` was used to serve assets in the development Vagrant
environment instead of Apache, so that ``collectstatic`` was not needed to
see changes. However, CSS files were converted to Stylus_ that year, which
required compilation for development and deployment.

In 2015, several changes were made to prepare for the move from SCL3 to AWS.
One change was to move assets from the ``/media`` folder, which is
traditionally used for user uploads, to the ``/kuma/static`` folder.
Another was adopting ``django-pipeline`` to compile assets, and ``WhiteNoise``
to serve them in production.

In 2017, MDN hosting moved from SCL3 to AWS. Apache was no longer used to
serve assets, and ``WhiteNoise`` was used in production as well. This dropped
the ability to serve old versions of assets, but a CDN with long caching times
mitigated issues around deployments. That same year, the CSS sources were
converted from Stylus to Sass_.

In 2019, the development team decided to adopt new tools such as React_ and
Webpack_ (ADR-004_).

.. _Stylus: http://stylus-lang.com/
.. _React: https://reactjs.org
.. _Webpack: https://webpack.js.org
.. _ADR-004: https://github.com/mdn/mdn/blob/master/ADRs/004-use-react.md