Deploying Paradigm Variants with Helm
=====================================

We use [Helm](https://helm.sh/) to deploy our stack.  Helm's templating means the same templates can generate manifests for all sizes of environment.

If you haven't already, setup Minikube by following [these instructions](../../docs/minikube.md).

We use `values.yaml` and `secrets.yaml` files supplied to Helm for each environment.  The secrets are encrypted with [helm-secrets](https://github.com/futuresimple/helm-secrets) so it's safe to keep them in the repo.

Instructions are for OSX and the [fish shell](https://fishshell.com/) but they should work on bash/Linux with a little modification.  Examples are for greenmatters but other sites are supported too.

We assume you've chosen the name `suchverytest` for your environment.  Adjust examples to match your own env name.  Apart from the namespace this is used for:

Given default values the result of these steps will be services for your Paradigm variant at (e.g.):

*   http://greenmatters.local/
*   http://static.greenmatters.local/
*   http://media.greenmatters.local/

If you're running multiple environments at once you'll want to change the values to give them unique hostnames.

Notes:

*   Default behaviour is to deploy a site that's password protected and served with headers to discourage robots.  We do this to prevent non-prod sites getting indexed.  Set `frontend.lockdown,enabled=false` to disable this behaviour on production deploys.
*   Because you can't apply basic auth to the contents of `<img>` tags held on separate domains (browsers never prompt for auth, instead showing a broken image) we need separate ingresses for the web frontend (htauth applied) and media (auth-less).
*   The gateway pods are deployed in a [StatefulSet](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/).  This is a deliberate choice: while it prevents autoscaling it means a long-lived cache of resized media (mounted from a PersistentVolume) can survive across restarts.


## Dependencies

*   A copy of [paradigm-api](https://github.com/f12/paradigm-api), either in the same namespace or pointed to by the X value.
*   A copy of [thumbor](https://github.com/f12/devops/tree/master/services) installed to the same namespace.  It's fine for the different products (greenmatters/distractify/etc) to share it.  Thumbor doesn't have an ingress of its own but gets exposed through our gateway Nginx pod.


## Building

On a prod/staging system CircleCI will take care of this for you.

```plain
$ cd greenmatters
$ docker build -t greenmatters:suchverytest .
```

The image tag **must match your chosen environment name**.


## Testing

You can render Helm templates without applying them by doing:

```plain
$ helm template --name=myrelease --namespace=suchverytest f12-paradigm-site
```


## Deploying

At present the chart deploys two separate 'applications'

*   `frontend` - the node.js application serving the site
*   `gateway` - a copy of Nginx that sits in front of frontend to do various redirects and url routing.  It would have been nice to use the standard Nginx ingress for this but its configuration system isn't yet advanced enough.

It is anticipated that we will deploy multiple paradigm-site instances to a namespace, one for each of f12's brands.  Thus use `--set name=f12-paradigm-site-<brandname>` in the install command for each.  It's okay to use an abbreviation for the site name.

For the sake of consistency make sure these all follow the same format.  For example:

*   `f12-paradigm-site-gm`
*   `f12-paradigm-site-df`


### Local dev

#### For the first time...

```plain
$ cd deployment/helm
$ helm install --namespace=suchverytest \
               --name=greenmatters      \
               --set replicaCount=1     \
               f12-paradigm-site
```

If running on Minikube, add /etc/hosts entry:

```plain
$ echo -e "\n# Ingress for greenmatters running under minikube" | sudo tee -a /etc/hosts
$ echo (minikube ip) greenmatters.local static.greenmatters.local media.greenmatters.local | sudo tee -a /etc/hosts
```

#### Updating

```plain
$ cd deployment/helm
$ helm upgrade greenmatters f12-paradigm-site
```

#### Prod-ish Environments on Gcloud

Assuming your environment is called `prototype`:

```plain
$ helm install --namespace=prototype             \
               --name=greenmatters               \
               --set name=f12-paradigm-site-gm   \
               --set frontend.replicaCount=2     \
               --set frontend.ingress.tls=true   \
               --set frontend.ingress.www_redirect=true \
               f12-paradigm-site
```


## Database Setup

The site is really just a thin layer atop [paradigm-api](https://github.com/f12/paradigm-api).  Assuming your paradigm-api is already populated with a recent dump of the production DB it'll have entries in the `applications` table already, but we'll need to tinker with them a little.

Fire up `kubectl proxy` then open up the [Data Explorer](http://localhost:8001/api/v1/namespaces/suchverytest/services/rethinkdb-rethinkdb-admin/proxy/#dataexplorer).

Next you'll want to figure out the UUID of the application you're messing with.  We'll use this to refer to it since other identifiers (e.g. `hostname`) are about to change.

```plain
r
  .db('paradigm_suchverytest')
  .table('applications')
;
```

Scroll through the applications it returns and fish out the `id` field of the one you want.  In the current production DB, greenmatters is `f94a73d9-026b-4d4c-b8e9-bff273b69a63`.


### Change Site Names in Paradigm-API

Start by changing the hostname the site responds on.  You can do this in [paradigm-ui](http://paradigm-ui.local/applications/f94a73d9-026b-4d4c-b8e9-bff273b69a63), or by editing the database entries manually.


```plain
r
  .db('paradigm_suchverytest')
  .table('applications')
  .filter({id: "f94a73d9-026b-4d4c-b8e9-bff273b69a63"})
  .update({
    "hostnames": ["paradigm-ui.local"],
    "host":     "greenmatters.local",
    "hostPreview": "greenmatters.local"
  });
```

On the CDN, we cache via "services" and each service has its own hostname - so when you purge www.distractify.com, it does not purge the images that are referenced on that page
they are different services with different settings.  For example the site service TTL is much lower than the media service.

This service names are referenced by paradigm-api when it makes calls to the CDN's API to flush asserts.

```plain
r
  .db('paradigm_suchverytest')
  .table('applications')
  .filter({"id": "f94a73d9-026b-4d4c-b8e9-bff273b69a63"})
  .update({cdn: {services: {
    media: {hostname: "media.greenmatters.local"},
    site: {hostname: "greenmatters.local"}
  }}});
```

A separate setting controls the hostname at which the site will reference media...

```plain
r
  .db('paradigm_suchverytest')
  .table('applications')
  .filter({"id": "f94a73d9-026b-4d4c-b8e9-bff273b69a63"})
  .update({
    "media": {"hostname": "media.greenmatters.local"},
    "site":  {"hostname": "greenmatters.local"}
  });
```

If you're changing the protocol the site is served over it needs to generate different links to media...

```plain
r
  .db('paradigm_suchverytest')
  .table('applications')
  .filter({"id": "f94a73d9-026b-4d4c-b8e9-bff273b69a63"})
  .update({"protocol": "https"})
;
```

Finally you'll want to flush the `cache` table so pages with the old hostnames in aren't returned:

```plain
r
  .db('paradigm_suchverytest')
  .table('cache')
  .delete()
;
```


### Flush Redis

After flushing the RethinkDB cache table you may still have cached data in Redis.

```plain
$ kubectl --namespace=stage exec -it stage-paradigm-redis-master-0 bash
$ redis-cli -a <redis_admin_pass> flushall
```


### Disable HTTPS

**For local development only** you may wish to disable https.  Deployment-side this is done by setting the `ingress.tls` values but you'll need to change the site's setting in paradigm-api so it knows to generate media/static links for http://, not https://.

```plain
r
  .db('paradigm_suchverytest')
  .table('applications')
  .filter({"id": "f94a73d9-026b-4d4c-b8e9-bff273b69a63"})
  .update({protocol: "http"});
```

### Ingress Notes
It took a lot of fiddling to make ingress work right.  Don't tinker with it unless you know what you're doing.

*   A separate ingress must be run to redirect the apex domain.  This is because `nginx.ingress.kubernetes.io/from-to-www-redirect` doesn't work with ACME certs snagged via [cert-manager](https://github.com/jetstack/cert-manager).
    *   This can't just redirect everything.  If it did, ACME challenges to `/.well-known/...` would fail and we'd have no staging.  Thus redirects are not applied to paths with `.well-known` in.
    *   But even that causes problems.  The only place to add our `return 302 $scheme://www...` line is in a `nginx.ingress.kubernetes.io/configuration-snippet` annotation.  But this is applied _already within_ the site's location block.  If we supply an absolute path, when cert-manager tries to create its `/.well-known/...` route the Nginx IngressController notices a location inside for a parent path and crashes out with an error.  Boom, you just brought down ingress for your cluster.  Thus as a hacky workaround we match `.well-known` anywhere within the URL.