thedreaming.org

Kubernetes for Developers - Part 4: Monitoring

Mon, 07 Mar 2022 00:00:00 +0000

Kubernetes offers a powerful API that allow workloads to discover other services and pods within Kubernetes. The kube-prometheus stack takes advantage of this to discover information about running workloads, monitor them, and report on their performance.

kube-prometheus is made up of a lot of different components that work together; kube-state-metrics collects metrics about your cluster, prometheus queries kube-state-metrics and your own applications to collect statistics and store them in a database, alertmanager sends emails or other alerts when certain metrics exceed certain tolerances, and finally grafana is used to pull all this together into dashboards.

Installing

You can get most of this monitoring solution up and running with very few commands, thanks to the kube-prometheus-stack helm chart (formerly called the “prometheus-operator” chart, if you want to go searching for other tutorials):

# Create the "monitoring" namespace
$ kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -

# Deploy kube-prometheus-stack into the monitoring namespace
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm upgrade --install --namespace monitoring \
    prometheus-stack prometheus-community/kube-prometheus-stack

This will create a namespace called “monitoring”, and deploy Prometheus-and-friends into this namespace. If you run:

$ kubectl -n monitoring port-forward svc/prometheus-stack-grafana 3000:80

You can get access to the Grafana dashboard via http://localhost:3000. The default username is “admin”, with the password “prom-operator”.

If you click on the Search icon on the left, you can see a list of dashboards that have already been created for you. One of my favorites is the “General/Kubernetes/Compute Resources/Namespace (Pods)” dashboard, which will show you details about how much CPU and RAM each pod is using - you can switch which namespace or you’re looking at in a dropdown at the top. You can use the “General/Kubernetes/Compute Resources/Pod” dashboard to explore the resources used by any single pod.

You can also get access to the Prometheus dashboard:

$ kubectl -n monitoring port-forward svc/prometheus-stack-kube-prom-prometheus 9090:9090

This will let you enter a “PromQL” query and see the results instantly. We’ll talk more about PromQL in a minute. You can also get access to the alertmanager dashboard to see what alerts are currently defined and which are firing:

$ kubectl -n monitoring port-forward svc/prometheus-stack-kube-prom-alertmanager 9093:9093

And that’s it! We have monitoring deployed. Job well done… Well, not quite. This is a cool demo, but’s it not quite production ready. Where does Prometheus store it’s data? How can I let users from my organization get access to Grafana without us all sharing a single admin password? How do I create my own cool dashboards?

Let’s dig a little deeper into Prometheus, and answer some of those questions.

Prometheus

Prometheus is a systems monitoring and alerting engine, and a time-series database. Unlike other monitoring solutions, Prometheus is a “pull” based solution - Prometheus will actively go out and query a “/metrics” endpoint to fetch statistics from services it monitors. The list of services to monitor can be configured through a configuration file, but in the world of Kubernetes where pods come and go, we’ll use special resources called “PodMonitor”s and “ServiceMonitor”s to describe what Prometheus should monitor - we can create these via YAML or a Helm chart for each service we install.

This makes it fairly trivial to get Prometheus to watch custom statistics for your applications. It’s easy to export a “/metrics” endpoint that provides a count of how many HTTP requests a service has handled, for example, and then see a graph of HTTP requests across all your applications. Or you might export a “gauge” for the number of jobs currently being processed, and then you can see the sum of all jobs being processed across your cluster at a glance.

Prometheus uses a custom query language called PromQL to query statistics. Let’s go back to the Prometheus dashboard:

$ kubectl -n monitoring port-forward svc/prometheus-stack-kube-prom-prometheus 9090:9090

And visit http://localhost:9090. If you type rest_client_requests_total into the “Expression” bar at the top and click “Execute”, you’ll get a big table of counts. Each row in the table will look something like:

rest_client_requests_total{code="200", endpoint="http-metrics", host="control-plane.minikube.internal:8443", ...

rest_client_requests_total is the metric we’re interested in, and all those other values are “labels” - there’s one row in this table for each distinct set of labels. We can narrow our search to a specific label with a query like rest_client_requests_total{job="kubelet"}, which should reduce the number of rows in the table considerably. If you switch to the “graph” tab you can see each of these values graphed into a line. You’ll notice these lines are continuously growing - this is because counters in Prometheus always increase. The query rate(rest_client_requests_total{job="kubelet"}[5m]) will split this up into 5m intervals, and then plot the rate of change, giving us a more useful graph. That’s just a taste of PromQL - a full PromQL tutorial is outside the scope of this tutorial.

“alertmanager” is actually part of Prometheus - basically you define “alerts” with a PromQL query. If that query returns a count, then alertmanager will fire an alert, and then can forward that alert via email or PagerDuty, or whatever else you like.

Prometheus Storage

Prometheus has to store all this data it’s collecting. With the quick demo we’ve created so far, it’s getting stored in the Pod, eating up free space on the host machine. If the pod restarts for some reason, all the data will be lost. As we know from the previous tutorial, we want to store all this data in a Volume.

There are going to be some configuration files for bringing up this monitoring solution; let’s create a new github project to store these in:

$ mkdir monitoring
$ cd monitoring
$ git init .

Then we’ll create a “values.yaml” file:

alertmanager:
  config:
    receivers:
      # TODO: Add email receivers here.
      - name: "null"
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        metadata:
          labels:
            storage: prometheus
        spec:
          storageClassName: standard
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi
prometheus:
  prometheusSpec:
    retention: 10d
    retentionSize: 160GB
    # Allow ServiceMonitors/PodMonitors from any namespace
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    storageSpec:
      volumeClaimTemplate:
        metadata:
          labels:
            storage: prometheus
        spec:
          storageClassName: standard
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 200Gi
grafana:
  persistence:
    enabled: true
    storageClassName: standard
    size: 10Gi

And now we can update our deployment with:

$ helm upgrade --install --namespace monitoring \
    -f values.yaml \
    prometheus-stack prometheus-community/kube-prometheus-stack

This will configure Prometheus to use a 200GB EBS volume, and give AlertManager a 50GB volume. Prometheus will retain data for 10 days, or for 160GB (whichever comes first). Note that Prometheus’s “retentionSize” doesn’t account for all the disk space that gets used, so we want to set this a healthy value below the total disk space we actually have. Also, you may notice that the retentionSize is expressed in “GB” while the storage is in “Gi” - that’s because the storage is going to Kubernetes, and the retentionSize is going to be passed as --storage.tsdb.retention.size 160GB to prometheus, and these two applications expect these values in different formats.

We also give Grafana a small 10GB drive - this lets users and charts you create persist across restarts. Generally if you create charts, it’s better to create them in Kubernetes, so they’ll be there if you want to recreate your deployment on another stack. We’ll look at that a bit later on. But it’s handy to be able to edit charts to prototype them.

This also has a “receivers” section which at the moment only has the default “null” receiver in it. There’s an excellent tutorial about setting up receivers from Grafana’s blog.

If you’re curious about what other values you can put in here, this is the Values.yaml file for kube-prometheus-stack. This depends on some other helm charts; everything under the grafana: section, for example, gets sent to the grafana helm chart. If you want to read up on Grafana configuration, the documentation for that chart will be what you’re after.

SOPS

In a minute, we’re going to configure Grafana to do single-sign-on using OAuth. To do this, we’re going to have to store some secrets in some configuration files. It’s a bad idea to commit secrets to a repository, so we’re going to use a tool from Mozilla called SOPS to help us store encrypted secrets in our repository.

To use SOPS, first we need to install it. You can find releases on GitHub, or on a Mac you can brew install sops.

In AWS, go to the Key Management Service and create a new symmetric key. Take note of the key’s ARN. In the root of your project create a file called “.sops.yaml”, and fill in the key ARN:

creation_rules:
  - kms: arn:aws:kms:us-west-2:<account_id>:key/<key_id>

Also in the root of your project, edit .gitignore and add the following:

*.plaintext.yaml

This will prevent any “plaintext” files from being committed to source control by accident.

Let’s give this a try to make sure it’s working. Create a file somewhere in your project called “mysecrets.plaintext.yaml:

githubPassword: supersecret

Then run SOPS:

$ sops -e mysecrets.plaintext.yaml > mysecrets.encrypted.yaml

If we open up mysecrets.encrypted.yaml, we’ll see it starts with the line:

githubPassword: ENC[AES256_GCM,data:K3/UeWyuM+vzHIY=,iv:I2W6+hUH6WMaD2nWAJ5+AFyzGpVq6rrRwG+YXJioe/U=,tag:k8jzDHJ8OYNRxbz6OrnWsA==,type:str]

followed by some metadata about how this value was encrypted. One of the nice things about SOPS is that the structure of our YAML file is preserved - we can see in GitHub’s history when certain values were changed or added.

We can decrypt this file with:

$ sops -d mysecrets.encrypted.yaml > mysecrets.plaintext.yaml

Grafana Single-Sign-On

We’re going to want to expose Grafana to other users in our organization, and we are not going to create individual user accounts for them. The solution to this is to use single-sign-on. In this example, we’re going to use GitHub as an OAuth provider, allowing anyone who is part of our GitHub org to access the dashboard.

To do this, we’re going to need a domain name. I like to use something like “ops.example.com” to group all my ops-related internal services together, so here I’d create a “grafana.ops.example.com”. We’ll use that as an example throughout this, but obviously you’ll need to replace this with the domain name you choose.

First, go visit https://github.com/<my-org>/. Click on the “Settings” tab, then click on “Developer Settings”, then “OAuth Apps” and click “Register an Application”. Call your application “Grafana”, set the “Homepage URL” to your organization’s homepage. Set the “Authorization callback URL” to https://granafa.ops.example.com/grafana/login/github.

Once you create the application, you’ll get a Client ID, and you’ll need to generate a “client secret”. Add both of these to a secrets.plaintext.yaml file in the root of your project:

# secrets.plaintext.yaml
grafana:
  adminPassword: super-secret
  envRenderSecret:
    GITHUB_CLIENT_ID: YOUR_GITHUB_APP_CLIENT_ID
    GITHUB_CLIENT_SECRET: YOUR_GITHUB_APP_CLIENT_SECRET

Then we’re going to make some updates to helm/monitoring/values.yaml:

alertmanager: ...
prometheus: ...
grafana:
  persistence:
    enabled: true
    storageClassName: standard
    size: 10Gi
  service:
    # Must be NodePort to work with ALB Ingress.
    type: NodePort
  grafana.ini:
    server:
      # Replace this with your domain name.
      domain: granafa.ops.example.com
      root_url: "https://%(domain)s/grafana"
      serve_from_sub_path: true
    auth.github:
      enabled: true
      allow_sign_up: true
      client_id: "$__env{GITHUB_CLIENT_ID}"
      client_secret: "$__env{GITHUB_CLIENT_SECRET}"
      scopes: user:email,read:org
      auth_url: https://github.com/login/oauth/authorize
      token_url: https://github.com/login/oauth/access_token
      api_url: https://api.github.com/user
      # Replace this with your GitHub org name.
      allowed_organizations: <your-org-here>
  # ingress:
  #   enabled: true
  #   # Replace this with your domain name.
  #   hosts: ["grafana.ops.example.com"]
  #   annotations:
  #     kubernetes.io/ingress.class: "alb"
  #     alb.ingress.kubernetes.io/scheme: "internet-facing"
  #     alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
  #     alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
  #     alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:us-west-1:<account-id>:certificate/<certificate-id>"

Be sure to correct the domain, and to update <your-org-here> with your GitHub organization name - note that this is case sensitive. We can deploy this with:

$ helm upgrade --install --namespace monitoring \
    -f values.yaml \
    -f secrets.plaintext.yaml \
    prometheus-stack prometheus-community/kube-prometheus-stack

The idea here is that when we do a deploy, helm will use the values from values.yaml, but then will merge in any values we put in the “secrets.plaintext.yaml” file to produce the file values passed to the templates (actually, it first takes the “values.yaml” defined in the chart itself, then merges in our values.yaml, then merges in our secrets file). Our secrets file sets a new default password for Grafana (although this will only take effect the first time we deploy this - it won’t change the existing password), and also sets up some environment variables that the Grafana chart will render into a Kubernetes “Secret” for us. These environment variables are available in the grafana.ini file, which we use to set the client_id and client_secret for OAuth.

You may notice the Ingress section is commented out. The grafana Helm chart creates a networking.k8s.io/v1:Ingress, and this isn’t supported by the AWS load balancer controller yet. So instead we need to create the ingress ourselves:

# ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: monitoring
  name: monitoring-ingress
  annotations:
    kubernetes.io/ingress.class: "alb"
    alb.ingress.kubernetes.io/scheme: "internet-facing"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:us-west-2:<account-id>:certificate/<certificate-id>"
spec:
  rules:
    - host: grafana.ops.example.com
      http:
        paths:
          - path: /*
            backend:
              serviceName: prometheus-stack-grafana
              servicePort: 80

Because Grafana has it’s own login system, we can just expose it directly to the Internet. You’ll notice the Prometheus and Alertmanager interfaces have no login - you can access them via port-forwarding, or if you want to you can expose them on the Internet behind something like OAuth2 Proxy, or use kube-auth-proxy, to add an authentication layer.

You should be able to visit https://grafana.ops.example.com and click the “Sign in with GitHub” button at the bottom. If you can’t connect at all, try a kubectl -n monitoring describe ingress to see if there are any errors from your ingress. If you get errors from GitHub, try right clicking and doing “Inpsect”, then switch to the Network tab and try again - there should be an error back from the request to github.com that will point you in the right direction.

If you need to check logs from grafana, you’ll need to do:

$ kubectl -n monitoring get pods
NAME                                                     READY   STATUS    RESTARTS   AGE
prometheus-stack-grafana-65b7b7b454-lnqw8                2/2     Running   0          3m34s
...

$ kubectl -n monitoring logs prometheus-stack-grafana-65b7b7b454-lnqw8 grafana

Note that last “grafana” at the end of the logs command - this is because the Grafana pod has more than one container running inside it (this is the “Ready: 2/2” in “get pods”), so you need to select which container you want to view the logs for.

Once you’ve signed into Grafana via github, logout and sign in again as the “admin” user. Then on the left under “Server Admin” => “Users”, go find your GitHub user, click on them, and under “Permissions”, change Grafana Admin from “No” to “Yes”. This makes your user a Grafana admin. Then, below that under “Organizations”, click “Change role” next to “Main Org.”, and make yourself an admin for the organization too (which will let you create new dashboards).

Adding Metrics to our Application

Now we have Prometheus and Grafana up and running, it would be nice if we could add some custom metrics to our application. Fortunately, this is very easy to do - we just need to add a “/metrics” endpoint in our app that serves up Prometheus statistics. The two most common metric types in Prometheus are counters and gauges. A counter is something that only ever increases, a gauge is something that gets set to a certain level and can go up or down.

Let’s say we have a transcoder server, and we want to keep track of how many jobs are currently in progress at any moment (via a gauge), and a count of how many jobs we’ve completed. It might be nice to be able to work out the average length of a job, so we could also keep a total count of seconds we’ve spent processing jobs. Jobs are either on a “normal” queue or a “priority” queue, so we want to add a label to each of these metrics. The “/metrics” endpoint would need to serve up something like this:

# HELP transcoder_server_process Gauge that is always 1.
# TYPE transcoder_server_process gauge
transcoder_server_process{version="1.1.0"} 1

# HELP jobs_in_progress Number of jobs in progress
# TYPE jobs_in_progress gauge
transcoder_server_jobs_in_progress{queue="normal"} 0
transcoder_server_jobs_in_progress{queue="priority"} 1

# HELP jobs_complete_count Total number of jobs processed (both failed and succeeded)
# TYPE jobs_complete_count counter
transcoder_server_jobs_complete_count{queue="normal"} 1
transcoder_server_jobs_complete_count{queue="priority"} 0

# HELP jobs_duration_seconds Total duration of jobs in seconds
# TYPE jobs_duration_seconds counter
transcoder_server_jobs_duration_seconds{queue="normal"} 20
transcoder_server_jobs_duration_seconds{queue="priority"} 0

Most of these are pretty self-explanatory, but the “transcoder_server_process” gauge might be a little confusing at a glance. Having a gauge that’s always “1” seems like nonsense, but it’s actually a very common technique in Prometheus. If we have two instances running, the two gauges will be added together to make 2, so we can use this gauge to keep an eye on how many instances are running. The “version” label lets us track what version is currently running in the cluster, and see when it changed.

Note that the “/metrics” endpoint doesn’t have to be served on the same port as your application - you can serve a customer facing application from your container on port 3000, for example, and serve the “/metrics” endpoint from 3001, effectively making the stats endpoint unreachable from the outside world.

Most languages have some kind of library for generating Prometheus stats. For example, in node.js there’s prom-client. Here’s a quick example of a metrics.ts file:

import * as http from 'http';
import * as promClient from 'prom-client';
import { QueueType, Stage } from './constants';

const PREFIX = 'transcoder_server';

const jobsInProgress = new promClient.Gauge({
    name: `${PREFIX}_jobs_in_progress`,
    help: 'Number of jobs in progress',
    labelNames: ['queue'],
});

const jobsComplete = new promClient.Counter({
    name: `${PREFIX}_jobs_complete_count`,
    help: 'Total number of jobs processed (both failed and succeeded)',
    labelNames: ['queue'],
});

const jobsDuration = new promClient.Counter({
    name: `${PREFIX}_jobs_duration_seconds`,
    help: 'Total duration of jobs in seconds',
    labelNames: ['queue'],
});

const jobsFailed = new promClient.Counter({
    name: `${PREFIX}_jobs_failed_count`,
    help: 'Total number of jobs that failed',
    labelNames: ['queue'],
});

let initialized = false;

/**
 * Start the metrics server running.
 */
export function createMetricsMiddleware(): (req: http.IncomingMessage, res: http.ServerResponse) => void {
    const register = promClient.register;

    // Lazy-initialize promClient.
    if (!initialized) {
        initialized = true;

        promClient.collectDefaultMetrics({
            labels: { app: 'transcoder_server' },
            register,
        });
    }

    return (req: http.IncomingMessage, res: http.ServerResponse) => {
        register
            .metrics()
            .then((result) => {
                res.setHeader('Content-Type', register.contentType);
                res.writeHead(200);
                res.end(result);
            })
            .catch((err) => {
                res.writeHead(500);
                res.end(err.message);
            });
    };
}

export function startServer() {
  const metricsMiddleware = createMetricsMiddleware();
  const server = http.createServer((req, res) => {
      switch (req.url) {
          case '/metrics':
              metricsMiddleware(req, res);
              break;
          case '/status':
              res.writeHead(200);
              res.end('ok');
              break;
          default:
              res.writeHead(404);
              res.end();
      }
  });
  server.listen(DEFAULT_PORT);
}

export function startJob(queue: QueueType) {
    jobsInProgress.inc({ queue });
}

export function endJob(queue: QueueType, duration: number, failed: boolean) {
    jobsInProgress.dec({ queue });
    jobsComplete.inc({ queue });
    jobsDuration.inc({ queue }, Math.floor(duration / 1000));
    if (failed) {
        jobsFailed.inc({ queue });
    }
}

When you’re creating your own metrics, keep note of the fact that each distinct set of labels will result in a new series of data in Prometheus - if you add a label like “username”, this is nice because you can track stats for each individual user, but it’s not so nice in that you’ll create a time series for each individual user, potentially eating up terrabytes of disk space. Even your “/metrics” endpoint would return potentially thousands of rows for each metric. Try to keep the cardinality of your metrics low.

You’ll note here we’re adding a “queue” label to each of our counters, but if you actually give this a try, Prometheus will add a few extra labels for us:

transcoder_server_jobs_in_progress{
  container="transcoder-server",
  endpoint="http",
  instance="20.0.114.172:3001",
  job="default/transcoder-server-int",
  namespace="default",
  pod="transcoder-server-int-priority-944d74885-r45s4",
  queue="priority"
}

This lets us easily grab counters from, for example, a specific pod to see if that pod is misbehaving in some way. At a glance, adding all these extra labels would seem to be counter to the advice we just discussed - to limit the cardinality of your labels. In practice it’s not such a problem - the “pod” here has a 1:1 correspondence with the “instance”, for example, so these together will only create a single series in the database.

Adding a PodMonitor

Now our application has a “/metrics” endpoint - we need to let Prometheus know where to find it. As mentioned above, this is done using a custom resource called a PodMonitor. In our service’s .yaml file, or somewhere in it’s Helm chart if it has one, we can add a PodMonitor:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: myService
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: transcoder-server
  podMetricsEndpoints:
  - port: http
    path: /metrics

Here the “matchLabels” tells Prometheus which pods it should be monitoring, and the podMetricsEndpoint tells Prometheus how to scrape metrics. After kubectl applying this file, Prometheus should start scraping data from our application.

That wraps it up for Prometheus and Grafana. In the next part of our series we’ll look into how Prometheus’s “alertmanager” can be used to alert us when something is going wrong in our cluster.

Kubernetes for Developers - Part 3: Volumes and Autoscaling

Mon, 07 Mar 2022 00:00:00 +0000

In part two, we deployed a service to EKS and set up an Ingress. In this section, we’re going to discuss provisioning volumes, StatefulSets, and autoscaling.

Storage

So far all of the services we’ve looked at have not used much storage. If we were creating pods for a database, or creating pods for a service like a video transcoder that needed a large scratch disk, we might want more than whatever disk space happens to be available on the pod’s host file system. We’d also want data to persist if the pod restarts or is moved to a new node.

The solution to this is called a “volume” - volumes are very similar to their Docker namesake. The basic idea behind a volume is that a pod creates a “PersistentVolumeClaim” or “PVC”, which Kubernetes will match up with a “PersistentVolume” or “PV”. Volumes can either be manually created ahead of time, or can be allocated dynamically as required. A persistent volume will “follow” a Pod around from node to node (or, alternatively, Kubernetes will ensure that the Pod can only be scheduled on nodes where the volume is available).

Kubernetes doesn’t have any idea what kind of hardware environment your cluster is running in, so much like how we installed the “AWS Load Balancer Controller” to handle Ingress, we’re going to install a “CSI Driver” to manage volumes. The name makes you think we’re going to be doing forensics and fighting criminals, but sadly it stands for “Container Storage Interface”.

If you’ve been following this tutorial, you may notice that we created a ServiceAccount called “ebs-csi-controller-sa” back in part 2, but we haven’t used it yet. Now is the time! We’re going to install the EBS CSI driver which will create and mount volumes for us:

$ helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver && \
  helm repo update && \
  helm upgrade -install aws-ebs-csi-driver aws-ebs-csi-driver/aws-ebs-csi-driver \
    --namespace kube-system \
    --set image.repository=602401143452.dkr.ecr.us-west-1.amazonaws.com/eks/aws-ebs-csi-driver \
    --set controller.serviceAccount.create=false \
    --set controller.serviceAccount.name=ebs-csi-controller-sa

Also like with the ingress, we could have multiple CSI drivers installed - there’s one for AWS EFS, one for “FSx for Lustre”, etc… You can also provision “local storage”, where the volume is stored on the disk of the host node. When you create a PVC, you need to tell it which of these many storage mechanisms to use. But, you also don’t want to specify in each and every PVC that it must use the “EBS CSI Driver” - this would make it challenging to move your application from one Kubernetes cluster to another. Instead you want to say “I want bulk persistent storage” or “I want storage that’s very fast”.

Kubernetes addresses this with an extra level of indirection called a “StorageClass”. You create a StorageClass that describes how to allocate resources, and then your PVCs and PVs make reference to the StorageClass. To create a StorageClass for EBS volumes we could do:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer

Now any time we use the “standard” StorageClass, Kubernetes will know to use the “ebs.csi.aws.com” provisioner. If you deploy your application to some other environment, you could define a “standard” StorageClass there which does something else. Minikube has a “standard” StorageClass created for you, with a simple CSI driver that allocates local disk space. Some CSI drivers may let you specify extra values in the StorageClass via annotations - whether the disk be an old-style spinning drive or an NVMe drive, for example.

The volumeBindingMode here describes when to bind a volume. By default it is “Immediate”, meaning that a volume will be created as soon as the PVC is created. This can cause problems though. Remember that AWS regions are divided up into Availability Zones, or AZs. In order for an EBS volume to be mounted on an EC2 instance, they have to be in the same AZ. If we use Immediate volume binding, then when the PVC is created, your EBS drive would need to be created in a particular AZ, but the associated Pod hasn’t been scheduled yet, so Kubernetes won’t know which AZ to create the EBS drive in. “WaitForFirstConsumer” tells Kubernetes to wait until there’s someone who wants to use a PVC before trying to allocate the PV.

We can create a PVC that uses this StorageClass directly (borrowing an example from the CSI driver docs):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 4Gi

And then we can create a Pod that uses the PVC:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: centos
      command: ["/bin/sh"]
      args:
        ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
      volumeMounts:
        - name: persistent-storage
          mountPath: /data
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: ebs-claim

Most often, though, we don’t create PVCs directly, just like we wouldn’t create pods directly. Instead we have some stateful service that wants to store data, and so we create a “StatefulSet”, which in turn creates the PVCs and pods together.

StatefulSets

In the first part of this tutorial we showed how to deploy stateless services using a Deployment. Sometimes, though, you have a service that is not stateless, such as a database. Stateless and stateful services behave very differently. Typically you want to have a fixed number of database pods running - a primary and a backup, or maybe you have three instances of your app running paxos or raft to pick the current primary. Unlike with a simple web server, you wouldn’t want Kubernetes to create extra database nodes on a whim, or to kill off databases without concern for their disk contents. Clearly a “Deployment” isn’t adequate here. We need something new, and that something is called a “StatefulSet”.

In a StatefulSet, each running pod has a unique ID assigned to it (postgres-1, postgres-2, etc…). A database is also kind of useless if it doesn’t store any data, so typically members of a stateful set also have a “persistent volume claim template” to create a PVC for each pod.

Pods in a StatefulSet can be moved to another physical node, but only if the volume can be moved. There are different kinds of volumes in Kubernetes - if you’re using EBS volumes then Kubernetes can shut down a pod on one node, detach the EBS volume, move the volume to a new node and restart it, and the pod will never know that’s it is now on a new node (at least, so long as the nodes are in the same AZ). On AWS, you might use a “local storage” driver for a “storage optimized” EC2 instance like an ie3n instance, with high-speed NVMe drives attached. In this case, since the volume can’t be moved, Kubernetes will not be able to move the pod, and will make sure it is always scheduled on the same node.

Let’s see an example. Here is a YAML file that will bring up a three-node cluster of Elasticsearch nodes:

apiVersion: v1
kind: Service
metadata:
  name: es
  labels:
    service: elasticsearch
spec:
  ports:
    - port: 80
      name: web
  clusterIP: None
  selector:
    service: elasticsearch
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  labels:
    service: elasticsearch
spec:
  serviceName: es
  replicas: 3
  selector:
    matchLabels:
      service: elasticsearch
  template:
    metadata:
      labels:
        service: elasticsearch
    spec:
      terminationGracePeriodSeconds: 300
      securityContext:
        fsGroup: 1000
      initContainers:
        - name: init
          image: busybox
          command: ["/bin/sh", "-c"]
          args: ["sysctl -w vm.max_map_count=262144 && ulimit -n 65536"]
          securityContext:
            privileged: true
      containers:
        - name: elasticsearch
          image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4
          ports:
            - containerPort: 9200
              name: http
            - containerPort: 9300
              name: tcp
          env:
            - name: cluster.name
              value: elasticsearch-cluster
            - name: node.name
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: discovery.zen.ping.unicast.hosts
              value: "elasticsearch-0.es.default.svc.cluster.local,elasticsearch-1.es.default.svc.cluster.local,elasticsearch-2.es.default.svc.cluster.local"
          volumeMounts:
            - name: data
              mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: standard
        resources:
          requests:
            storage: 50Gi

This creates a “headless service” called “es”. It’s called a “headless service” because the clusterIP is set to “none”, which means Kubernetes won’t do any load balancing between the members of the service for us.

If you apply this, you’ll see three pods get created (likely you’ll see one pod get created first, and then once it starts the other two will show up). Unlike with a “deployment”, where pods get names with a random suffix, here the pods will be named “elasticsearch-0”, “elasticsearch-1” and “elasticsearch-2”. Each pod will have a PVC and PV associated with it. Each pod gets a domain name like “elasticsearch-0.es.default.svc.cluster.local”, based on the name of the pod and the name of the service.

Updates work a little differently for StatefulSets too. The default option here is updateStrategy: {type: "RollingUpdate"}, where Kubernetes will replace each instance in the StatefulSet one-by-one. Here it’s important to define readiness checks for each pod (which we haven’t discussed yet) so that the rolling update knows when a node is ready and done upgrading. The other option here is type: "OnDelete", which makes it so the StatefulSet controller will not automatically update pods - you have to manually delete pods one-by-one, and handle the upgrade yourself.

Autoscaling

Now that we know how to create deployments and stateful sets, let’s talk about Autoscaling. This is the idea that we can ramp up the number of pods we have in a deployment, or the amount of resources allocated to pods automatically. There are three different kinds of autoscaling:

Cluster autoscaling lets us scale up (or down) the number of nodes in our cluster. There’s no use creating new pods if we run out of nodes to run them on.
Horizontal autoscaling is about scaling the number of pods that are currently running. If our web backend has a high CPU usage, or high memory usage, or even some custom stat we want to track, we can increase the number of pods to try to reduce these.
Vertical autoscaling is about dynamically setting the CPU and memory limits for a pod - if a pod starts taking up too much CPU, we’ll increase the CPU allocated to it (which may cause some pods to be rescheduled to new nodes). This is less often used - we’re not going to discuss this here.

Cluster Autoscaling

Autoscaling your cluster is obviously going to depend a great deal on how you’re running your cluster (GCP vs. AWS vs. other cloud providers). The rest of these tutorials have focused on AWS, so we’ll keep to that theme. When you set up your nodegroups in EKS, you can specify a minimum and maximum number of nodes.

If you are running “availability-zone aware” loads, you need to be aware that the auto-scaler treats every node in a node group the same. For example, if you have a StatefulSet that uses EBS volumes, each pod in that StatefulSet can only start on a node in the same AZ as its associated EBS volume. In this case, you need to set up a nodegroup in each AZ. In eksctl, this would look like:

nodeGroups:
  - name: ng1-public-2a
    instanceType: t3.small
    availabilityZones: ["eu-west-2a"]
    minSize: 1
    maxSize: 4
    tags:
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/<CLUSTER-NAME>: owned
  - name: ng1-public-2b
    instanceType: t3.small
    availabilityZones: ["eu-west-2b"]
    minSize: 1
    maxSize: 4
    tags:
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/<CLUSTER-NAME>: owned

If all you’re doing is running stateless services, you can omit the “availabilityZones” and just create a single node group. We’re going to create some EBS volumes later, though, so we’ll go with the two-nodegroup option.

Once you have your nodegroups created, you need to install the cluster-autoscaler, which you can do with this helm command:

$ helm upgrade --install auto-scaler autoscaler/cluster-autoscaler \
  --namespace kube-system \
  --set 'autoDiscovery.clusterName'=my-cluster \
  --set 'rbac.serviceAccount.create'=false \
  --set 'rbac.serviceAccount.name'=cluster-autoscaler \
  --set 'extraArgs.expander'=least-waste \
  --set 'extraArgs.balance-similar-node-groups=true'

Now, if you have two nodes running, and scheduling a new pod fails (because, for example, the resources section in the pod definition requests more CPU or memory than is available), then AWS will automatically create a new node to schedule pods on.

Horizontal Autoscaling

In order to horizontally scale pods, you need to create a “HorizontalPodAutoscaler”, or “HPA”:

# 2048.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game
  labels:
    app: web
spec:
  # No `replicas` - the replica count is controlled by the HPA
  selector:
    matchLabels:
      app: game
  template:
    metadata:
      labels:
        app: game
    spec:
      containers:
        - name: web
          image: alexwhen/docker-2048:latest
          ports:
            - name: http
              containerPort: 80
          # Specify that this "wants" 20% of a CPU.
          resources:
            requests:
              cpu: 200m
            limits:
              cpu: 1000m
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: game
  labels:
    app: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: game
  minReplicas: 1
  maxReplicas: 4
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 800m

Notice that we don’t specify a replicas in the deployment - the HPA will take care of setting this value for us. If average CPU usage across all pods in the game deployment is greater than 80%, we’ll create more pods, up to a maximum of 4. Similarly, if the CPU usage is low, we’ll scale down the number of pods to a minimum of 1. You can learn more about the specific algorithm that is used to determine when to scale up or down by reading the official docs.

Also note that we now specify a resources section for the container, which says this pod would like at least 20% of a CPU, and is allowed to use a maximum of 100% of a CPU. If we don’t specify a resources.requests, then the CPU utilization for this pod will be undefined in the Kubernetes API, and the HPA will ignore our pods.

Conclusion

This part of our tutorial covered three important concepts - volumes, stateful sets, and auto scaling. Next time we’re going to talk about monitoring your Kubernetes cluster with Prometheus and Grafana.

Node.JS: Upgrading indirect dependencies

Tue, 22 Feb 2022 00:00:00 +0000

You want to install an npm package - let’s say amqp-connection-manager. This particular package has a dependency on another package called promise-breaker, so when you install ampq-connection-manager, promise-breaker gets installed automatically (although you won’t see it in your package.json). Now let’s suppose there’s a security vulnerability in promise-breaker. You want to upgrade promise-breaker, but it isn’t even in your package.json! How do you go about doing this?

SemVer

First, a quick explanation of “semantic versioning”. Node.js packages have version numbers like “5.3.2”. Here “5” is called the major version, “3” is the minor version, and “2” is the patch version. When a new version of a package comes out, it should increment the major version if there are breaking changes. If not, it should upgrade the minor version if there are new features, or the patch version if there are only bug fixes. This means you know it should be reasonably safe to upgrade from 5.3.2 to 5.3.3 (since this only contains bug fixes) or to 5.4.0 (which has some new features, and maybe some bug fixes) but upgrading to 6.0.0 means breaking changes and you’ll probably have to do some work to do that upgrade. “0.x.y” versions are special; here the “x” is usually treated as the major version, and the “y” as the “minor/patch” version.

If you peer into the package.json for amqp-connection-manager, you’ll see that it relies on:

  "dependencies": {
        "promise-breaker": "^5.0.0"
  }

Notice the ^ (called a “caret”) - this is important! The caret means that this package relies on 5.0.0, or any version higher than 5.0.0 with a “5” in the major version number. This means if you already have promise-breaker 5.1.0 installed when you try to install amqp-connection-manager, npm won’t install 5.0.0, it will just use the 5.1.0 you already have installed. And, this also means if a new 5.x.x version of promise-breaker comes out, you can upgrade to it without upgrading amqp-connection-manager.

The caret also correctly interprets versions with a leading 0 - ^0.1.2 means this depends on 0.1.x for any x >= 2. Sometimes you will also see tilde ranges like ~5.2.1. These are very similar to caret ranges - in most cases you can use them interchangably.

If you find a package that doesn’t have a caret or tilde range, for example if you find a package that contains something like:

  "dependencies": {
        "promise-breaker": "5.0.0"
  }

Then you’re in trouble. There’s ways to force npm or yarn to upgrade to a higher version, but the best way to resolve this is to raise a PR against the package that’s importing promise-breaker, and get them to upgrade (and preferably get them to replace “5.0.0” with caret or tilde range).

Upgrading with npm

So, back to our original example - you installed ampq-connection-manager, and you want to upgrade it’s dependency promise-breaker. promise-breaker will not be in your package.json, but you might notice that it is in your package-lock.json. If you rm -rf node_modules and npm install, then npm will just reinstall the old version of promise-breaker, because that’s what’s in your lock file.

Before doing anything else, make sure you’ve upgraded to the latest version of npm, because the upgrade command in npm has been notoriously buggy in the past:

$ npm install -g npm

If you need to upgrade promise-breaker because of a security problem from npm audit, you can npm audit fix, which will try to automatically fix call security problems. This may upgrade other libraries too.

If you know that you want to upgrade a direct dependency of one of your dependencies, then the command you want to run is:

$ npm upgrade --depth 2 promise-breaker

This will search through your package-lock.json for packages named “promise-breaker” to upgrade, but it will only do your direct dependencies (at depth 1) and their direct dependencies (at depth 2). If you depend on a library that depends on amqp-connection-manager, then promise-breaker would be at --depth 3. You can figure this out by running npm ls promise-breaker, which will show you how far down the dependency tree promise-breaker is. On small projects you might be able to get away with something like npm upgrade --depth 999 promise-breaker, but you might find this a bit slow on larger projects.

If npm upgrade --depth ... doesn’t work for you (which it sometimes doesn’t - notoriously buggy, and sometimes for whatever reason you have to use an older version of npm) then there are a few other options you can try here. You can npm uninstall amqp-connection-manager && npm install amqp-connection-manager - this will often fix the problem. You can also npm install promise-breaker@5.2.0 && npm uninstall promise-breaker, which will force it up to 5.2.0 and then remove it from package.json. The “nuclear option” is rm -rf node_modules package-lock.json && npm install, which will regenerate your package-lock.json from scratch, although this may upgrade several other libraries in the process.

Upgrading with Yarn

Yarn makes this a little easier - if you run:

$ yarn upgrade amqp-connection-manager

will upgrade amqp-connection-manager to the latest version (matching semver) and all it’s dependencies. Note that if you run yarn upgrade promise-breaker, this won’t do what you want it to - you need to know the pacakge that includes promise-breaker. The easiest way to figure this out is with:

$ yarn why promise-breaker

which will tell you why the dependency is in your dependency tree.

Happy upgrading!

Kubernetes for Developers - Part 2: EKS and Ingress

Thu, 21 Oct 2021 00:00:00 +0000

In part one, we created a service and we learned how to connect to our service from inside the cluster, and how to connect to it from outside the cluster with with port forwarding - how would we go about exposing this service on the Internet?

The answer is a Kubernetes resource called an Ingress, which describes how traffic gets into your cluster and which services traffic gets routed to. There are lots of different kinds of Ingress to choose from - you get to pick one and install it on your cluster. What an Ingress looks like in terms of network architecture will depend very much on which Ingress you choose.

If you were to install ingress-nginx, for example, then your ingress would consist of nginx running in several pods within your cluster (and then some method for getting all inbound traffic to those nginx pods, so it can distribute it to your services). Traefik-ingress is another popular choice, because of it’s built in support for fetching certificates from LetsEncrypt. In this tutorial, we’re going to use AWS and EKS, so we’re going to go the simple route and install the AWS load balancer controller, which will create ALBs for our ingress traffic.

In order to do that, we’re going to set up a cluster on AWS’s EKS service, then we’re going to learn probably more than we want to know about security and Service Accounts, and we’ll learn how to set up an Ingress. We’ll also look at how to automatically configure a DNS entry for our service in Route 53 and setup SSL with a certificate from AWS.

Creating a Cluster in EKS

We did our first tutorial with minikube, but in order to expose a service to the Internet, we need a cluster that’s connected to the Internet and which has features like a load balancer. These are both things which minikube is sadly missing. So, for this tutorial we’re going to launch a cluster on AWS using Elastic Kubernetes Service (EKS). Note that this is not free - an EKS cluster costs about $0.10 USD per hour, depending on which region you launch it in, and then there are additional costs for the EC2 instances that get launched for running workloads. Make sure you destroy your cluster when you’re done working through this tutorial - 10 cents an hour sounds cheap, but if you follow this tutorial and leave it running in us-east-1 for a month by accident, you’ll be look at approximately $90 in AWS charges.

You need a few tools installed first; kubectl to talk to Kubernetes (which you already have installed if you followed along with part one), eksctl to create a cluster, aws-iam-authenticator for authentication, and helm to install helm charts. On a Mac, if you’re using brew, you can install all of these with: brew install kubectl eksctl aws-iam-authenticator helm. Installing these on other operating systems is left as an exercise for the reader.

We’re going to create our cluster with eksctl, a tool for creating EKS clusters. In order to create a cluster with eksctl, first we’re going to create a configuration file that describes our cluster, called “cluster.yaml”:

# cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: my-cluster
  region: us-west-1

iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: aws-load-balancer-controller
        namespace: kube-system
      wellKnownPolicies:
        awsLoadBalancerController: true
    - metadata:
        name: ebs-csi-controller-sa
        namespace: kube-system
      wellKnownPolicies:
        ebsCSIController: true
    - metadata:
        name: external-dns
        namespace: kube-system
      wellKnownPolicies:
        externalDNS: true

managedNodeGroups:
  - name: ng-1
    instanceType: t3.small
    desiredCapacity: 2

This will create a cluster named “my-cluster”, with a managed node group containing two t3.small nodes. (There are also unmanaged node groups, although that’s a bit beyond the scope of this tutorial.) There’s a big “iam” section here which we’ll explain a bit later…

One thing to note here - we’re using a t3.small. This is about the smallest useful instance for running EKS. Every running pod in an EKS cluster is assigned it’s own unique IP address from your VPC. A t3.small has a maximum of 3 network interfaces, and a maximum of 4 IPv4 addresses per interface. This means a single t3.small can run at most 12 pods - if you try to run more than that, new pods will not be able to be scheduled. A t3.micro has a particularly small pool here, as it will only be able to run four pods in total. EKS actually creates some pods behind-the-scenes on each node using a “DaemonSet”, so a t3.micro will run out of IPs very quickly, and is not really appropriate for using with EKS.

We can bring this cluster up with:

eksctl create cluster -f cluster.yaml

This will take quite a while - go get yourself a coffee. This does a lot of things including creating a VPC for your new cluster to run in, the EKS cluster itself, the EC2 nodes to run your cluster, security groups, an Internet gateway, and much more. Not only does it have a lot of things to create, it’s based on CloudFormation which is unfortunately not know for being particularly fast.

When it’s eventually done, it will write credentials to access your cluster to “~/.kube/config”, and you should be able to run kubectl commands. For example, you should be able to see a list of all nodes in your cluster with:

$ kubectl get nodes
NAME                                           STATUS   ROLES    AGE    VERSION
ip-192-168-61-124.us-west-1.compute.internal   Ready    <none>   140m   v1.20.7-eks-135321
ip-192-168-87-194.us-west-1.compute.internal   Ready    <none>   140m   v1.20.7-eks-135321

When you’re done playing with the cluster, you can destroy it (and everything running on it) with:

eksctl delete cluster -f cluster.yaml

Be careful with this command - it goes without saying that deleting your production cluster by accident is not a great way to impress your boss and win customers.

What happened to minikube?

If you started this tutorial in part one, you noticed that kubectl was controlling your minikube cluster, but now it’s controlling your EKS cluster! What happened to minikube?

If you have a look in ~/.kube/config, you’ll see that kubectl now knows about two different clusters, and has authentication credentials for both of them. Kubectl has a concept called a “context” - at any given point you are in a particular context, controlling a particular cluster. You can see a list of contexts:

$ kubectl config get-contexts
CURRENT   NAME               CLUSTER      AUTHINFO           NAMESPACE
*         jason@my-cluster   my-cluster   jason@my-cluster
          minikube           minikube     minikube           default

The “*” here shows which one is the current active context, and you can switch the current context with kubectl config use-context <context-name>.

If you switch to a new laptop or want to manage an EKS cluster created by another user, you can use this AWS command to “login” to an existing EKS cluster:

$ aws eks --region [region] update-kubeconfig --name [cluster_name] --alias [cluster_name]

If you tried switching contexts, switch back to the context for “my-cluster” now.

Security - IAM and ServiceAccounts

So now we have an EKS cluster, we’re ready to create an Ingres… Well, not quite. Before we can create an Ingress, we have to install “AWS load balancer controller” to create ALBs for us, and before we can do that, we have to talk a little about security. AWS load balancer controller is actually going to run on Pods inside our cluster. This means somehow our cluster needs to have permission to create and update load balancers in AWS. In the early days of EKS, the easiest way to do this would have been to add an IAM role that allows modifying ALBs to every node in the cluster. eksctl even automates this, so in your cluster.yaml file, you would have created a nodegroup like this:

managedNodeGroups:
  - name: ng-not-very-secure
    instanceType: t3.micro
    desiredCapacity: 2
    iam:
      withAddonPolicies:
        albIngress: true # Add permission to create ALBs

But this assigns an IAM role to the entire node - this means any workload running on the node could theoretically create or modify an ALB. This is potentially a bit of a security problem; if someone finds a security hole in one of your services and figures out how to run commands in that Pod, they could potentially cause quite a bit of damage.

There is a resource we haven’t talked about yet called a “ServiceAccount”. A ServiceAccount is to Kubernetes a bit like a “user” is to a typical UNIX system. Every pod is associated with a ServiceAccount, and the ServiceAccount has various permissions and roles that affect what that pod can and can’t do. For example, we might want to make it so a particular Pod is allowed to enumerate a list of running services via the Kubernetes API, or to create other new pods - Pods can’t do either of those by default, but if you create a ServiceAccount and a ServiceRole, and associate that ServiceAccount to your pod, you could give the pod permission to do these sorts of things. If you’re interested in learning more about this, go read up on “Kubernetes RBAC”.

We’ve already created some pods - what ServiceAccount are they associated with? We can see a list of ServiceAccounts on the cluster with:

$ kubectl get serviceaccount
NAME      SECRETS   AGE
default   1         161m

We’re not interested in giving pods access to the Kubernetes API though - what we’d really like to do here is to somehow assign an IAM role to an individual Service Account, and then attach that Service Account to our load balancer controller’s pod. This is exactly what the imaginatively named “IAM Roles for Service Accounts”, or IRSA, lets us do. IRSA is based on OpenID Connect (OIDC); we create an OIDC provider for our cluster, then roles are constructed with a reference to that OIDC provider and to the Service Account it will be bound to. The Service Account has an annotation added to it that links back to the IAM role. IRSA, OIDC; this a lot of acronyms, and the process of setting this all up manually is a bit involved, but fortunately eksctl automates all of this fot us. This is what that long “iam” section in the cluster.yaml file we glossed over does:

iam:
  withOIDC: true
  serviceAccounts: ...

If you left that bit out of your cluster.yaml file, you can still set this up yourself with eksctl. You can also see this example from eksctl to see some other commonly configured service accounts. Once that’s done, you can install the controller with:

$ helm repo add eks https://aws.github.io/eks-charts && \
  kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master" && \
  helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller \
   -n kube-system \
   --set clusterName=my-cluster \
   --set serviceAccount.create=false \
   --set serviceAccount.name=aws-load-balancer-controller

A quick aside here about helm - we just ran some Helm commands. We’re going to talk about it in more detail in an upcoming tutorial, but for now you can think about Helm as a bit like a “pacakge manager” for Kubernetes. That helm upgrade --install command we ran generated a bunch of YAML files and kubectl applyed them to our cluster for us. Also, notice that we specified our clusterName here to Helm as a parameter, along with the serviceAccount.name we want the pods to use. This particular Helm chart needed the cluster name to use in the generated YAML.

Ingress

We’re finally ready to create an ingress! Let’s update the 2048.yaml file we created in part one:

# 2048.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game
  labels:
    app: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: game
  template:
    metadata:
      labels:
        app: game
    spec:
      containers:
        - name: web
          image: alexwhen/docker-2048:latest
          ports:
            - name: http
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: gameservice
spec:
  selector:
    app: game
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      targetPort: http
      name: http
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gameingress
  annotations:
    kubernetes.io/ingress.class: "alb"
    alb.ingress.kubernetes.io/scheme: "internet-facing"
spec:
  rules:
      http:
        paths:
         - path: /
           backend:
             serviceName: gameservice
             servicePort: http

If you followed along from part one of the tutorial, the Deployment and Service should be familiar. We did make a small change here - the Service used to be type: ClusterIP, which meant the service was given an IP address that could only be reached from inside the cluster. Instead we’ve now specified type: NodePort - this means that a port will be opened on each node in the cluster to allow access to this service from outside the cluster. This is required to work with the AWS load balancer controller.

The “Ingress” at the bottom is the new part. The metadata for the ingress contains an annotation: kubernetes.io/ingress.class: "alb". Annotations are basically free-form strings, like labels, but they’re used to set “extra” configuration on a resource - configuration that’s used by some Kubernetes controller or operator, but which isn’t part of the official spec. You could have multiple different kinds of ingress controllers configured on the same Kubernetes cluster, so the AWS load balance controller, by default, will ignore any Ingress resources that don’t have this annotation. Put another way, this annotation marks this Ingress as one that the AWS load balancer controller controller should pay attention to. You can see a list of all the annotations that the AWS Load Balancer Controller supports in the documentation.

The spec part is where things get interesting - we have a set of rules for inbound traffic, and within that we have rules for http (and HTTPS) traffic. Within the http block, we have a set of paths. In our case we just have the one: all traffic to any path that starts with “/” will be forwarded to the “gameservice”. You can specify multiple paths here though - you could forward all traffic to “/users” to the user service, and all traffic to “/messages” to the message service; this is how you would split your backend up into multiple microservices.

Once you’ve updated the “2048.yaml” file, apply it, and then if everything has gone according to plan you should have a new ingress:

$ apply -f 2048.yaml
deployment.apps/game unchanged
service/gameservice unchanged
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/gameingress created

$ kubectl get ingress
NAME          CLASS    HOSTS   ADDRESS                                                             PORTS   AGE
gameingress   <none>   *       internal-k8s-default-gameingr-xxx-yyy.us-west-1.elb.amazonaws.com   80      2m33s

It can take a few minutes for AWS to actually provision the ALB, and the “address” will be shown long before that happens. Log into the EC2 console for AWS, go to “Load Balancers”, and find your ALB - when the “State” switches from “Provisioning” to “Active”, it’s ready to go. At this point you should be able to visit the URL in a browser, and see your service running. Note that you need to visit the “http://” version, not the “https://” version, because we haven’t set up HTTPS yet.

You may also notice Kubernetes complain that extensions/v1beta1 is deprecated and will be removed in v1.22+. At some point in the near future, this ingress example will change to look like the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gameingress
  annotations:
    kubernetes.io/ingress.class: "alb"
    alb.ingress.kubernetes.io/scheme: "internet-facing"
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gameservice
                port:
                  name: http

However, at the time of this writing the AWS load balancer controller does not support the new networking.k8s.io/v1 API, so we have to use the older format for the ingress.

Namespaces and Logs

I mentioned that when we installed the AWS load balancer controller, it would run in pods in our cluster. But, you’ll notice if you run kubectl get pods, you can’t see them:

$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
game-77878d75db-svmgd   0/1     Pending   0          88s
game-77878d75db-t8hdh   0/1     Pending   0          88s

What gives? Where are our pods? The reason you can’t see them is because of something called a “namespace”. Namespaces let you divide up your cluster into “virtual” clusters. Whenever we create a resource, we can specify which namespace it will be created in via the metadata.namespace field. Our AWS load balancer controller is running in a special namespace called the “kube-system” namespace.

We can list all the namespaces in our cluster:

$ kubectl get namespaces
NAME              STATUS   AGE
default           Active   146m
kube-node-lease   Active   146m
kube-public       Active   146m
kube-system       Active   146m

Any resource we create, where we don’t explicitly specify a namespace, will end up in the “default” namespace. We can list pods in the “kube-system” namespace like so:

$ kubectl --namespace kube-system get pods
NAME                                            READY   STATUS    RESTARTS   AGE
aws-load-balancer-controller-66d949fb97-cbbr4   1/1     Running   0          95m
aws-load-balancer-controller-66d949fb97-lbmn8   1/1     Running   0          95m
aws-node-7bn2r                                  1/1     Running   0          128m
aws-node-kskx9                                  1/1     Running   0          128m
coredns-546f4f657c-4t9cl                        1/1     Running   0          141m
coredns-546f4f657c-v7ckd                        1/1     Running   0          141m
kube-proxy-wff9v                                1/1     Running   0          128m
kube-proxy-x544f                                1/1     Running   0          128m

And there are our load balancer pods. You can replace --namespace with -n to save some typing. If something is going wrong, and the pod is not running, we can examine the state of the pod with:

$ kubectl -n kube-system describe pod aws-load-balancer-controller-66d949fb97-lbmn8

Frequently in the “events” section at the bottom of “describe”, you’ll be able to see what’s preventing the pod from starting. If the pod is starting but isn’t doing what you expect, or is crashing, we can get logs from each of these pods:

$ kubectl -n kube-system logs aws-load-balancer-controller-66d949fb97-lbmn8

Since there’s a pair of them, you’ll have to figure out which one is currently the “leader” by looking through the logs - it’s the one doing the actual work, the other once is just there to take over in case the first one fails.

If you spend a lot of time working in a particular namespace, it can be tedious to keep typing “–namespace” over and over again. Back when we were looking at contexts, to control which cluster we were interacting with, you may have noticed that in the list of contexts there was a column called “namespace”. We can set the default namespace for a context:

$ kubectl config set-context minikube-dev \
  --namespace=dev --cluster=minikube --user=minikube

If you run the above command and use the “minikube-dev” context, then when you issue any command, it will operate the “dev” namespace by default.

DNS and HTTPS

We can now access our service through the ALB, but it would be nice if we could access the service through a domain name. You could manually configure CNAME records to point to the ALB, but it would be nice if we could get Kubenetes to automatically update our DNS records every time we bring up or update an ingress. The tool we’re going to use to do this is a service called external-dns, which like the AWS Load Balancer Controller, we’re going to run on our cluster. In order to do this, first you’ll have to register a domain name. external-dns supports many DNS registries, but for the purposes of this tutorial we’re going all in on AWS, and we’re going to use Route 53.

If you already have a domain name and hosted zone, you can use them - we’ll add a new subdomain and won’t touch any of the existing records. Otherwise you’ll need to log into the AWS management console, and go to Route53, go to “Registered domains” and register a new domain name, then go to “Hosted Zones” and create a new zone. Let’s pretend you’ve registered “thedreaming.org” as your domain name.

To install external-dns, you can follow along with the official installation instructions for external-dns, but Bitnami has a nice Helm chart that will install external-dns for us, so for simplicity’s sake we’ll use it here:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm upgrade --install external-dns --dry-run \
  --namespace kube-system \
  --set provider=aws \
  --set aws.zoneType=public \
  --set serviceAccount.create=false \
  --set serviceAccount.name=external-dns \
  --set txtOwnerId=UNIQUE_ID_HERE \
  --set "domainFilters[0]"=DOMAIN_NAME \
  bitnami/external-dns

Replace UNIQUE_ID_HERE with some unique ID that won’t change over the life of your cluster (the cluster name, for example) - external-dns will set this on each record it creates so it can figure out which domain names it “owns”. If you have multiple clusters that all create records in the same hosted zone, make sure the txtOwnerId is different in each cluster, otherwise they may clobber each other’s records. Replace DOMAIN_NAME with the name of your domain (e.g. “thedreaming.org”) - this will get external-dns to only modify records for this domain name and no others. You can use domainFilters in a case where you’re being extra paranoid - let’s say you work for “bigcorp.com”, and you want to make extra sure no one accidentally reconfigures the DNS records for “www.bigcorp.com” - you could create a “k8s.bigcorp.com” subdomain and set this to the “domainFilters”. The external-dns would be able to update “k8s.bigcorp.com” and any subdomain of “k8s.bigcorp.com”, but would refuse to try to setup anything outside that subdomain.

You can run:

$ kubectl --namespace=kube-system get pods --selector "app.kubernetes.io/name=external-dns"

to verify the pod is up and running (note the use of a selector to get specific pods).

After we have external-dns setup, let’s create a certificate for our service. If you really want to automate absolutely everything, you can use cert-manager for this - a service that runs on your cluster and automatically fetches certificates via LetsEncrypt. But, if you don’t mind manually creating the certificate, you can create the certificate via AWS Certificate Manager. If you have no certificates provisioned, click on “Get started” under “Provision certificates”, otherwise click on “Request a certificate”. From there:

Choose “Request a public certificate” and click on the “Request a certificate” button.
Enter your domain name (e.g. “thedreaming.org”) as the domain, click “Add another name to this certificate” and then enter your domain again prefixed with “*.” to make a wildcard certificate (e.g. “*.thedreaming.org”.) Click “Next”.
Choose “DNS Validation”, and then click “Next”, then “Review”.
If everything looks good, click “Confirm and request”.
You now have to “validate” that you own these domains. You should see a screen with “Pending validation” next to both entries. Since both are for the same top-level domain, when you validate one it should also validate the other. Open one of these up and it will give you instructions for validating the domain name, although if your domain is on Route 53 there should be a “Create record in Route 53”. Once the record has been created, click “Continue” (even though validation is still pending). This will take you to your list of certificates, and you can refresh the page until validation is complete.

Once you’ve created your certificate, of if you already have a certificate for the domain name in question, take note of the certificate ARN (something like “arn:aws:acm:us-west-1:XXXX:certificate/YYYY”). Now we can update our ingress. Replace the ingress in 2048.yaml with this updated version:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gameservice
  annotations:
    kubernetes.io/ingress.class: "alb"
    alb.ingress.kubernetes.io/scheme: "internet-facing"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:us-west-1:150...:certificate/277..."
spec:
  rules:
    - host: 2048.thedreaming.org
      http:
        paths:
         # The ALB will only match this path if we come in via HTTP.
         - path: /*
           backend:
             serviceName: ssl-redirect
             servicePort: use-annotation
         - path: /*
           backend:
             serviceName: gameservice
             servicePort: http

Replace the “certificate-arn” with your certificate-arn, and “thedreaming.org” with your domain name. Notice here we specify the port to redirect to for HTTPS traffic via the “alb.ingress.kubernetes.io/ssl-redirect”. We have to specify this as a string: “443”, instead of as a bare number. This is because annotations can only have string values in Kubernetes.

This ingress looks a little strange, because it has two paths for “/*”. There’s some funny-business going on here in the controller, which you can read about in more detail at the bottom of this page, but the short version is that the first one will only match when we come in via HTTP, and will be ignored for HTTPS (because it would cause an infinite loop).

Then, apply our 2048.yaml file again:

$ kubectl apply -f 2048.yaml
deployment.apps/game unchanged
service/gameservice unchanged
ingress.extensions/gameservice configured

In the AWS management console, you should see a new A record has been created for our domain name, and you should now be able to visit that domain and get to the HTTPS version of the site.

Conclusion

In part two of this tutorial we learned how to bring up a cluster in AWS, we learned a bit about security, and we learned about how to set up an Ingress to bring traffic into our cluster. In part three we’re going to look at some important concepts like volumes, stateful sets, and autoscaling.

Kubernetes for Developers - Part 1: Introduction to Kubernetes

Wed, 20 Oct 2021 00:00:00 +0000

This series is a practical introduction to Kubernetes and Amazon’s EKS. I come from the developer side of things, rather than the ops side of things, so this is written from that perspective. In this first tutorial, we’re going to discuss some basic Kubernetes concepts and play around with a local cluster running in a VM. This tutorial assumes you are already somewhat familiar with Docker - if you know how to write a Dockerfile, and run an application in Docker, (or at least understand what those concepts are) you should be good.

Scheduling Workloads

Kubernetes (often abbreviated “K8s”, in the same way that “internationalization” is often abbreviated to “i18n” - developers are a lazy bunch), is a system that lets you run “workloads” (i.e. docker containers) on a cluster of nodes. The act of deciding what node to run a workload on is known as “scheduling” a workload. What Kubernetes provides you is a standardized framework for creating workloads, for service discovery (figuring out how to connect to those workloads internal), for exposing services to the Internet, and provides a powerful API for inspecting what workloads are running and interacting with them.

The Kubernetes mantra is “cattle, not pets”. As a rule (with some exceptions we’ll talk about in future tutorials), you don’t care which nodes a given workload is scheduled on. If a workload dies, then we simply dispose of it and create a new copy. If a Kubernetes node needs to be upgraded, we create a new node with the upgrade, then simply dispose of the old node, and let Kubernetes handle moving workloads off the old node to the new one. Kubernetes provides lots of functionality to make sure this sort of thing goes smoothly and the way you want it to, but without having to kill off individual workloads one-by-one - Kubernetes will take care of moving workloads around in a graceful manner.

Creating a Cluster

We can’t do much with Kubernetes without a cluster of nodes to run containers on, so let’s create a cluster. There are a few ways you can do this for free. For this tutorial, we’re going to use minikube, which will set up an entire Kubernetes cluster in a VM on your local machine. This is a great way to play around with Kubernetes. (I’d also point you towards “k3s” which is a great solution for running a small Kubernetes cluster on your own hardware, such as on a Raspberry Pi.)

To get minikube running, you need a few tools installed first; kubectl to talk to Kubernetes, and of course minikube. If you have Docker installed, you may already have kubectl installed, as it often comes bundled. On a Mac, if you’re using brew, you can install both of these and start minikube with:

$ brew install kubectl minikube
$ minikube start --vm-driver=hyperkit

If you’re on another platform, you’ll need to install a hypervisor such as VirtualBox first. You can get detailed instructions from the Kubernetes documentation.

Once minikube is installed and running, you can run kubectl to interact with your Kubernetes cluster. For example, you can list all the nodes in your cluster with kubectl get nodes:

$ kubectl get nodes
NAME       STATUS   ROLES                  AGE    VERSION
minikube   Ready    control-plane,master   2m7s   v1.22.2

You can stop your minikube VM with minikube stop, or delete the VM entirely with minikube delete.

Pods

The basic workload unit in Kubernetes is a Pod - you can think of a Pod like a running Docker container. (That’s a little bit of a lie, but it’s close enough for this tutorial.) You can create Pods directly in Kubernetes if you wish, but you rarely would - instead you would create a Deployment or a StatefulSet, which would in turn create Pods, and also take care of restarting those pods if they get destroyed, manage upgrading Pods, and so on.

But, let’s create a pod, just to get some experience with kubectl. There are two ways we can go about doing this. We can create a pod directly from kubectl with kubectl run:

$ kubectl run my-pod --image=alexwhen/docker-2048:latest
pod/my-pod created

This is very similar to the docker run command - it starts up a pod for us. There are times when this is handy, but creating resources manually is error prone and not reproducible. More typically what we do with Kubernetes is create a YAML file that describes what we want deployed, then run kubectl apply to “apply” that YAML file - to create resources that are in the YAML file but not on the node, or to update resources that already exist. This way we can commit our YAML file to git or some other source control, and we have a reproducible set of instructions for what we want to deploy. Here’s a sample YAML file for creating a Pod:

# pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-pod-2
  labels:
    env: test
spec:
  containers:
    - image: "alexwhen/docker-2048:latest"
      name: docker-2048
      ports:
        - name: http
          containerPort: 80

All YAML descriptors in Kubernetes have an apiVersion, kind, metadata, and spec section.

The apiVersion and kind here tell us we want to create a “Pod”; each kind of resource in Kubernetes is associated with a particular apiVersion.
metadata contains data about this resource - in the case the name of the pod, and some labels.
spec describes the details of what we’re trying to create - here we specify the pod has one container, where to find the image for the container, and a list of ports that the container exposes.

In the “metadata” section, “labels” are free-form metadata you can attach to any object in Kubernetes - you can set whatever labels you like on a resource. You might specify an “app: front-end” to specify this is the front-end part of your application, or you might specify that this deployment is part of “env: prod”. It’s entirely up to you. You can use “selectors” in Kubernetes to pick a subset of resources - to find all the pods that are part of “env: prod” for example. Frequently the metadata section will also contain “annotations” - we’ll talk more about those in the next part of this tutorial when we talk about “Ingress”.

Once we write this to pod.yaml, we can create this pod with:

$ kubectl apply -f pod.yaml
pod/my-pod-2 created

We can see what pods exist with:

$ kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
my-pod     1/1     Running   0          5m26s
my-pod-2   1/1     Running   0          16s

You may see your pod is in the “ContainerCreating” state for a little bit while Kubernetes picks a node to run it on, downloads the Docker image, and starts the pod. If you run kubectl get pods --watch it will print a line every time a pod’s status changes, and you can see your pod get updated in realtime. We can see detailed information about a specific pod by name, including a little bit of it’s history, with:

$ kubectl describe pod my-pod

Or get a YAML description of the pod via:

$ kubectl get pod -o json my-pod

Note that the YAML description of a pod includes all the things we added in our YAML file, along with some defaults we didn’t set, and some current status from Kubernetes. We can destroy pods, either destroying the pod we created manually:

$ kubectl delete pod my-pod
pod "my-pod" deleted

Or from the YAML file:

$ kubectl delete -f pod.yaml
pod "my-pod-2" deleted

But you’ll notice if we delete a pod, it doesn’t get recreated. Isn’t this is the whole point of Kubernetes? It’s supposed to create and manage pods for us! We’ll see how to get Kubernetes to create a collection of pods for us by creating a “Deployment” in just a moment.

kubectl - a RESTful API

A little bit of an aside, but you’ll notice right away that kubectl is very “REST-like”. You can get/describe pods, delete pods, and apply changes. You may notice an easy 1:1 mapping between these and the GET, DELETE, POST, and PATCH commands you’d find in REST, and in fact the Kubernetes API is entirely REST based under-the-hood.

These “verbs” - get, describe, delete - apply to pods and to a number of other “nouns” or “resources” in Kubernetes. You can get a full list of available resource types with kubectl api-resources - if you give it a try, it’s a long list. Kubernetes is extensible, so if there’s a resource that’s missing from that list, you can even add your own through “operators”. The learning curve in Kubernetes is largely about learning what these various resources are and how to configure them.

kubectl comes with a handy help function here in the form of kubectl explain. If you forget what a pod is, you can run kubectl explain pod, and it will give you a description of the pod, and all the fields that can appear on a pod and what they mean. If you want to know what the “image” field is for, you can run kubectl explain pod.spec.containers.image, and it will tell you all about that field.

Deployments

Back to pods. Let’s say you want to run a stateless service of some sort - an API layer running on node.js or Java, or perhaps a web server. Describing these as “stateless” just means we can run one pod by itself, or ten pods in parallel. They can run on different machines (so long as they have some kind of load balancer in front of them). If one dies, we can replace it with another pod, possibly running on a different node. Kubernetes manages this kind of workload with a “deployment”.

Once we launch a deployment, Kubernetes takes care of making sure the specified number of pods are running, restarts pods if they fail, reschedules pods to a new node if a node fails. We can tell Kubernetes how much CPU and RAM a pod needs at minimum and how much it can burst to, and Kubernetes will make sure we don’t put too many instances of a service on a single node, and will limit the CPU and RAM used. Kubernetes will even take care of rolling upgrades when we want to deploy a new version of a workload. We’re getting ahead of ourselves though. Let’s describe a simple deployment in a YAML file:

# 2048.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game
  labels:
    app: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: game
  template:
    metadata:
      labels:
        app: game
    spec:
      containers:
        - name: web
          image: alexwhen/docker-2048:latest
          ports:
            - name: http
              containerPort: 80

I know what you’re thinking; “What is with this giant wall of YAML? Am I sure I want to learn Kubernetes? What life decisions lad me to this point? Maybe I should give up on DevOps and open a gluten free bakery…”

This is definitely intimidating the first time you see it, but let’s go through this piece by piece. The top level of any YAML document in Kubernetes looks much the same. Just like with our Pod above, the combination of apiVersion and kind tells Kubernetes that we’re trying to create a Deployment. Unlike Pod, which was part of the “v1” apiVersion, a Deployment is part of “apps/v1”. metadata here is just like it was for the pod, but here it gives a name and labels for the deployment itself. spec is the part of any Kubernetes document that differs, depending on what we’re creating.

The keen-eyed amongst you will have noticed that we have two metadata sections - one at the top level, and one inside spec.template. The template describes the YAML for each pod in this deployment, so the metadata inside the template is metadata we’re going to apply to each pod (as opposed to the metadata at the top level, which is for the actual “Deployment” object). Here, each pod will get a app: game label. The template also specifies the spec for each pod, describing what image to fetch, what ports to expose on each pod, and so on.

spec.replicas tells us how many copies of this pod we want to create. spec.selector is a “label selector”, which tells us how to figure out if the pods for this deployment are already running - the selector says we’re going to select pods that have the label app: game, so if there are already two pods running with those labels, and they’re configured to match the spec, then we don’t have to change them. In any deployment you write, spec.selector must be a subset of spec.template.metadata. Often you apply the same labels to the deployment that you do to the pod, so often these labels are repeated three times in this file.

If we put this all in a file called “2048.yaml”, and run kubectl apply -f 2048.yaml, we should see a single Deployment, and two Pods get created (and also a “ReplicaSet” - this is responsible for making sure the correct number of pods are running):

$ kubectl get deployments
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
game   2/2     2            2           6s

$ kubectl get replicasets
NAME              DESIRED   CURRENT   READY   AGE
game-6bcc6885f8   2         2         2       10s

$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
game-6bcc6885f8-fplx6   1/1     Running   0          14s
game-6bcc6885f8-x6rnr   1/1     Running   0          14s

Now, let’s try deleting one of these pods:

$ kubectl delete pod game-6bcc6885f8-fplx6

$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
game-6bcc6885f8-q239n   1/1     Running   0          5s
game-6bcc6885f8-x6rnr   1/1     Running   0          1m10s

We deleted the pod, and Kubernetes created a new one for us, since we told it we always want there to be two!

Note that if a new version of our application comes out and we want to upgrade, all we need to do is update the image in the above YAML file and apply it again - Kubernetes will perform a rolling upgrade, adding new pods running the new version, then destroying pods running the old version, until we’re back at two pods all running the new version. If you don’t want a rolling update and instead want to terminate all the old pods and then bring up new pods, you can do that too, by setting spec.strategy.type to “Recreate” - this will cause all the old pods to be terminated and then new pods to be created to replace them. “Recreate” will will incur a brief service outage, since there will be a very brief time when your old application has been terminated, and your new application is still starting up.

Now we have pods running, and they have a web server that each of them is exposing on their own port 80. How do we access that web server? We can access an individual pod using the kubernetes port-forward command:

$ kubectl port-forward game-6bcc6885f8-q239n 3000:80
Forwarding from 127.0.0.1:3000 -> 80
Forwarding from [::1]:3000 -> 80

Now if you open up a web browser and visit http://localhost:3000, you should see the game being served from the pod. But, this isn’t very practical - I want to load balance traffic between these two pods. How do I do that, and how do I either find this pod from another pod inside Kubernetes, or access it from the Internet?

StatefulSets and DaemonSets

I wanted to briefly mention StatefulSets and DaemonSets here, although we will look into these in greater detail in another article.

Deployments are appropriate for stateless services like backend services, but they’re not appropriate for all services. If you wanted to run a MongoDB database in a replica set, for example, you’d always want exactly the same number of pods running at all times. You’d want each pod to have some kind of “volume” to read and write data to, and that volume would have to follow the pod around from node to node if the workload is rescheduled. The resource you’d want to create in this situation is called a “StatefulSet” - every pod in a StatefulSet gets a unique name (e.g. “postgres-1”, “postgres-2”, …), and you can create a “PersistentVolumeClaim” for each pod in the set.

The other, less often used, resource is the DaemonSet - if you create a DaemonSet, then Kubernetes will ensure there is one pod running for the DaemonSet on every node in your cluster. These are useful for collecting metrics about nodes in the cluster. kube-proxy, a service which provides simple TCP and UDP forwarding within the cluster, runs as a DaemonSet on each node.

Services

A Service is another kind of Kubernetes resource - as the name implies, it describes the actual service that a group of pods exposes. It’s rare you create a deployment without also creating a service. Let’s update our 2048.yaml file so it looks like this:

# 2048.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game
  labels:
    app: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: game
  template:
    metadata:
      labels:
        app: game
    spec:
      containers:
        - name: web
          image: alexwhen/docker-2048:latest
          ports:
            - name: http
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: gameservice
spec:
  selector:
    app: game
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 80
      targetPort: http
      name: http

And then run kubectl apply -f 2048.yaml again. This will create a new “service” object which describes how to connect to our service:

$ kubectl get service gameservice
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
gameservice   ClusterIP   10.100.212.52   <none>        80/TCP    4s

Looking at the spec, again we see a selector describing which pods we want this service to connect to. The type: ClusterIP means that this service will be assigned an IP address internally - anyone who connects to port 80 of this service will have their connection sent to one of the pods running this service, to the “http” port (which we defined above in the deployment).

We can connect to this service in a couple of ways - first we can use kubectl port-forward svc/gameservice 3000:http, just like we did above, but now when we connect to http://localhost:3000, our connection will be forwarded to one of the pods at random.

From inside the cluster, we can find this service with a DNS query. Let’s get a shell in an ubuntu image running inside the cluster, and see if we can find the service. First we’ll install curl, and then we’ll use curl to try to grab the HTML contents of the web page:

$ kubectl run -it --rm --image=ubuntu:latest -- bash
root@bash:/# apt-get update && \
  apt-get install -y -qq curl
root@bash:/# curl http://gameservice/

  # or

root@bash:/# curl http://gameservice.default.svc.cluster.local/

This provides us with a powerful way to link our applications together - each application can find services it wants to interact with just by looking up the service via DNS. Note that we can connect to the service via a fully qualified domain name like “gameservice.default.svc.cluster.local” (“default” here is the namespace we are currently running it - we’ll talk more about namespaces in part two), or we can just use the “gameservice” domain name to find the one running in the local namespace. This works because Kubernetes automatically configures a search path in /etc/resolv.conf.

We can also create services for resources that aren’t in Kubernetes:

apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: prod
spec:
  type: ExternalName
  externalName: postgres.example.com

This is called an “external service”, and here if we try to look up the DNS entry for “postgres”, we’ll get back a CNAME record for “postgres.example.com”.

Logs and Shells

If our pod’s container writes output to stdout, sometimes it’s handy to see the output. Much like the docker logs command, kubectl provides a kubectl logs command that will show logs for a pod:

kubectl logs [podname]

Our “2048” pods don’t print any output, so the logs are empty. Let’s start a redis container so we can see some logs:

$ kubectl run redis --image=redis:latest
pod/redis created

$ kubectl logs redis
1:C 20 Oct 2021 19:57:03.787 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 20 Oct 2021 19:57:03.787 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 20 Oct 2021 19:57:03.787 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 20 Oct 2021 19:57:03.788 * monotonic clock: POSIX clock_gettime
...

If we want to watch logs in realtime, then similar to the tail -f command, we can use the “-f” flag to tail the logs. The “–since” will limit the output to recent output. So this command will print all logs from the last 10 minutes, then keep showing logs as they come in:

$ kubectl logs -f --since=10m redis
...

The --previous flag is also handy - this shows logs from a previous instance of pod. Handy when a pod crashes or restarts, and you want to see the error that caused it to die. “fluentd” is a popular log aggregator that can fetch logs from all your deployments and collect them together for you.

One last thing that’s handy for troubleshooting an existing pod; you can get a shell on an existing pod with kubectl exec:

$ kubectl exec -it redis -- sh
#

Conclusion

That’s it for part one of this tutorial - we learned about kubectl, we learned how to create pods, deployments, and services. We learned how to get a shell inside the cluster, and we learned about service discovery through DNS. In part two, we’ll look at how to create a cluster in AWS using eksctl and how to forward traffic from the Internet to one or more of our services via an “Ingress”.

An immutable version of MobX: SoloX

Tue, 10 Aug 2021 00:00:00 +0000

In this post, we’re going to develop a state management library for React, which is similar in many ways to the popular MobX library, but which uses immutable data structures. The upside is that this library is increadibly small and easy to use, and you don’t need to worry about wrapping your React components with observer higher-order components. The biggest downside is that you need to be a bit more explicit about when you want to update state, but thanks to the immer library, updates will still be done as if the state is a regular mutable javascript object.

State Management

Managing state in a React application is one of the most important parts of the application, but is often overlooked. One rule of thumb when dealing with state is to keep the state as close as you can to the component that needs it. When you have a very simple react component, you can store some state about that component using state or using the useState() hook:

export const Toggle: React.FC<unknown> () {
  const [ toggled, setToggled ] = useState(false);
  return <button onClick={() => setToggled(!toggled)}>{ toggled ? 'On' : 'Off' }</button>
}

If you have some state that needs to be shared between two components, you can store that state in a common ancestor, and then pass it down to each component as props.

When we start getting into more complicated state, with a lot of business logic governing how that state is updated, this can quickly get out of hand. One widely used solution to this problem is to move state into a state management library like redux or zustand. These libraries typically treat state as a “global” thing, with a single global store storing state for the whole application. This violates our “keep state as close as you can to where you use it” principal. For all but the most complicated apps, libraries like these can frequently be overkill, and some (cough redux) introduce significant boilerplate.

Sometimes, we want something inbetween - something self contained that manages state and actions, but which isn’t global. In the MVC paradigm, this would be the “model” and the “controller”. You can come up with a “controller” component with a bunch of useState()s to define some state and useCallback()s to create actions to update that state. This works reasonably well, but has the serious downside that in order to unit test such a thing, you need to render the controller component, which means you need React and either need JSDom or need to run your tests in a browser.

MobX is an excellent state library which allows you to create an instance of a “store” anywhere and use it locally. If you’re starting a new project from scratch, then MobX should be on your short list for state management. One downside to MobX however is that is uses mutable state, so any components that make use of a MobX store need to be wrapped in an observer higher-order-component. In an existing project, this can cause problems, because MobX starts “infecting” your non-MobX components, and if you miss an observer somewhere you can spend time chasing down components that don’t re-render when you expect them to. What we want is something like MobX, but that uses immutable state, so we can just drop it into an existing project.

In this post, we’re going to write exactly such a library from scratch - a poor-man’s MobX, or perhaps a “MobX superlegerra”. We’ll call this library “SoloX”.

What we want is:

Some immutable state - the “model”.
Some actions that we can call from our react components that replace that state with updated state - the “controller”.
Some way to get a React component to re-render when all or part of the state updates.

Immer

We’re going to use the excellent immer library to handle the immutable part of this for us; it’s easy to use, and we will be able to write “actions” that just edit state as if it were mutable state. If you’ve never used immer before, immer has this handy produce(state, recipe) function. produce() takes an immutable state object, and a recipe function that looks like it’s mutating state directly, but actually what’s going on is that immer has created a proxy for our state, and is recording what updates we make. Immer produces a new immutable object with those edits applied. Here’s a quick example:

const oldState = { age: 7 };

const newState = produce(oldState, (state) => {
  state.age++;
});

console.log(oldState.age); // 7
console.log(newState.age); // 8

SoloX

First of all, we’re going to define a new ImmutableModelStore class, which is going to hold some state and make it easy to update it:

export class ImmutableModelStore<S> {
  /** The current state for this store. */
  public current: Immutable<S>;

  constructor(initialState: S) {
    this.current = produce(initialState, () => void 0) as Immutable<S>;
  }

  /**
   * update will synchronously update the state of the model.
   *
   * The `fn` passed in must be a synchronous function.  It will be passed a
   * mutable version of the state, which can be edited directly.  Updates will
   * be applied when `fn` returns.
   */
  public update(fn: (state: Draft<S>) => void): void {
    this.current = produce(this.current, (state: Draft<S>) => {
      fn(state);
    }) as Immutable<S>;
  }
}

This is pretty simple - in the constructor, we’re passing the initial state through immer to make it read only, and then users can call update() to update the state. Now let’s use this to create a “controller”. Borrowing from the MobX tutorial:

interface Task {
  task: string;
  completed: boolean;
  assignee: string | null;
}

interface TodosState {
  todos: Task[];
}

class TodoController {
  // Create our "model".
  public state = new ImmutableModelStore<TodosState>({ todos: [] });

  // Generate derived state from the model.
  get completedTodosCount() {
    return this.state.current.todos.filter((todo) => todo.completed === true)
      .length;
  }

  // An action to add a new todo.
  addTodo(task: string) {
    this.state.update((state) => {
      state.todos.push({
        task,
        completed: false,
        assignee: null,
      });
    });
  }
}

const todoController = new TodoController();

And now we have a controller/store with actions and computed values, just like a MobX store. But crucially, the store’s state is immutable, so we can just pass that state straight to a React component as a prop, and the component will update whenever the state changes. Well… almost.

How do we pass this to a React component? If we create a new controller inside a component, and pass the controller down as a prop, the controller itself will never change, so child components will never re-render. If we pass down controller.state.current as a prop, then the child will re-render whenever the state updates, but we need some way to trigger that “top level” component that created the controller to re-render. What we want here is a useControllerState() hook that lets us “select” state from the controller, and will trigger a re-render whenever state updates. That way a component will only re-rerender when selected values change. In order for this to work, we need some way to “subscribe” to our state. Let’s update our ImmutableModelStore class to add the ability to subscribe to changes:

export class ImmutableModelStore<S> {
  public current: Immutable<S>;
  private _subscribers: ((newValue: Immutable<S>) => void)[] = [];

  constructor(initialState: S) {
    this.current = produce(initialState, () => void 0) as Immutable<S>;
  }

  /** ... */
  public update(fn: (state: Draft<S>) => void): void {
    this.current = produce(this.current, fn) as Immutable<S>;
  }

  /**
   * Subscribe to changes in this store's state.
   *
   * @param subscriber The function to call when the state changes.
   * @returns a function to call to unsubscribe.
   */
  public subscribe(subscriber: (newState: Immutable<S>) => void): () => void {
    this._subscribers.push(subscriber);

    return () => {
      this._subscribers = this._subscribers.filter((s) => s !== subscriber);
    };
  }
}

Note this is almost identical to the class in SoloX - the “real” one has some extra error checking, and has a trick to allow re-rentrant/nested calls into update(), but otherwise it’s the same. And now we can write our hook:

export function useControllerState<S>(store: ImmutableModelStore<S>) {
  const [result, setResult] = useState<Immutable<S>>(store.current);

  useEffect(() => {
    const unsubscrube = store.subscribe(setResult);
    return () => unsubscrube(setResult);
  }, [store]);

  return result;
}

This hook uses a useControllerState() to keep track of the current state in result, and then uses a useEffect() hook to subscribe to the ImmutableModelStore and update the result as required. Let’s make this slightly fancier - it would be nice if we could “select” certain values from the state, and only re-render if they change similar to redux’s useSelector(). Doing this correctly is a bit more complicated, but here’s the full function:

/**
 * Returns a ref to the passed in object that updates whenever the passed in object
 * changes.  This is handy when you want to create a callback, and you don't want
 * the callback to change because a value it depends on changes.
 *
 * @returns [ref, changed]
 */
function useLatest<T>(thing: T): [React.MutableRefObject<T>, boolean] {
  const ref = useRef<T>(thing);
  const changed = ref.current !== thing;
  ref.current = thing;
  return [ref, changed];
}

function defaultSelector<S, R>(state: Immutable<S>): R {
  return state as any as R;
}

function defaultIsEqual<R>(oldState: R, newState: R): boolean {
  return oldState === newState;
}

export function useControllerState<S, R = Immutable<S>>(
  store: ImmutableModelStore<S>,
  selector?: (state: Immutable<S>) => R,
  isEqual?: (oldState: R, newState: R) => boolean
) {
  // Keep track of the "latest" selector and isEqual so we don't have to
  // unsubscribe/resubscribe if they change.
  const [latestSelector, selectorChanged] = useLatest(
    selector || defaultSelector
  );
  const [latestIsEqual] = useLatest(isEqual || defaultIsEqual);

  // Hold the selected result in a ref - if the selector changes, we want to update
  // the result without forcing another re-render.
  const resultRef = useRef<R>(latestSelector.current(store.current));

  // The currently selected state, used to force a re-render.
  const [, setResult] = useState<R>(() =>
    latestSelector.current(store.current)
  );

  // If the selector changes, may need to update the selected result.
  if (selectorChanged) {
    const newResult = latestSelector.current(store.current);
    if (!latestIsEqual.current(resultRef.current, newResult)) {
      resultRef.current = newResult;
    }
  }

  // Update resultRef and force a re-render if the store changes.
  useEffect(() => {
    const unsubscribe = store.subscribe((state: Immutable<S>) => {
      const newResult = latestSelector.current(state);
      if (!latestIsEqual.current(resultRef.current, newResult)) {
        resultRef.current = newResult;
        setResult(newResult);
      }
    });

    return () => unsubscribe();
  }, [latestIsEqual, latestSelector, store]);

  return resultRef.current;
}

And finally we can put this all together into a React component:

const MyComponent: React.FC<{ controller: TodoController }> = (props) => {
  const { todos } = useControllerState(controller.state);

  // Note that we need to use useControllerState to access computed values, too.
  const completedTodosCount = useControllerState(
    controller.state,
    () => controller.completedTodosCount
  );

  return (
    <div>
      <ul>
        {todos.map((todo, index) => (
          <li id={index}>{todo.task}</li>
        ))}
      </ul>
      Completed: {completedTodosCount}
    </div>
  );
};

Organizing music with Musicbrainz Picard

Sun, 22 Nov 2020 00:00:00 +0000

I’ve spent some time playing with a couple of different tools for organizing my MP3 collection. If you have a large music collection, you probably have some old songs in there you ripped from CDs, and they might have some metadata that’s not quite up-to-snuff. There’s probably some spelling mistakes in there, and maybe you have some files where the artist is “Blink 182” and some are “blink-182” - things like this. What we want is some automated way of fixing all this metadata, and sorting files into the correct folder based on their metadata. Beets is very popular program for this, but it’s not being actively maintained. MusicBrainz Picard has a nice UI and is pretty easy to use.

Backup

Before we start, we’re using some software which is going to automatically overwrite data in files and move files around on disk. This would be a really good time to make a backup of your whole music library.

MusicBrainz Picard

In Preferences, I set up the following:

Under “Metadata”, check “Translate artist names to this locale where possible” (I have some Japanese artists in my collection, such as Yoko Kanno, and while I can recognize their Japanese names in context, I wouldn’t want to try to type one into a searchbox.)

Under “Cover Art”, I enabled “Embed cover images into tags” and “Save cover images as separate files” (lots of apps like to see a “cover.jpg” file in the save folder as the album) and then I checked all the “Cover Art Providers”.

Under “Fingerprinting” I clicked on the “Get API Key…”. Most of my music was purchased on iTunes, but I have some very old MP3s from the pre-iTunes days which don’t have any metadata, so I can use Picard to look them up based on their music fingerprint, similar to how apps like SoundHound or Shazam work.

Under “File Naming” I selected “Move files when saving” and “Move additional files”. I set “Move additional files” to be “*.jpg *.png *.pdf” (so it will move album covers and liner notes when I move a track - notice this means you should not try to move a file out of Downloads directly with Picard - it will end up moving whatever random jpgs and pdfs you have in your downloads folder.) I’m going to make a full copy of my music library into a folder called “to-sort” and open this in Picard, and every time I save a file in Picard, I want it to move it to the destination directory “music” (ultimately we want this “music” directory to be the one that Navidrome reads from, so ideally this is a network share on the device we’re going to run navidrome on). I used this as my naming script:

$if($eq(%compilation%,1),Compilations,$if2(%albumartist%,%artist%,Unknown Artist))/
$if(%albumartist%,%album%/,Unknown Album/)
$if($gt(%totaldiscs%,1),%discnumber%-,)$if($and(%albumartist%,%tracknumber%),$num(%tracknumber%,2) ,)$if(%_multiartist%,%artist% - ,)%title%

You can read the MusicBrainz scripting documentation, or check out their forum for pre-built scripts (some people get a little crazy with their scripts).

There are quite a few plugins available if Picard doesn’t do something you need it to do.

Once you have Picard all set up, click “Add Folder”, and add some music from your “to-sort” folder (I prefer not to add the whole thing at once - I’ll add about 10-20 albums or so at a time). Click on “Unclusterd Files” and then the “Cluster” icon at the top to sort them into groups based on album (careful - if you have a bunch of MP3s where the album is “Unknown”, they’ll all get lumped together). Then, pick the songs or albums you want to find and click the “Lookup” button to look up your music - this will move music into albums on the right hand side.

You can click on individual tracks to see exactly what Picard is going to change. If Picard gets something wrong, you can either click the track to manually edit it, or if it gets entirely the wrong album you can search for an album in the box at the top, and then drag-and-drop tracks into the correct album. When you drag-and-drop a track, you can drag it to the album title to get Picard to figure out where it should go, or you can drag-and-drop it to the specific track in the Album. Click the album and click the “Save” icon to write the entire album to disk, and move it to the correct spot.

You can also pick unsorted tracks on the left hand side, and edit their metadata and click “Save” to write updates and move them on disc - sometimes MusicBrainz just doesn’t have the release you’re looking for.

Organizing our Music with Beets

If you want to try out Beets, this is what I did to get it set up. Using Beets requires some proficiency with the command line, and you should know how to install software like python or ffmpeg. Note that as of this writing, the latest published version of Beets doesn’t work with the latest version of Python, and since Beets hasn’t seen a new release in about a year, this means we’re not likely to see a new one soon. So, if you want to go this route, you have to install Beets from master - the very latest copy, which may have some weird bugs.

pip3 install https://github.com/beetbox/beets/tarball/master requests
# If you want to use the `lastgenre` plugin:
pip3 install lastgenre
# If you want to use the `lyrics` plugin:
pip3 install requests

Now create a file call .config/beets/config.yaml and put this in it:

# This is where your music will end up after being processed
directory: ~/music

# This is where beets will store it's db
library: ~/.config/beets/musiclibrary.db

import:
  # Copy music to ~/music.  Change to `move: yes` to move.
  copy: yes
  # Write updates to the music when it is imported.
  write: yes

# Automatically get art, lyrics, and fix genre
plugins: fetchart lyrics lastgenre

# How music will be written to disk.
# See https://beets.readthedocs.io/en/stable/reference/config.html#path-format-config
paths:
    default: $albumartist/$album%aunique{}/$track $title
    singleton: $artist/Non-Album/$title
    comp: Compilations/$album%aunique{}/$track $title

Beets will try to find cover art for albums from the Internet, but if you want a specific image for an album, you can save it as a cover.jpg file in the same folder as the album, and beets will pick it up. If you find beets gets some weird image that you weren’t expecting, you can extract the image from an MP3 file with ffmpeg:

ffmpeg -i file.mp3 cover.jpg

(There’s an open PR for beets which makes it so beets can extract cover art from your music files automatically.)

And now we can start importing music:

beet import /path/to/your/music

This can take a while, and beets is going to ask you a lot of questions. You can read the details of how to answer them here.

Setting up HTTPS on a Synology with Let's Encrypt and Route 53

Wed, 18 Nov 2020 00:00:00 +0000

The Synology now comes with a built in “Let’s Encrypt” client, but unforunately it only supports HTTP-01 challenge, which means if you want to use it you need to open up your Synology to the Internet. The Internet is a scary place, so we’re going to use the DNS-01 challenge to validate we own our domain name.

Register a Domain

We need to register a domain name, so this will cost some money annually. To generate a certificate with Let’s Encrypt, you need to “prove” you own the domain name. Usually you use an app like Certbot or Lego to do this proof for you, but it either boils down to serving a randomly generated string from your web server from a specific path, or placing that randomly generated string in a TXT DNS record. To do the first, we’d have to open a hole in our firewall, and we don’t want to do that. To do the second, we’ll have to use an API to talk to our DNS provider. My preferred DNS provider has an API, but it’s in beta and you have to jump through some hoops to get access, so let’s do this with Amazon’s AWS.

So start off by going to AWS’s “Route 53” and registering a domain name. Let’s say you get “example.com” - we’ll use this as our example domain name all through this article, but you’ll need to replace this with whatever domain name you register. Once you register it, go to the “Hosted Zones” section on the left, pick your domain, and click “Create Record”. Choose “Simple Routing” and then click “Define simple record”. Set the record name to “synology.example.com”. Under “ValueRoute traffic to” pick “IP Address or another value depending on the record type”. In the box below that, enter your synology’s IP address (e.g. 192.168.0.10). Click “Define Simple Record” then “Create records”. This creates what is called an “A Record” (short for “address record”) for “synology.example.com”. Then, find something else to do for a bit, because it can take a while for this to propagate. You can check if it’s done by running nslookup synology.example.com from your terminal until you get back an answer.

IAM User

We want to create a new IAM policy and IAM user - we don’t want to use our “root” user’s credentials in AWS, because we’re going to have to store these credentials on the synology, and if someone gets access to our synology we don’t want them to take over our entire AWS account. First, go to Route 53 again, click on “Hosted zones” and take note of the “Hosted Zone ID” for the domain in question. Now head over to IAM, click on “Policies” on the left, and then “Create policy”. Switch to the JSON tab, and copy paste the following (update your domain name in the ID and replace “YOUR-ZONE-ID-HERE” with your actual zone ID):

{
    "Version": "2012-10-17",
    "Id": "lets encrypt - example.com",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "route53:ListHostedZones",
                "route53:ListHostedZonesByName"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
               "route53:GetChange",
               "route53:ChangeResourceRecordSets",
               "route53:ListResourceRecordSets"
            ],
            "Resource": [
                "arn:aws:route53:::hostedzone/YOUR-ZONE-ID-HERE",
                "arn:aws:route53:::change/*"
            ]
        }
    ]
}

Click “Review policy”, then give you policy a name (like “lets_encrypt_example.com”), and click “Create policy”. Now click on the “Users” tab on the left, click “Add user”, give the user a name (again I used “lets_encrypt_example.com”) and check the checkbox for “Programmatic access”. Click “Next” and then click on the “Attack existing policies directly” tab, and select the policy you just created from the list (it’s easiest if you search for it by name). Copy the “Access Key ID” and “Secret Access Key”, and keep them somewhere safe. We’ll need them in a minute. Make sure you protect these! These are like a password for this user.

Setup the Synology

The Synology has an nginx instance listening on port 80 and port 443. There’s no way to easily disable this nginx service from the GUI. But we can use this nginx service ourselves. If our goal is just to provide HTTPS access to the Synology UI (and perhaps to other services running on the Synology), then basically, we’re going to generate a certificate, copy it overtop of the Synology’s nginx certificate, and then restart the nginx server, and do this automatically whenever the certificate needs to be renewed.

If, on the other hand, our goal is to expose some services running on the Synology in Docker to the public internet, we might be better off setting up something like a Traefik reverse proxy running on some port other than 443, and then we can use our router’s port forwarding feature to forward external traffic to that port. This makes it easier to prevent anyone from getting to the Synology admin interface. This guide doesn’t cover this option, but there are lots of good guides out there about how to do this.

Enable SSH Access

We’re going to do some tinkering in the command prompt (technically you could do the whole Traefik thing via the Docker UI on the Synology, but it’ll be easier via SSH). To do this, open the “Control Panel”, pick “Terminal & SNMP”, then check “Enable SSH service” and click “Apply”.

Install lego

We’re going to install lego, a go-based implementation of the ACME protocol for talking to Let’s Encrypt:

wget https://github.com/go-acme/lego/releases/download/v4.1.0/lego_v4.1.0_linux_$(dpkg --print-architecture).tar.gz
tar xvzf lego_v4.1.0_linux_$(dpkg --print-architecture).tar.gz lego
sudo mkdir -p /usr/local/sbin
sudo mv ./lego /usr/local/sbin
sudo chown root:root /usr/local/sbin/lego
# Make sure it works!
./lego --version

At this point, we’re ready to give this a try and generate a certificate. We’ll create certificates as the root user, and store them somewhere other users on the system can’t get access to them. Run:

# Become root
sudo su -
mkdir letsencrypt
chmod 700 letsencrypt
cd letsencrypt

Inside this new letsencrypt folder, create a file called config.sh. Replace these variables as required:

# config.sh
export AWS_ACCESS_KEY_ID=REPLACE_ME
export AWS_SECRET_ACCESS_KEY=REPLACE_ME
export LE_DOMAIN="example.com"
export LE_EMAIL_ADDRESS="your@email.address"

And then create a second file called lego-create.sh:

#!/usr/bin/env bash
# lego-create.sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

# Can't use default key-type ec256 with Synology.
lego --key-type rsa4096 --accept-tos \
  --path="${DIR}/.lego" \
  --dns="route53" \
  --email="${LE_EMAIL_ADDRESS}" \
  --domains="${LE_DOMAIN}" \
  --domains="*.${LE_DOMAIN}" \
   run

Then give it a run:

chmod 700 ./lego-create.sh
./lego-create.sh

This can take several minutes, so if it looks like it’s hung, just wait a little bit. When this is done, it will write a few files to ./.lego/certificates/example.com.*.

Add Certificates to Synology

If this worked, then now we have some certificates! We need to get the Synology to use them. The Synology stores certificates in /usr/syno/etc/certificate/_archive/. Each certificate in this folder is stored with a seemingly random name. There’s an INFO file here in JSON format which we might be able to edit directly, but instead let’s import our certificates via the GUI.

Download the certificates you just created to your local machine. Go to the “Control Panel”, pick “Security”, and then pick the “Certificate” tab and click “Add”. Choose “Add a new certificate”, then add a description (e.g. “example.com”) and click next, then upload the certificates you generated; the “.key” for for the “Private Key”, the “.crt” file for the “Certificate”, and the “.issuer.crt” file for the “Intermediate certificate”. Then click on the “Configure” button, and set your new ceritifcate to be the system default and the FTPS certificate. Left click on your new certificate, then right click and pick “Edit” from the context menu, and then check “Set as default certificate” (if you skip this step, you’ll have to come back here and configure any new services you create to use the certificate).

If all is successful, you should now be able to visit https://synology.example.com:5001/ without any security warnings.

Automatic renewals

We have certificates installed, but if you look in the Synology GUI, you’ll see they expire in about three months. We need to automate fetching new certificates from Let’s Encrypt, copying them into the Synology’s folder, and then restarting the nginx server.

Run the following, still as root:

cat /usr/syno/etc/certificate/_archive/INFO

And you should get some output like this:

{
   "QlWeCZ" : {
      "desc" : "",
      "services" : []
   },
   "xoEBak" : {
      "desc" : "example.com",
      "services" : [...]
   },
}

The “xoEBak” section with “example.com” in it is the Synology identifier for our ceritificate. Let’s add this to our config.sh file:

# config.sh
# ...
export SYNOLOGY_CERT_ID="xoEBak" # Replace this

Create a new script to install certificates in /root/letsencrypt called “installcerts.sh”:

#!/usr/bin/env bash
# installcerts.sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source "${DIR}/config.sh"
LEGO_FOLDER="${DIR}/.lego/certificates"
SYNOLOGY_FOLDER="/usr/syno/etc/certificate/_archive/${SYNOLOGY_CERT_ID}"

# Copy certificates
cp ${LEGO_FOLDER}/${LE_DOMAIN}.crt ${SYNOLOGY_FOLDER}/cert.pem
cp ${LEGO_FOLDER}/${LE_DOMAIN}.issuer.crt ${SYNOLOGY_FOLDER}/chain.pem
cat ${LEGO_FOLDER}/${LE_DOMAIN}.crt ${LEGO_FOLDER}/${LE_DOMAIN}.issuer.crt > ${SYNOLOGY_FOLDER}/fullchain.pem
cp ${LEGO_FOLDER}/${LE_DOMAIN}.key ${SYNOLOGY_FOLDER}/privkey.pem

# Restart the Synology nginx server
/usr/syno/etc/rc.sysv/nginx.sh reload

And then, create another new script to renew certificates, “lego-renew.sh”. This will only renew certificates if they expire in the next 20 days, and will run “installcerts.sh” only if the certificates were renewed:

#!/usr/bin/env bash
# lego-renew.sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source "${DIR}/config.sh"

# Can't use default key-type ec256 with Synology.
lego --key-type rsa4096 --accept-tos \
  --path="${DIR}/.lego" \
  --dns="route53" \
  --email="${LE_EMAIL_ADDRESS}" \
  --domains="${LE_DOMAIN}" \
  --domains="*.${LE_DOMAIN}" \
   renew --days 20 --renew-hook="${DIR}/installcerts.sh"

Let’s test out our renew script and our installcerts script:

$ chmod 700 ./*.sh
$ ./lego-renew.sh
2020/11/18 15:30:58 [example.com] The certificate expires in 89 days, the number
of days defined to perform the renewal is 20: no renewal.
$ ./installcerts.sh
alias-register stop/waiting

Finally, we need to automate renewal. In the Synology UI, go to “Control Panel”, “Task Scheduler”, click “Create” => “Scheduled Task” => “User Defined Script”. Set the task name to “Renew LE Certs”, make sure the “User” is “root”. Under the “Task Settings” tab, set the “User-defined script” to “/root/letsencrypt/lego-renew.sh”. Check the “Send run details by email” box, enter your email address, and check “Send run details only when the script terminates”, and then click “OK”.

Finishing up

Under “Control Panel” => “Network”, in the “DSM Settings” tab, check “Automatically redirect HTTP connection to HTTPs for DSM desktop. This will make it so if you go to http://synology.example.com:5000, it will redirect you to https://synology.example.com:5001.

If you want to access the DSM via port 443, under “Domain” on the same page, check “Enable customized domain”, and set the value to “synology.example.com”. Now you can login to the synology via https://synology.example.com directly. (Note if you do this that you will not be able to access it via https://synology.example.com:5001).

If you have other services running on the Synology, you can make it so you can access them by domain name. If you have a Navidrome instance running on port 4533, for example, you could add a new A Record in Route 53 for “navidrome.example.com” pointing it at the IP of your synology, and then in “Control Panel” => “Application Portal”, under the “Reverse Proxy” tab, click “Create”. Under source, set the “Protocol” to “HTTPS”, the “Hostname” to “navidrome.example.com”, and the “Port” to “443”. In the “Destination” set “Protocol” to “HTTP”, “Hostname” to “localhost”, and set the “Port” to “4533”. Now you should be able to visit https://navidrome.example.com and it will take you straight to the navidrom interface.

Ubuntu - Wake from bluetooth keyboard

Fri, 13 Nov 2020 00:00:00 +0000

I recently installed Ubuntu on an old Mac Mini, and I ran into a problem where I couldn’t get a bluetooth keyboard to wake the device (actually a Harmony Hub remote, but it presents itself as a bluetooth keyboard). I did some digging and found this offline page which describes a method for enabling wake-from-suspend for USB devices. This is an attempt at slightly simplifying this.

Quoting from that page:

Under SYSFS, a typical USB device connexion chain will ressemble something like /sys/devices/pci0000:00/0000:00:1d.0/usb5/5-0:1.0.

To be able to resume your computer with this USB device, you will need to have a wake-up power supply on every active device of the chain. In this way, even with the computer suspended, a minimum power will be supplied to the devices, so that they can send the wake-up signal.

With SYSFS, the power supply of an active device is controlled by its ./power/wakeup file. By simply writing enabled in that file, you are configuring it to get the wakeup power supply during suspended mode.

As an example, to enable wakeup power supply for the root PCI bus we need to write enabled in /sys/devices/pci0000:00/0000:00:1d.0/power/wakeup

sudo sh -c "echo enabled > /sys/devices/pci0000:00/0000:00:1d.0/power/wakeup"

So to enable a USB device to be used to resume your computer, you simply need to enable every active device on the connexion chain from the PCI bus till the device itself.

It’s as simple as that.

Find USB Devices

Your bluetooth adapter probably shows up as a USB device:

$ lsusb
Bus 002 Device 005: ID 05ac:8242 Apple, Inc. Built-in IR Receiver
Bus 002 Device 008: ID 05ac:828a Apple, Inc.
Bus 002 Device 004: ID 0a5c:4500 Broadcom Corp. BCM2046B1 USB 2.0 Hub (part of BCM2046 Bluetooth)
Bus 002 Device 003: ID 0424:2512 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
Bus 003 Device 002: ID 046d:c52b Logitech, Inc. Unifying Receiver
...

These are the devices I found on my machine. Below are two scripts which I stole from Nicolas Bernaerts’s website, since it’s down (mainly I posted them here so I can find them again if I need to). Create these two files, run sudo apt install zenity if you don’t have it installed already, and then run /usr/local/sbin/select-resume-devices from a terminal window. Note that you need to run this from the desktop because the script uses zenity to display a dialog to let you pick which devices to enable. Once you pick devices, this will write a file named /etc/udev/rules.d/90-hid-wakeup-enable.rules which will configure these devices at boot time. After you run the script, you need to reboot for changes to take effect.

In my case, I tried enabling sleep for the “Apple, Inc.” device and the BCM2046B1 Hub, and this fixed my problem. Probably I only needed to enable one of them, but I had to reboot for this change to take effect, and I’m too lazy to figure out which one I really need. :P Note that in my case, to enable wake for the bluetooth hub, I had to modify this line in select-resume-devices:

+LIST_DEVICE=(`lsusb | grep -v "0000:0000" | sed 's/^.*[0-9a-f]\:[0-9a-f]* \(.*\)$/\1/g'`)
-LIST_DEVICE=(`lsusb | grep -v "0000:0000" | grep -iv "hub" | sed 's/^.*[0-9a-f]\:[0-9a-f]* \(.*\)$/\1/g'`)

Here are the scripts:

/usr/local/sbin/enable-wakeup:

#!/bin/bash
#
# Script to enable a USB devices to be used to resume computer from sleep
#
# Depends on udevadm package
# 09/05/2012 - V1.0 by Nicolas Bernaerts

# if device has been removed, exit
[ "$ACTION" = "remove" ] && exit 0

# set PATH as it is not set for udev scripts
PATH="/usr/sbin:/usr/bin:/sbin:/bin"

# set device SYSFS path
DEVICE_SYSFS="/sys$1"
CURRENT_SYSFS=$DEVICE_SYSFS

# get device product and vendor name from parent SYSFS
DEVICE_VENDOR=`udevadm info --query=all -p "$CURRENT_SYSFS/../" | grep "ID_VENDOR_ID=" | cut -d "=" -f 2`
DEVICE_PRODUCT=`udevadm info --query=all -p "$CURRENT_SYSFS/../" | grep "ID_MODEL_ID=" | cut -d "=" -f 2`
DEVICE_LABEL=`lsusb | grep "${DEVICE_VENDOR}:${DEVICE_PRODUCT}" | sed 's/^.*[0-9a-f]\:[0-9a-f]* \(.*\)$/\1/g'`

# loop thru the SYSFS path, up to PCI bus
CARRY_ON=1
while [ $CARRY_ON -eq 1 ]
do
  # get the first three letters of current SYSFS folder
  FIRST_LETTERS=`basename $CURRENT_SYSFS | sed 's/^\(...\).*$/\1/g'`

  # if current SYSFS is PCI bus, stop the loop
  if [ "$FIRST_LETTERS" = "pci" ] || [ "$FIRST_LETTERS" = "/" ] ; then
    CARRY_ON=0

  # else,
  else
    # if possible, enable wakeup for current SYSFS
    WAKEUP_FILE="${CURRENT_SYSFS}/power/wakeup"
    if [ -f $WAKEUP_FILE ]; then
      echo "enabled" > $WAKEUP_FILE
    fi

    # go to father directory of current SYSFS
    CURRENT_SYSFS=`dirname $CURRENT_SYSFS`
  fi
done

# log the action
LOG_HEADER="USB device ${DEVICE_VENDOR}:${DEVICE_PRODUCT}"
logger "${LOG_HEADER} - Description : ${DEVICE_LABEL}"
logger "${LOG_HEADER} - SysFS path  : ${DEVICE_SYSFS}"
logger "${LOG_HEADER} - Device is enabled to handle suspend/resume"

/usr/local/sbin/select-resume-devices:

#!/bin/bash
#
# Script to create resume udev rules for USB devices
#
# Depends on zenity
# 09/05/2012 - V1.0 by Nicolas Bernaerts

# udev file to generate
UDEV_FILE="/etc/udev/rules.d/90-hid-wakeup-enable.rules"

# set separator as CR
IFS=$'\n'

# list all USB devices, excluding root & hubs
LIST_DEVICE=(`lsusb | grep -v "0000:0000" | grep -iv "hub" | sed 's/^.*[0-9a-f]\:[0-9a-f]* \(.*\)$/\1/g'`)

# loop thru the devices array to generate zenity parameter
for DEVICE in "${LIST_DEVICE[@]}"
do
  # if needed, remove [xxx] from device name as it gives trouble with grep
  DEVICE=`echo "$DEVICE" | sed 's/\[.*\]//g'`

  # add it to the parameters list
  ARR_PARAMETER=( FALSE ${DEVICE} ${ARR_PARAMETER[@]} )
done

# display the dialog box to choose devices
TITLE="Wakeup - Enable USB devices"
TEXT="Please, select USB devices you want to use to resume your computer"
CHOICE=`zenity --list --width=600 --height=250 --text=$TEXT --title=$TITLE --checklist --column "Select" --column "Device name" "${ARR_PARAMETER[@]}"`

# slit the device choice into an array
IFS="|" read -a ARR_CHOICE <<< "$CHOICE"

# if at least one device has been selected, initialise udev rules file
[ ${#ARR_CHOICE[@]} -gt 0 ] && echo "# udev rule for USB wake-up of selected devices" > $UDEV_FILE
[ ${#ARR_CHOICE[@]} -gt 0 ] && echo "#" >> $UDEV_FILE

# loop thru the selected devices to create udev rules
for DEVICE_NAME in "${ARR_CHOICE[@]}"
do
  # get current device data
  DEVICE_DATA=`lsusb | grep "${DEVICE_NAME}" | sed 's/^.*ID \([0-9a-f]*\):\([0-9a-f]*\).*$/\1|\2/g'`
  DEVICE_VENDOR=`echo $DEVICE_DATA | cut -d"|" -f1`
  DEVICE_PRODUCT=`echo $DEVICE_DATA | cut -d"|" -f2`

  # create udev rule for current device
  DEVICE_RULE="SUBSYSTEM==\"usb\", ATTRS{idVendor}==\"$DEVICE_VENDOR\", ATTRS{idProduct}==\"$DEVICE_PRODUCT\" RUN+=\"/usr/local/sbin/enable-wakeup \$env{DEVPATH}\" "

  # add udev rule for current device
  echo "# ${DEVICE_NAME}" >> $UDEV_FILE
  echo ${DEVICE_RULE} >> $UDEV_FILE
done

# if at least one device has been selected, display notification
TITLE="USB resume enabled"
TEXT="Your USB devices are resume enabled.\nTo finalize configuration you have to do one of these actions :\n- replug USB devices\n- reboot the computer"
[ ${#ARR_CHOICE[@]} -gt 0 ] && notify-send --icon=media-eject $TITLE $TEXT

Setting up Jest unit tests in a React + Typescript project

Mon, 09 Nov 2020 00:00:00 +0000

Here’s what I do when I want to set up a Jest on a React project.

First, we need to install some dependencies:

npm install --save-dev jest babel-jest ts-jest chai chai-jest identity-obj-proxy

What are all of these?

jest is a unit test runner from Facebook. You probably already know this, because you’re reading this.
ts-jest is a transform for jest which compiles typescript files. If you’re using babel to compile your typescript files, you can skip this.
babel-jest is like ts-jest, but uses babel to transform files - handy if you have a project with some mixed typescript and javascript.
chai is an assertion library. Jest already comes with an expect built in, but if you’re coming from mocha you probably already use chai, and it’s somewhat more expressive and has a lot of plugins available.
chai-jest is a plugin for chai which has supports jest mocks. It basically re-implements a bunch of the jest-specific asserts on the built-in expect object. See the documentation for a list of assertions available.
identity-obj-proxy is a handy library for cases where we import files like CSS modules. If you import styles from 'styles.css', then we can configure jest to import ‘identity-obj-proxy’ for *.css, and then when you do styles.container, it will resolve to “container” instead of throwing an exception.

Then we’re going to create some files to set up jest. First, jest.config.js:

// jest.config.js
module.exports = {
  globals: {
    "ts-jest": {
      // Tell ts-jest about our typescript config.
      // You can specify a path to your tsconfig.json file,
      // but since we're compiling specifically for node here,
      // this works too.
      tsConfig: {
        target: "es2019",
      },
    },
  },
  // Transforms tell jest how to process our non-javascript files.
  // Here we're using babel for .js and .jsx files, and ts-jest for
  // .ts and .tsx files.  You *can* just use babel-jest for both, if
  // you already have babel set up to compile typescript files.
  transform: {
    "^.+\\.jsx?$": "babel-jest",
    "^.+\\.tsx?$": "ts-jest",
    // If you're using babel for both:
    // "^.+\\.[jt]sx?$": "babel-jest",
  },
  // In webpack projects, we often allow importing things like css files or jpg
  // files, and let a webpack loader plugin take care of loading these resources.
  // In a unit test, though, we're running in node.js which doesn't know how
  // to import these, so this tells jest what to do for these.
  moduleNameMapper: {
    // Resolve .css and similar files to identity-obj-proxy instead.
    ".+\\.(css|styl|less|sass|scss)$": `identity-obj-proxy`,
    // Resolve .jpg and similar files to __mocks__/file-mock.js
    ".+\\.(jpg|jpeg|png|gif|eot|otf|webp|svg|ttf|woff|woff2|mp4|webm|wav|mp3|m4a|aac|oga)$": `<rootDir>/__mocks__/file-mock.js`,
  },
  // Tells Jest what folders to ignore for tests
  testPathIgnorePatterns: [`node_modules`, `\\.cache`],
  testURL: `http://localhost`,
};

We need to create the “mocks/file-mock.js” referenced above in “moduleNameMapper”, too. This file will be imported in place of “.jpg” or “.mp3” or similar files:

// __mocks__/file-mock.js
module.exports = "test-file-stub";

Resolving Custom Paths

In your tsconfig.json file, you can set up a “paths” section:

"paths": {
    "components/*": ["src/components/*"],
}

This lets you import Button from "components/Button", and helps you avoid long chains of “../../..” in your code. But, we need to make jest know about these imports. The easiest way to do this, if you’re already using webpack, is via the jest-webpack-resolver. In your webpack configuration, you probably already have something like:

const TsconfigPathsPlugin = require('tsconfig-paths-webpack-plugin');

module.exports = {
    ...
    resolve: {
        modules: ['node_modules'],
        plugins: [new TsconfigPathsPlugin({ extensions })],
        extensions: ['.js', '.jsx', '.ts', '.tsx'],
    },
    ...
}

So we’re going to take advantage of the fact that webpack knows how to resolve things already:

npm install --save-dev jest-webpack-resolver

And then somewhere in your jest.config.js add:

module.exports = {
    ...
    resolver: 'jest-webpack-resolver',
    ...
}

If you are not using webpack, then check out tsconfig-paths-jest as a possible alternative.

Running tests

At this point you should be able to:

npx jest

And it should run your tests!

Example Test File

Here’s a quick example of a test file - if you have a Button.tsx, you might put this in Button.test.tsx in the same folder. Note that this uses the @testing-library/react and @testing-library/user-event libraries:

import { render } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import React from "react";
import chai from "chai";
import chaiDom from "chai-dom";

chai.use(chaiDom);
const { expect } = chai;

describe("Test Suite", () => {
  beforeEach(() => {
    // TODO: Uncomment this if you're using `jest.spyOn()` to restore mocks between tests
    // jest.restoreAllMocks();
  });

  it("click on a button", () => {
    const { getByText } = render(<button>Hello World</button>);

    // `getByText` comes from `testing-library/react` and will find an element,
    // or error if the element doesn't exist.  See the queries documentation
    // for info about other query types:
    // https://testing-library.com/docs/dom-testing-library/api-queries
    const button = getByText("Hello World");

    // `userEvent` is a library for interacting with elements.  This will
    // automatically call `React.act()` for you - https://reactjs.org/docs/test-utils.html#act.
    userEvent.click(button);
  });
});

Note that all tests run in node.js, where this is no browser. By default, Jest will initialize a jsdom environment for you, which gives you a window and a document and will let you render nodes to a virtual screen. But, if you’re writing a test for a module that doesn’t need to interact with the DOM, you can speed up a test by using the “node” jest environment which will skip all of that:

/**
 * @jest-environment node
 */

import chai from "chai";
import chaiJest from "chai-jest";
chai.use(chaiJest);

const { expect } = chai;

describe("Test Suite", () => {
  it("should do the thing", () => {
    // TODO
  });
});