using skaffold

Background

This assumes you have a deps.edn and you're using jib to create a container image. In other words, you can run the following command.

clj -Tjib build

Skaffold can create a workflow where updates to your files not only build and push images, but also update kubernetes deployment specs with the new image, wait for the pod with your change to become ready, and possibly even setup a local port forward for you to test it. Skaffold also has a "dev" mode which can sync files with a running pod while you're developing. This means that you can set up skaffold to detect changes to your .clj files, sync them with a running pod, and then reload those namespaces. This certainly doesn't replace your repl but it's an interesting way to insert a very fast feedback loop in to code that's running in a local or remote kubernetes cluster.

Create a Skaffold file

In the skaffold file, the build section is the one that is clojure specific. In the example below, we're using jibbit to deploy a project to GKE cluster (using GCR as the image registry).

apiVersion: skaffold/v2beta27
kind: Config
metadata:
  name: gcr-clj-web

build:
  artifacts:
  - image: gcr.io/personalsdm-216019/gcr-clj-web
    custom:
      buildCommand: clj -Tjib build
      dependencies:
        paths:
        - src/**/*.clj
    sync:
      manual:
      - src: src/**/*.clj
        dest: /

deploy:
  kubectl:
    manifests:
    - resources/k8s/deployment/deployment.yaml
    - resources/k8s/deployment/service.yaml

portForward:
- resourceType: service
  resourceName: gcr-clj-web
  port: 8080

At minimum, you'd change metadata.name and build.artifacts.image. After that, you start skaffold in dev mode.

skaffold dev

It will run through a workflow to push at least one image, deploy the spec, wait for the pod, and possibly do a port forward if you include the portForward section. You'll probably need a deployment.yaml and service.yaml spec, but you won't need to update the image tags as part of this workflow. Skaffold will take care of that.

Skaffold decides on the image name

To make sure that skaffold and jib don't conflict, let skaffold control the image name. Skaffold will run the clj -Tjib build with an environment that has the IMAGE variable set to the image name jibbit should use. One way you can integrate with this is to use the :tagger keyword in the :target-image map.

{:target-image {:type :registry
                :image-name "gcr.io/personalsdm-216019/gcr-clj-web"
                :tagger {:fn jibbit.tagger/environment :args {:varname "IMAGE"}}
                :authorizer {:fn jibbit.gcloud/authorizer}}}

I left the :image-name in my configuration (for my non-skaffold builds) but it is completely over-ridden by whatever skaffold wants. The :authorizer is still configured to use my local gcloud configuration for authentication and project selection. It would be something else if you're building to EKS, or a private kubernetes cluster in local kube config.

Caveats for Syncing

Skaffold has a feature which is enabled in the above skaffold.yaml file. build.artifacts[0].sync will watch for changes to clj files and will upload them to the top read-write layer of the container in the running pod! If your deployment is setup to detect these file updates, and reload the namespace, you'll have a workflow where local .clj file changes will re-load on the remote server.

• does not work with :aot mode right now. If your jib.edn sets :aot to true, the container JVM process will not have a src directory in the classpath
• there can be issues if the *.clj files in the container's application layer are owned by root, but the container process is running as non-root. This breaks skaffold syncing. If you are using the :user option in jib.edn to make sure the container runs as a non-root user (which is a good idea), then you may need to remove the :user option and let the container run as root. If your base image is the default distroless/java or you've used one of the official openjdk images then jib will package your .clj files with a "nobody" ownership that wee know will work. This way you can continue to run non-root but syncing will still work.