Missing documentation on deploying custom plugins on vault running on Kubernetes

[techs07]

Hi Team,

I want to push some custom backend plugin to vault cluster running on Kubernetes environment. I have gone through the documentation and could find anything relevant to kubernetes.
These are the available documentation so far and there is no details about pushing plugin on vault kubernetes cluster.
https://www.vaultproject.io/docs/plugin
https://www.vaultproject.io/docs/internals/plugins
https://learn.hashicorp.com/vault/developer/plugin-backends

Could you please update details like

1. Pushing the custom apps to pods
2. Replication of plugin across nodes/pods.
3. use of persistence volume to store plugins, just like raft integrated storage.. So that Plugin can persist after node or pod failure.

Environment details:
Vault raft cluster with 3 nodes.
Vault version: 1.4.1

Thanks

[techs07]

sharing some steps which I have followed to setup custom plugins in Kubernetes. Hope it will help others, till we get best practices to setup plugins.

1. Set plugin location helm charts. here we will try to keep in raft storage so that plugins persist across the life cycle of pods. if you set raft path as '/vault/data/' , use plugin directory under that path.
   plugin_directory = "/vault/data/plugins/"

2. copy plugin to all PVC. If you have three pods then copy plugins to all three PVC.
   kubectl cp <plugin_path> vault-0:/vault/data/plugins/
   Repeat for other pods.
   kubectl cp <plugin_path> vault-1:/vault/data/plugins/
   kubectl cp <plugin_path> vault-2:/vault/data/plugins/

3. generate shasum
   shasum -a 256 <plugin_path>

4. Configure plugin by providing shasum
   vault write sys/plugins/catalog/plugin_name
   sha256=
   command="plugin_name"

5. Reload the plugin if you are updating any existing plugin.
   $ cat payload.json
   {
   "plugin": "mock-plugin"
   }

$ curl
--header "X-Vault-Token: ..."
--request PUT
--data @payload.json
http://127.0.0.1:8200/v1/sys/plugins/reload/backend

6. You are all set. configure the plugin and use it.

These plugin will persist even in new pods, provided you are using same PVV.

If anyone has better solution, please update the post. Thanks

[stefan-zh]

There is now an official guide on how to upgrade a plugin in Vault: https://www.vaultproject.io/docs/upgrading/plugins#upgrading-vault-plugins
It's exactly the same procedure that @techs07 outlines, but @techs07 has provided the coding steps.