Update rook to v1.4.2

[surajssd]

Change Notes:

• Update Rook to v1.4.2. Release notes: https://github.com/rook/rook/releases/tag/v1.4.2

• Update the CephCluster CR available in rook-ceph component, taken from: https://github.com/rook/rook/blob/v1.4.2/cluster/examples/kubernetes/ceph/cluster.yaml.

• Update rook toolbox config to include new env vars from the changed secret keys.

• Add tolerations and affinity for provisioner and plugins pods.

---

Update process:

kubectl apply -f https://raw.githubusercontent.com/kinvolk/lokomotive/master/assets/charts/components/rook/templates/resources.yaml
lokoctl component apply rook rook-ceph

Fixes: #852

[invidian]

LGTM

[surajssd]

Ceph Upgrade steps

Steps:

These steps are curated from the upgrade doc provided by rook: https://rook.io/docs/rook/master/ceph-upgrade.html.

• Keep note of the CSI images:

kubectl --namespace rook get pod -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}' -l 'app in (csi-rbdplugin,csi-rbdplugin-provisioner,csi-cephfsplugin,csi-cephfsplugin-provisioner)' | sort | uniq

• Ensure autoscale is on

Ensure that the output of the command ceph osd pool autoscale-status | grep replicapool says on (in the last column) and not warn in the toolbox pod. If it says warn. Then run the command ceph osd pool set replicapool pg_autoscale_mode on to set it to on. This is just to ensure we are not facing: rook/rook#5608.

Read more about the toolbox pod here: https://github.com/kinvolk/lokomotive/blob/master/docs/how-to-guides/rook-ceph-storage.md#enable-and-access-toolbox.

NOTE: If you see this error [errno 5] RADOS I/O error (error connecting to the cluster) in toolbox pod then tag the toolbox pod image to a specific version like rook/ceph:v1.3.2.

• Ceph Status

Run the following in the toolbox pod:

watch ceph status

Ensure that the output says that the health is HEALTH_OK. Match the output such that everything looks fine as explained here: https://rook.io/docs/rook/master/ceph-upgrade.html#status-output.

• Pods in rook namespace:

Watch the pods status in another from the rook namespace in another terminal window. Just running this will be enough:

watch kubectl -n rook get pods

• Watch for the rook version update

Run the following command to keep an eye on the rook version update as it is rolls down for all the components:

watch --exec kubectl -n rook get deployments -l rook_cluster=rook -o jsonpath='{range .items[*]}{.metadata.name}{"  \treq/upd/avl: "}{.spec.replicas}{"/"}{.status.updatedReplicas}{"/"}{.status.readyReplicas}{"  \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'

• Watch for the Ceph version update

Run the following command to keep an eye on the Ceph version update as the new pods come up:

watch --exec kubectl -n rook get deployments -l rook_cluster=rook -o jsonpath='{range .items[*]}{.metadata.name}{"  \treq/upd/avl: "}{.spec.replicas}{"/"}{.status.updatedReplicas}{"/"}{.status.readyReplicas}{"  \tceph-version="}{.metadata.labels.ceph-version}{"\n"}{end}'

• Keep an eye on the events in the rook namespace

kubectl -n rook get events -w

• Ceph Dashboard

Keep it open in one window, but sometimes it is more hassle than any help. It keeps reloading and logs you out automatically. See this on how to access the dashboard: https://github.com/kinvolk/lokomotive/blob/master/docs/how-to-guides/rook-ceph-storage.md#access-the-ceph-dashboard.

• Grafana dashboards

Keep an eye on the Grafana dashboard but the data here will always be old and the most reliable state of the system will come from the watch running inside toolbox pod.

• Run updates

kubectl apply -f https://raw.githubusercontent.com/kinvolk/lokomotive/master/assets/charts/components/rook/templates/resources.yaml
lokoctl component apply rook rook-ceph

• Verify that the csi images are updated:

kubectl --namespace rook get pod -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}' -l 'app in (csi-rbdplugin,csi-rbdplugin-provisioner,csi-cephfsplugin,csi-cephfsplugin-provisioner)' | sort | uniq

• Final checks:

Once everything is up to date then run following commands in the toolbox pod:

ceph status
ceph osd status
ceph df
rados df

[invidian]

LGTM

[knrt10]

+1