docs: Add "How to setup storage using Rook Ceph?"

This document enlists steps that will help user setup storage using Rook
Ceph component.

Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

Author: surajssd

docs/how-to-guides/rook-ceph-storage.md

@@ -0,0 +1,259 @@
+# Storage with Rook Ceph on Packet cloud
+
+## Contents
+
+- [Introduction](#introduction)
+- [Prerequisites](#prerequisites)
+- [Deploy storage worker pool](#deploy-storage-worker-pool)
+  - [Config](#config)
+  - [Deploy the worker pool](#deploy-the-worker-pool)
+- [Deploy Rook](#deploy-rook)
+  - [Config](#config-1)
+  - [Deploy the component](#deploy-the-component)
+- [Deploy Rook Ceph](#deploy-rook-ceph)
+  - [Config](#config-2)
+  - [Deploy the component](#deploy-the-component-1)
+- [Access Ceph dashboard](#access-ceph-dashboard)
+- [Enable and access toolbox](#enable-and-access-toolbox)
+- [Enable monitoring](#enable-monitoring)
+- [Make default storage class](#make-default-storage-class)
+- [Additional resources](#additional-resources)
+
+## Introduction
+
+This document explains how to set up storage using Rook and Rook Ceph components of Lokomotive. Rook is a storage aggregator which supports various storage backends, but Lokomotive only supports Ceph at the moment. Lokomotive supports block storage with Ceph.
+
+## Prerequisites
+
+- A Lokomotive cluster is accessible via `kubectl` deployed on a supported provider.
+
+## Deploy storage worker pool
+
+#### Config
+
+Deploy a cluster with at least one worker pool dedicated to Rook Ceph. A dedicated worker pool configuration should look like the following:
+
+```tf
+cluster "packet" {
+  ...
+
+  worker_pool "storage" {
+    count = 3
+    node_type = "c2.medium.x86"
+
+    labels = "storage.lokomotive.io=ceph"
+    taints = "storage.lokomotive.io=ceph:NoSchedule"
+  }
+}
+```
+
+- The number of machines provided using `count` should be an odd number greater than three.
+- Type of node, provided using `node_type`, should be one that has multiple disks like `c2.medium.x86` or `s1.large.x86`. Find out more servers [here](https://www.packet.com/cloud/servers/).
+- To steer Rook Ceph workload on these storage nodes provide `labels`.
+- Provide `taints` so that other workload can be **steered away** by default. This setting is not mandatory, but isolating storage workloads from others is recommended.
+
+#### Deploy the worker pool
+
+```bash
+lokoctl cluster apply -v
+```
+
+## Deploy Rook
+
+#### Config
+
+```tf
+component "rook" {
+  node_selector = {
+    "storage.lokomotive.io" = "ceph"
+  }
+
+  toleration {
+    key      = "storage.lokomotive.io"
+    operator = "Equal"
+    value    = "ceph"
+    effect   = "NoSchedule"
+  }
+
+  agent_toleration_key    = "storage.lokomotive.io"
+  agent_toleration_effect = "NoSchedule"
+
+  discover_toleration_key    = "storage.lokomotive.io"
+  discover_toleration_effect = "NoSchedule"
+}
+```
+
+- `node_selector` should match the `labels` provided in the `worker_pool`.
+- `toleration` should match the `taints` mentioned in the `worker_pool`.
+- `agent_toleration_key` and `discover_toleration_key` should match the key of the `taints` provided in the `worker_pool`.
+- `agent_toleration_effect` and `discover_toleration_effect` should match the effect of the `taints` provided in the `worker_pool`.
+
+#### Deploy the component
+
+```bash
+lokoctl component apply rook
+```
+
+Verify the operator pod in the `rook` namespace is in the `Running` state (this may take a few minutes):
+
+```console
+$ kubectl -n rook get pods -l app=rook-ceph-operator
+NAME                                  READY   STATUS    RESTARTS   AGE
+rook-ceph-operator-76d8687f95-6knf8   1/1     Running   0          81m
+```
+
+## Deploy Rook Ceph
+
+#### Config
+
+```tf
+component "rook-ceph" {
+  monitor_count = 3
+
+  node_affinity {
+    key      = "storage.lokomotive.io"
+    operator = "Exists"
+  }
+
+  toleration {
+    key      = "storage.lokomotive.io"
+    operator = "Equal"
+    value    = "ceph"
+    effect   = "NoSchedule"
+  }
+
+  storage_class {
+    enable = true
+  }
+}
+```
+
+- `monitor_count` should be an odd number greater than three but not higher than the `count` of workers in the `worker_pool`.
+- `node_affinity` should match the `labels` provided in the `worker_pool`.
+- `toleration` should match the `taints` provided in the `worker_pool`.
+- `storage_class` should be provided if workloads have PVC.
+
+#### Deploy the component
+
+```bash
+lokoctl component apply rook-ceph
+```
+
+Verify the osd pods in the `rook` namespace are in the `Running` state (this may take a few minutes):
+
+```console
+$ kubectl -n rook get pods -l app=rook-ceph-osd
+NAME                               READY   STATUS    RESTARTS   AGE
+rook-ceph-osd-0-6d4f69dbf9-26kzl   1/1     Running   0          67m
+rook-ceph-osd-1-86c9597b84-lmh94   1/1     Running   0          67m
+rook-ceph-osd-2-6d97697897-7bprl   1/1     Running   0          67m
+rook-ceph-osd-3-5bfb9d86b-rk6v4    1/1     Running   0          67m
+rook-ceph-osd-4-5b76cb9675-cxkdw   1/1     Running   0          67m
+rook-ceph-osd-5-8c86f5c6c-6qxtz    1/1     Running   0          67m
+rook-ceph-osd-6-5b9cc479b7-vjc9v   1/1     Running   0          67m
+rook-ceph-osd-7-7b84d6cc48-b46z9   1/1     Running   0          67m
+rook-ceph-osd-8-5868969f97-2bn9r   1/1     Running   0          67m
+```
+
+## Access Ceph dashboard
+
+Ceph dashboard provides valuable visual information. It is an essential tool to monitor the Ceph cluster. Here are steps on how to access it.
+
+Find the admin password:
+
+```bash
+kubectl -n rook get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
+```
+
+Port forward the service locally:
+
+```bash
+kubectl -n rook port-forward svc/rook-ceph-mgr-dashboard 8443:8443
+```
+
+Now open the following URL: [https://localhost:8443](https://localhost:8443) and enter the username `admin` and password obtained from the first step.
+
+## Enable and access toolbox
+
+Ceph is a complex software system, and not everything that goes on into the Ceph cluster is visible at the rook layer of abstraction. So command-line interface to interact with Ceph cluster is useful. Ceph toolbox helps you access the ceph cluster using `ceph` cli utility. Here you can configure the Ceph cluster setting and debug the cluster if needed.
+
+To deploy the toolbox, the `rook-ceph` component config should set the flag `enable_toolbox` to `true`.
+
+```tf
+component "rook-ceph" {
+  enable_toolbox = true
+
+  ...
+}
+```
+
+Verify the deployment of toolbox pod to see if it is ready:
+
+```console
+$ kubectl -n rook get deploy rook-ceph-tools
+NAME              READY   UP-TO-DATE   AVAILABLE   AGE
+rook-ceph-tools   1/1     1            1           39s
+```
+
+Access the toolbox pod using the following command:
+
+```
+kubectl -n rook exec -it $(kubectl -n rook get pods -l app=rook-ceph-tools -o name) bash
+```
+
+Once inside the pod you can run usual `ceph` commands:
+
+```console
+[root@rook-ceph-tools-674c8dd89d-pp4x9 /]# ceph status
+  cluster:
+    id:     1a73e7b2-c825-414e-81f0-0f70b19478d3
+    health: HEALTH_OK
+
+  services:
+    mon: 3 daemons, quorum a,b,c (age 50m)
+    mgr: a(active, since 51s)
+    osd: 9 osds: 9 up (since 48m), 9 in (since 48m)
+
+  data:
+    pools:   1 pools, 32 pgs
+    objects: 0 objects, 0 B
+    usage:   9.0 GiB used, 3.0 TiB / 3.1 TiB avail
+    pgs:     32 active+clean
+```
+
+## Enable monitoring
+
+Monitor `rook` and `rook-ceph` components using the `prometheus-operator` component. To enable your `rook` component config should have the flag `enable_monitoring` set to `true`. Deploy the `prometheus-operator` component before.
+
+```tf
+component "rook" {
+  enable_monitoring = true
+
+  ...
+}
+```
+
+## Make default storage class
+
+It is recommended to make the storage class as default if rook ceph is the only storage provider in the cluster. This setting helps to satisfy any PVC request made in the cluster. The `rook-ceph` component config should look like the following:
+
+```tf
+component "rook-ceph" {
+  ...
+
+  storage_class {
+    enable  = true
+    default = true
+  }
+}
+```
+
+## Additional resources
+
+Rook docs:
+
+  - [Ceph toolbox](https://rook.io/docs/rook/master/ceph-toolbox.html).
+  - [Ceph dashboard](https://rook.io/docs/rook/master/ceph-dashboard.html).
+  - [Ceph direct tools](https://rook.io/docs/rook/master/direct-tools.html).
+  - [Ceph advanced configuration](https://rook.io/docs/rook/master/ceph-advanced-configuration.html).
+  - [Disaster recovery](https://rook.io/docs/rook/master/ceph-disaster-recovery.html).