update docs for next release (#73)

kondratyevd · web-flow · commit d4d6946b9873 · 2025-08-01T13:48:10.000-04:00
diff --git a/docs/advanced-monitoring.rst b/docs/advanced-monitoring.rst
@@ -23,17 +23,66 @@ in your ``values.yaml``:
 
 .. code-block:: yaml
 
-    tracing_sampling_rate: 0.01
+    envoy:
+      tracing_sampling_rate: 0.01
 
     opentelemetry-collector:
       enabled: true
+      config:
+        exporters:
+          otlp:
+            endpoint: http://supersonic-tempo:4317
+          otlphttp:
+            endpoint: http://supersonic-tempo:4318
+          prometheusremotewrite:
+            endpoint: http://supersonic-prometheus-server:9090/api/v1/write
 
     tempo:
       enabled: true
+      tempo:
+        metricsGenerator:
+          enabled: true
+          remoteWriteUrl: http://supersonic-prometheus-server:9090/api/v1/write
+
+.. note::
+
+    In the example above, endpoints and remote write URLs are configured to point to
+    the Prometheus server and Grafana Tempo services, which will most likely
+    have names like ``<release-name>-prometheus-server`` and ``<release-name>-tempo``.
 
 The ``tracing_sampling_rate`` parameter controls how frequently requests are
 traced. A value of ``0.01`` means that one in 100 requests will be traced.
 
+Additionally, you will need to enable tracing in Triton Inference Server, which is
+done by passing additional flags to ``tritonserver`` command. The following example
+shows how tracing is configured for CMS SuperSONIC instance:
+
+.. code-block:: bash
+
+      /opt/tritonserver/bin/tritonserver \
+      --model-repository=/cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre7/external/el9_amd64_gcc12/data/RecoBTag/Combined/data/models/ \
+      --model-repository=/cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre7/external/el9_amd64_gcc12/data/RecoTauTag/TrainingFiles/data/DeepTauIdSONIC/ \
+      --model-repository=/cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre7/external/el9_amd64_gcc12/data/RecoMET/METPUSubtraction/data/models/ \
+      --model-repository=/cvmfs/cms.cern.ch/el9_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre7/external/el9_amd64_gcc12/data/RecoEgamma/EgammaPhotonProducers/data/models/ \
+      --trace-config mode=opentelemetry \
+      --trace-config=opentelemetry,resource=pod_name=$(hostname) \
+      --trace-config opentelemetry,url=supersonic-opentelemetry-collector:4318/v1/traces \
+      --trace-config rate=100 \
+      --trace-config level=TIMESTAMPS \
+      --trace-config count=-1 \
+      --allow-gpu-metrics=true \
+      --log-verbose=0 \
+      --strict-model-config=false \
+      --exit-timeout-secs=60
+
+.. note::
+
+    In the example above, the url should point to the OpenTelemetry Collector service,
+    which will most likely have a name ``<release-name>-opentelemetry-collector``.
+
+For tracing in Triton, the rate is the inverse of the ``tracing_sampling_rate``
+parameter in the Envoy Proxy configuration: rate=100 means 1% of requests will be traced.
+
 .. warning::
 
     Triton Inference Server supports OpenTelemetry tracing only in versions 24.x or later.
@@ -43,7 +92,36 @@ Displaying Tracing Data in Grafana
 
 If Grafana is enabled in your ``values.yaml``, you can display the tracing data
 in the Grafana dashboard. In order to achieve this, Grafana needs to have a
-Tempo datasource configured. 
+Tempo datasource configured:
+
+.. code-block:: yaml
+
+    grafana:
+      enabled: true
+      datasources:
+        datasources.yaml:
+          datasources:
+            - name: prometheus
+              type: prometheus
+              access: proxy
+              isDefault: true
+              url: http://supersonic-prometheus-server:9090
+              jsonData:
+                timeInterval: "5s"
+                tlsSkipVerify: true
+            - name: tempo
+              type: tempo
+              url: http://supersonic-tempo:3100
+              access: proxy
+              isDefault: false
+              basicAuth: false
+              jsonData:
+                timeInterval: "5s"
+                tlsSkipVerify: true
+                serviceMap:
+                  datasourceUid: "prometheus"
+                nodeGraph:
+                  enabled: true
 
 If OpenTelemetry Collector and Tempo are enabled, the default Grafana dashboard
 will include an interactive server map, where you can study tracing data in detail
diff --git a/docs/configuration-guide.rst b/docs/configuration-guide.rst
@@ -107,22 +107,9 @@ Triton version must be specified in the ``triton.image`` parameter in the values
        cpu: 2
        memory: 16G
 
-- In addition, you can use ``triton.affinity`` to steer Triton pods to nodes with specific GPU models:
-
-.. code-block:: yaml
-
-   affinity:
-     nodeAffinity:
-       requiredDuringSchedulingIgnoredDuringExecution:
-         nodeSelectorTerms:
-           - matchExpressions:
-             - key: nvidia.com/gpu.product
-               operator: In
-               values:
-                 - NVIDIA-A10
-                 - NVIDIA-A40
-                 - NVIDIA-L40
-                 - NVIDIA-L4
+- In addition, you can use ``triton.nodeSelector``, ``triton.tolerations``,
+  ``triton.annotations``, and ``triton.affinity`` to steer Triton pods to specific nodes.
+  This is particularly useful for co-locating Triton pods with Envoy proxy to reduce latency.
 
 
 4. Configure Envoy Proxy
@@ -242,9 +229,12 @@ Prometheus is needed to scrape metrics for monitoring, as well as for the rate l
        server:
          ingress:
            enabled: true
-           hostName: "<prometheus_url>"
            ingressClassName: "<ingress_class>"
-           annotations: {}
+           hosts:
+              - "<prometheus_url>"
+           tls:
+             - hosts:
+                 - "<prometheus_url>"
 
   The parameters you will most likely need to configure in your values file are related to
   Ingress for web access to Prometheus UI.
@@ -297,7 +287,13 @@ Grafana is used to visualize metrics collected by Prometheus.
 We provide a pre-configured Grafana dashboard which includes many useful metrics,
 including latency breakdown, GPU utilization, and more.
 
-Grafana is installed as a subchart with most of the default values pre-configured.
+If you have a Grafana instance already installed, you can deploy SuperSONIC dashboars
+by copying one of the JSON files from the
+`SuperSONIC repository <https://github.com/fastmachinelearning/SuperSONIC/tree/main/helm/supersonic/dashboards>`_.
+
+If you don't have a Grafana instance already installed, you can deploy one as a subchart of SuperSONIC,
+in which case the dashboard will be automatically deployed.
+
 You can further customize the Grafana installation by passing parameters from
 official Grafana `values.yaml <https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml>`_ file
 under the ``grafana`` section of the SuperSONIC values file:
@@ -308,9 +304,12 @@ under the ``grafana`` section of the SuperSONIC values file:
      enabled: true
      ingress:
        enabled: true
-       hostName: "<grafana_url>"
        ingressClassName: "<ingress_class>"
-       annotations: {}
+       hosts:
+          - "<grafana_url>"
+       tls:
+         - hosts:
+             - "<grafana_url>"
 
 The values you will most likely need to configure in your values file are related to
 Grafana Ingress for web access, and datasources to connect to Prometheus,
@@ -320,7 +319,7 @@ Grafana Ingress for web access, and datasources to connect to Prometheus,
   :height: 200
   :alt: SuperSONIC Grafana Dashboard
 
-10. (Optional) Enable KEDA Autoscaler
+10. Enable KEDA Autoscaler
 ==========================================
 
 Autoscaling is implemented via `KEDA (Kubernetes Event-Driven Autoscaler) <https://keda.sh/>`_ and
diff --git a/docs/getting-started.rst b/docs/getting-started.rst
@@ -33,17 +33,12 @@ Installation
          helm install <release-name> fastml/supersonic -n <namespace> -f <your-values.yaml>
 
       Use a unique meaningful lowercase value as <release-name>, for example
-      ``supersonic-cms-run3``.
-      This value will be used as a prefix for all resources created by the chart,
-      unless ``nameOverride`` is specified in the values file.
+      ``supersonic-cms``.
+      This value will be used as a prefix for all resources created by the chart.
 
-      Successfully executed ``helm install`` command will print a link to auto-generated Grafana dashboard
+      Successfully executed ``helm install`` command will print a link to a Grafana dashboard
       and other useful information.
-   
-   .. figure:: img/grafana.png
-      :align: center
-      :height: 250
-      :alt: Supersonic Grafana dashboard
+
 
 Uninstall SuperSONIC
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
