Metrics for Grafana/Prometheus

[MrSipping]

Hi, how can i retrieve some metrics to pass later on grafana/prometheus. I can't find any information on the site

[lcrilly]

Did you look at https://unit.nginx.org/usagestats/ ?

Here's a simple Python script to extract the metrics in Prometheus format. You can configure this as an "application" if the control socket has a TCP port.

import os
import http.client
import json

def application(environ, start_response):
    connection = http.client.HTTPConnection(os.getenv('ADDR'))
    connection.request("GET", "/status")
    response = connection.getresponse()

    statj = json.loads(response.read())
    connection.close()

    metrics = []
    metrics.append("unit_connections_accepted_total {}".format(statj["connections"]["accepted"]))
    metrics.append("unit_connections_active {}".format(statj["connections"]["active"]))
    metrics.append("unit_connections_idle {}".format(statj["connections"]["idle"]))
    metrics.append("unit_connections_closed_total {}".format(statj["connections"]["closed"]))
    metrics.append("unit_requests_total {}".format(statj["requests"]["total"]))

    for application in statj["applications"].keys():
        metrics.append("unit_application_" + application + "_processes_running {}".format(statj["applications"][application]["processes"]["running"]))
        metrics.append("unit_application_" + application + "_processes_starting {}".format(statj["applications"][application]["processes"]["starting"]))
        metrics.append("unit_application_" + application + "_processes_idle {}".format(statj["applications"][application]["processes"]["idle"]))
        metrics.append("unit_application_" + application + "_requests_active {}".format(statj["applications"][application]["requests"]["active"]))

    start_response('200 OK', [("Content-Type", "text/plain")])
    yield str.encode("\n".join(metrics) + "\n")

[tkumark]

I am new to this. Can you suggest this how to implement this when using unit as container. I am evaluating unit with nodejs in a container. How do I run above scirpt for a container solution.

[tippexs]

Hi @tkumark for what ever reason this issue slipped of my radar! Sorry for that. The code shared by @lcrilly can be configured as a python application. More information about this. http://unit.nginx.org/configuration/#python

This application can be attached to a listener for example on port `9090.

There is furthermore no problem having multiple applications on Unit in a single container.

Let me know if that helps and in case you have any further questions do not hesitate to leave another comment in this thread.

[meezaan]

Hi @tkumark for what ever reason this issue slipped of my radar! Sorry for that. The code shared by @lcrilly can be configured as a python application. More information about this. http://unit.nginx.org/configuration/#python

This application can be attached to a listener for example on port `9090.

There is furthermore no problem having multiple applications on Unit in a single container.

Let me know if that helps and in case you have any further questions do not hesitate to leave another comment in this thread.

@tippexs What user would this application need to run as. When running in a container I can ssh into the container and curl --unix-socket /var/run/control.unit.sock http://localhost/status returns the metrics (but obviously that's the root user). If I run the same command as an application user or using a Python or PHP I'm not running as root and so the request fails (with a curl error 7). Running as root may become a security problem, so what would be the correct way to do this?

[meezaan]

OK, I have a working solution as a sidecar on k8s. Will share details shortly.

[meezaan]

OK, I stand corrected. I have a running sidecar, but that means I have to add another app in my main app container to expose the stats, and that is not ideal. @lcrilly @tippexs would it be possible to have a route in the unit conf file that exposes the status, but only allowed on 127.0.0.1, for instance without the need for me to write a Python or php script? Without that being exposed, at least I'm not able to see how I could deploy a completely independent prometheus exporter as a sidecar. Thank you!

[lcrilly]

Wrote up a more complete solution for exposing a Prometheus endpoint with the default Unix socket control port.

https://gist.github.com/lcrilly/ad516de8378dd8ae5303a942e67d55b5

[meezaan]

I'll test out it out next week. Thanks @lcrilly.

[lcrilly]

@meezaan did this work for you?

[meezaan]

@meezaan did this work for you?

@lcrilly Sorry I have not got around to it, but it's now the highest item on my list (priorities can change quickly :) so I'll tackle it either this weekend or early next week). So expect something here early next week.

[meezaan]

@meezaan did this work for you?

@lcrilly This will require the same hacks as before. There are a couple things:

• /var/run/control.unit.sock is not created until runtime, so unless I bake a custom startup script into my docker container which runs after Unit starts up I can't really change any groups or permissions on this.
• This will still require me to bake a custom script (that's the prometheus.php in the gist) into my docker container to expose the stats. What would be ideal for a sidecar implementation is that Unit is able expose the /status endpoint to a non root user but only on a specific hostname (localhost or 127.0.0.1) without specifying anything else. Then we can deploy sidecars and use them to and use them to extract the status JSON and have the sidecar actually create the Prometheus format of metrics from the /status json. As a reference, Apache allows exposing statistics like this and that's how all the sidecars work with it.

I envision this kind of a unit config (and just config, no additional scripts to be baked in) in my main application config to expose the status endpoint:

{
  "listeners": {
    "*:8080": {
      "pass": "applications/app"
    },
    "127.0.0.1:9090": { 
      "pass": "applications/unit_status"
    }
  },
  "applications": {
    "app": {
      "type": "php",
      "root": "/var/www/html",
      "index": "index.php",
      "script": "index.php"
    },
    "unit_status": {
      "type": "unit_status", # perhaps some special type to denote status
      "group": "any", # In theory, this should not matter because the user is choosing to expose the status, although this is easier imagined than done!
      "environment": {
          "control_socket": "/var/run/control.unit.sock"
      }
    }
  }
}

And now a curl to /status should work, provided the IP is 127.0.0.1 or hostname localhost.

Or something like this.

Essentially, the Prometheus conversion needs to happen in a different container, totally independent of Unit. I hope that makes sense?

[umlumpa]

meezaan

is it works? for everyone?

[meezaan]

meezaan

is it works? for everyone?

No. Not yet. Still does not have what I believe is needed to run an independent sidecar.