feat: decoupled prometheus exporter's calculation and output #12383

SkyeYoung · 2025-06-26T09:49:41Z

Description

This PR decouples the calculation and output processes of the Prometheus exporter. The "calculation" is performed in the privileged agent process at intervals defined by the refresh_interval(default is 15s) and written to a shared dict, while the "output" (i.e., the /apisix/prometheus/metrics API) is moved to the worker process, which only reads and returns the cached data in the shared dict.

The above are just the core changes. In fact, I encountered many other problems, which have been commented or annotated in the corresponding positions, and will not be repeated here.

For the testing part, since the Prometheus exporter currently refreshes data every 15 seconds, I used a smaller interval in the relevant tests to pass the original tests.

Which issue(s) this PR fixes:

Fixes #

Stress Testing

How to

install wrk2, git clone apisix
deploy etcd(./test.sh init-etcd), nginx(as upstream, ./test.sh start-nginx)
make run
create 10k routes(./test.sh create)
enable promethues(./test.sh enable-prometheus)
run benchmark(./test.sh benchmark)

Results

Key Performance Indicators

Scenario	CPU Usage	Memory Usage	P50 Latency	P90 Latency	P99 Latency
Prometheus Disabled	33.0%	144.3MB	3.58ms	5.99ms	8.99ms
Prometheus Enabled	38.3% (+5.2%)	100.2MB (-44.1MB)	4.22ms (+0.64ms)	7.07ms (+1.08ms)	10.66ms (+1.67ms)
1 Metrics Reqs	38.6% (+5.6%)	101.3MB (-42.9MB)	4.25ms (+0.68ms)	7.87ms (+1.87ms)	12.17ms (+3.18ms)
3 Metrics Reqs	39.1% (+6.1%)	100.1MB (-44.2MB)	3.88ms (+0.30ms)	7.17ms (+1.18ms)	11.92ms (+2.93ms)

Performance Impact Summary

Scenario	CPU Impact	Memory Impact	P50 Latency Impact	P90 Latency Impact	P99 Latency Impact
Prometheus Disabled (Baseline)	0.0%	0.0MB	0.00ms	0.00ms	0.00ms
Prometheus Enabled	+5.2%	-44.1MB	+0.64ms	+1.08ms	+1.67ms
1 Metrics Reqs	+5.6%	-42.9MB	+0.68ms	+1.87ms	+3.18ms
3 Metrics Reqs	+6.1%	-44.2MB	+0.30ms	+1.18ms	+2.93ms

Checklist

I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

…heus-exporter-concurrency

apisix/plugins/prometheus/exporter.lua

SkyeYoung · 2025-06-27T09:31:33Z

apisix/plugins/prometheus/exporter.lua

@@ -454,10 +458,11 @@ local function collect(ctx, stream_only)
    local config = core.config.new()

    -- config server status
-    local vars = ngx.var or {}
-    local hostname = vars.hostname or ""
+    local hostname = core.utils.gethostname() or ""


Because API disabled in the context of ngx.timer, context: ngx.timer.

…heus-exporter-concurrency

This reverts commit 521f4c9.

…heus-exporter-concurrency

SkyeYoung · 2025-07-22T09:19:57Z

apisix/plugins/prometheus/exporter.lua

+    -- FIXME:
+    -- Now the HTTP subsystem loads the stream plugin unintentionally, which shouldn't happen.
+    -- It breaks the initialization logic of the plugin,
+    -- here it is temporarily fixed using a workaround.
+    if ngx.config.subsystem ~= "stream" then
+        return
+    end


As mentioned in the comments, the http subsystem also loads the stream plugins. This is an issue that needs to be resolved.

Please create an issue for this. thx @SkyeYoung

I'll create it a little later

bzp2010 · 2025-07-22T10:00:38Z

apisix/plugin.lua

-            local enabled = core.table.array_find(http_plugin_names, "prometheus") ~= nil
-            local active  = exporter.get_prometheus() ~= nil
-            if not enabled then
-                exporter.destroy()
-            end
-            if enabled and not active then
-                exporter.http_init()
-            end


Add some description under this comment explaining why we removed it and moved to init and destroy hooks.

The original code skipped plugin.init() and old_plugin.destroy() used in https://github.com/apache/apisix/blob/6fb9bf94281525c1fca397f681b4890b69440369/apisix/plugin.lua, and implemented the overload of the prometheus plugin for some reason that I have not yet understood (perhaps because prometheus.lua originally did not contain two functions init and destroy).

The initial reason was that even after separating the init_prometheus part and placing it at the end of init_worker, directly calling exporter_timer() would still cause an error. After debugging, another initialization logic was found here. This is obviously redundant.

Currently, we provide init and destroy functions in prometheus.lua, allowing the initialization and reloading of the prometheus plugin to be handled within the plugin's own files, reducing coupling.

This also allows the prometheus plugin to revert to the mechanism provided by plugin.lua, reducing special cases, lowering the cost of understanding, and making the code easier to maintain.

bzp2010 · 2025-07-22T10:07:10Z

apisix/plugin.lua

-        require("apisix.plugins.prometheus.exporter").http_init(prometheus_enabled_in_stream)
-    elseif not is_http and core.table.array_find(stream_plugin_names, "prometheus") then
-        require("apisix.plugins.prometheus.exporter").stream_init()
+    if is_http and (enabled_in_http or enabled_in_stream) then


NOTE

We will always only handle metrics generation in the http subsystem.

This will ensure that there is no duplication of execution on http and stream to waste compute resources.

This simplifies the design.

Whether or not the user has http enabled (i.e., whether or not it is in stream only mode), an http block for the Prometheus export API and its server block (:9091) will always be present, otherwise Prometheus would be pointless. This means that we can always have an http subsystem context for periodic generation of timers and metrics anyway, even if we are currently in stream only mode.

Please add some comments to the code to document the design intent. @SkyeYoung

It might be better to add a link to this PR comment here.

https://github.com/apache/apisix/pull/12383/files#r2221993953

@bzp2010 I think this part of the code can be found by modifying the history, just like the old code.

bzp2010 · 2025-07-22T10:19:52Z

apisix/plugins/prometheus.lua

@@ -35,6 +34,7 @@ local _M = {
    priority = 500,
    name = plugin_name,
    log  = exporter.http_log,
+    destroy = exporter.destroy,


NOTE

This will always destroy the plugin (the prometheus instance in it) when reloading it using the Admin API and loading it again based on the latest configuration.
If a reload is performed after a plugin is removed from the profile list, this plugin will not be restored again. Until the next reload.

Technically, exporter.destroy just backs up that instance of the prometheus module and copies it to another variable.
This will cause the export API to stop working, at which point it will always return a {}, which is consistent with the current behavior.
Under the hood, the timer will also stop working, no longer generating metrics based on interval timing, and the metrics computation overhead introduced by APISIX is completely eliminated.
When the next plugin reload occurs, if prometheus is re-enabled, the timer will resume running.

Regarding the background timer introduced by the prometheus third-party library, unfortunately, it never stops running.
It is registered with ngx.timer.every to perform the task of synchronizing the shdict at regular intervals, and this overhead cannot be paused or resumed by external intervention unless we fork and modify the library itself.

So this destruction does not mean that the prometheus instance is actually destroyed, the synchronization timer is stopped, and the shdict is cleared. none of this happens.

bzp2010 · 2025-07-22T10:22:54Z

apisix/plugins/prometheus.lua

@@ -55,4 +55,11 @@ function _M.api()
 end


+function _M.init()


NOTE

We turned to using the built-in hooks of the plugin system, namely init to initialize the prometheus instance and prometheus metrics registration.

Note, however, that data padding only occurs the first time the plugin is started (it is usually when the worker is started, i.e. the init_prometheus call in init.lua http_init_worker) and every timer.
This initialization just registers the metrics, but doesn't really populate the data.

bzp2010 · 2025-07-22T10:25:49Z

apisix/plugins/prometheus.lua

+function _M.init()
+    local local_conf = core.config.local_conf()
+    local enabled_in_stream = core.table.array_find(local_conf.stream_plugins, "prometheus")
+    exporter.http_init(enabled_in_stream)


NOTE

The prometheus plugin, loaded by the http subsystem, will register http metrics there, and will decide whether to register stream metrics (xrpc) depending on whether the stream subsystem has been started.
This is mainly for metrics generation needs in privileged processes, stream data is not really reported at any phase in the http subsystem.

bzp2010 · 2025-07-22T10:36:16Z

apisix/plugins/prometheus/exporter.lua

-        local version, err = config:server_version()
-        if version then
-            metrics.etcd_reachable:set(1)
+        if yieldable then


NOTE

Metrics contains the etcd reachability report and the etcd latest modified index report, which relies on communication with etcd.
According to openresty's restrictions, yield is prohibited in the init_worker phase, i.e. cosocket-based communication with etcd is not allowed.
So we skip a capture here until the timer does them.

bzp2010 · 2025-07-22T10:44:26Z

apisix/plugins/prometheus/exporter.lua

+        return
+    end
+
+    if not prometheus then


NOTE

This is used for dynamic disable (by plugin reload API) of the plugin, i.e. done in exporter.destroy.
This is where we stop if the prometheus instance is "destroyed", and as you can see, this will happen before scheduling the next timer task, which means the timer will stop.

Technically, this is the advantage of ngx.timer.at over ngx.timer.every, every is not terminable, the developer can't get an "instance" of a timer to pause or stop it.
But by using ngx.timer.at, we can precisely control whether or not to continue scheduling the next timed task, which allows us to stop the timer. If you need to resume it, just re-execute ngx.timer.at(0).

bzp2010 · 2025-07-22T10:48:01Z

apisix/plugins/prometheus/exporter.lua

+        return
+    end
+
+    exporter_timer(false, false)


NOTE

The initialization of the timer will perform an acquisition task synchronously, i.e. the first acquisition will always occur in the init_worker phase, which provides initial access to the metrics data.

If at any time , the metrics data (the string in the prometheus-cache shdict) is not available, the API will report an error and log it. This is by design not very likely to happen.

bzp2010 · 2025-07-22T10:49:14Z

apisix/plugins/prometheus/exporter.lua

+    local cached_metrics_text = shdict_prometheus_cache:get(CACHED_METRICS_KEY)
+    if not cached_metrics_text then
+        core.log.error("Failed to retrieve cached metrics: data is nil")
+        return 500, "Failed to retrieve metrics: no data available"


Is it more standard to return a JSON with reference to
https://github.com/apache/apisix/pull/12383/files#diff-390eaff60bfa1071dd1850bce9c7689452eaa13f07bf45a927921ecf05886d1bR580 ?

bzp2010 · 2025-07-22T10:52:40Z

apisix/plugins/prometheus/exporter.lua

    if not prometheus then
-        core.response.exit(200, "{}")
+       return core.response.exit(200, "{}")


JFI, this behavior seems to be inconsistent with what is in get_cached_metrics and we need to confirm which mode should be used. cc @membphis

BTW, prometheus to nil will happen when the plugin is dynamically disabled.

the current way is good to me

apisix/plugins/prometheus/exporter.lua

membphis · 2025-07-22T11:45:09Z

apisix/plugins/prometheus/exporter.lua

    if not prometheus then
-        core.response.exit(200, "{}")
+       return core.response.exit(200, "{}")


the current way is good to me

membphis · 2025-07-22T11:47:07Z

conf/config.yaml.example

@@ -170,6 +170,7 @@ nginx_config:                     # Config for render the template to generate n
  meta:
    lua_shared_dict:              # Nginx Lua shared memory zone. Size units are m or k.
      prometheus-metrics: 15m
+      prometheus-cache: 10m


pls add some comments, tell users when they need to modify it

membphis · 2025-07-22T11:50:17Z

apisix/plugins/prometheus/exporter.lua

+        core.log.error("Failed to collect metrics: ", res)
+        return
+    end
+    shdict_prometheus_cache:set(CACHED_METRICS_KEY, res)


need to capture the return value, it maybe fail

if err, tell user what is the reason, and tell the user to change the default size if the shdict is small

membphis · 2025-07-23T01:43:54Z

apisix/plugins/prometheus/exporter.lua

+    local _, err, forcible = shdict_prometheus_cache:set(CACHED_METRICS_KEY, res)
+
+    if err then
+        core.log.error("Failed to save metrics to shdict: ", err)


need more information, eg: the name of shdict and the size of value

if the value is to large, will fail when we call set method

membphis · 2025-07-23T01:45:15Z

apisix/plugins/prometheus/exporter.lua

    core.response.set_header("content_type", "text/plain")
-    return 200, core.table.concat(prometheus:metric_data())
+    local cached_metrics_text = shdict_prometheus_cache:get(CACHED_METRICS_KEY)


just confirm, no "err" is returned?

I checked the part code and found that the second return value, marked as "flags" in its documentation, actually returns an error when an error occurs.

Let me fix this part of the code.

apisix/plugins/prometheus/exporter.lua

membphis

LGTM

membphis · 2025-07-23T05:49:51Z

apisix/plugins/prometheus/exporter.lua

+
+    -- Clear the cached data after cache_exptime to prevent stale data in case of an error.
+    local _, err, forcible = shdict_prometheus_cache:set(CACHED_METRICS_KEY, res, cache_exptime)
+


remove this blank line

apisix/plugins/prometheus/exporter.lua

bzp2010 · 2025-07-23T05:59:03Z

apisix/plugin.lua

-        require("apisix.plugins.prometheus.exporter").http_init(prometheus_enabled_in_stream)
-    elseif not is_http and core.table.array_find(stream_plugin_names, "prometheus") then
-        require("apisix.plugins.prometheus.exporter").stream_init()
+    if is_http and (enabled_in_http or enabled_in_stream) then


It might be better to add a link to this PR comment here.

https://github.com/apache/apisix/pull/12383/files#r2221993953

SkyeYoung added 11 commits June 26, 2025 09:43

feat: decoupled prometheus exporter's calculation and output

d32ec9f

fix: require process

35a38ad

fix

55619cf

fix: part of exportor logic

467363a

rewrite: use FFI to get nginx_status

e34a924

chore

d91509b

Merge remote-tracking branch 'upstream/master' into young/perf/promet…

d2b3bd4

…heus-exporter-concurrency

chore: simplify logic

b22e74f

chore: rm useless logic

27eec34

fix: lint

a42006f

fix: lint

740c39b

SkyeYoung commented Jun 27, 2025

View reviewed changes

apisix/plugins/prometheus/exporter.lua Show resolved Hide resolved

SkyeYoung commented Jun 27, 2025

View reviewed changes

SkyeYoung added 17 commits July 7, 2025 03:45

fix: try init cached metrics to pass tests

e4800f5

try fix logic

97e00e7

fix: init only one time

171406d

fix: lint

57959e6

chore: try fix tests

a22b9a7

test: fix prometheus related cases

27ac5c9

Merge remote-tracking branch 'upstream/master' into young/perf/promet…

1fb6217

…heus-exporter-concurrency

fix: logic

521f4c9

Revert "fix: logic"

226a14c

This reverts commit 521f4c9.

fix: init etcd logic

f35b711

Merge remote-tracking branch 'upstream/master' into young/perf/promet…

df84c76

…heus-exporter-concurrency

fix: logic

b998b64

fix

0d61202

fix: test case

3601800

fix: test cases

9523fed

test: improve stability

6ae0448

chore(ngx_tpl): rm prom privileged_agent conf

005cc83

SkyeYoung commented Jul 22, 2025

View reviewed changes

bzp2010 reviewed Jul 22, 2025

View reviewed changes

membphis reviewed Jul 22, 2025

View reviewed changes

SkyeYoung added 5 commits July 22, 2025 13:12

fix: lint

72c6f22

fix: add usefull comments and logs

9d5e45c

chore: add comments

61f4812

fix: standardized error return

cf66013

fix: log forciable

6de5129

membphis self-requested a review July 23, 2025 01:34

SkyeYoung requested a review from bzp2010 July 23, 2025 01:44

membphis reviewed Jul 23, 2025

View reviewed changes

SkyeYoung added 2 commits July 23, 2025 02:24

fix: return and log shared.DICT.get err

0b6f1a9

fix: add more explicit error messages when ngx.shared.DICT.set fails

432af12

membphis requested changes Jul 23, 2025

View reviewed changes

apisix/plugins/prometheus/exporter.lua Outdated Show resolved Hide resolved

apisix/plugins/prometheus/exporter.lua Outdated Show resolved Hide resolved

SkyeYoung added 2 commits July 23, 2025 03:44

fix: use ngx.timer.every to reduce potential problems

4d77fc2

fix: add cache exptime

cd96ecd

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jul 23, 2025

SkyeYoung requested a review from membphis July 23, 2025 04:43

membphis previously approved these changes Jul 23, 2025

View reviewed changes

SkyeYoung commented Jul 23, 2025

View reviewed changes

apisix/plugins/prometheus/exporter.lua Outdated Show resolved Hide resolved

chore: rm blank line

f28295d

SkyeYoung dismissed membphis’s stale review via f28295d July 23, 2025 06:06

SkyeYoung requested a review from membphis July 23, 2025 06:07

bzp2010 approved these changes Jul 23, 2025

View reviewed changes

SkyeYoung mentioned this pull request Jul 24, 2025

bug: the stream plugins should not be loaded in the http subsystem #12458

Open

membphis approved these changes Jul 24, 2025

View reviewed changes

nic-6443 approved these changes Jul 25, 2025

View reviewed changes

SkyeYoung merged commit 30ae5df into apache:master Jul 25, 2025
39 of 40 checks passed

SkyeYoung deleted the young/perf/prometheus-exporter-concurrency branch July 25, 2025 01:30


		-- Clear the cached data after cache_exptime to prevent stale data in case of an error.
		local _, err, forcible = shdict_prometheus_cache:set(CACHED_METRICS_KEY, res, cache_exptime)

feat: decoupled prometheus exporter's calculation and output #12383

feat: decoupled prometheus exporter's calculation and output #12383

Uh oh!

Conversation

SkyeYoung commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Which issue(s) this PR fixes:

Stress Testing

How to

Results

Key Performance Indicators

Performance Impact Summary

Checklist

Uh oh!

Uh oh!

SkyeYoung Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SkyeYoung Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bzp2010 Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SkyeYoung Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SkyeYoung Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bzp2010 Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

NOTE

Uh oh!

bzp2010 Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

NOTE

Uh oh!

Choose a reason for hiding this comment

NOTE

Uh oh!

Choose a reason for hiding this comment

NOTE

Uh oh!

Choose a reason for hiding this comment

NOTE

Uh oh!

Choose a reason for hiding this comment

NOTE

Uh oh!

bzp2010 Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

NOTE

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

SkyeYoung commented Jun 26, 2025 •

edited

Loading

SkyeYoung Jun 27, 2025 •

edited

Loading

SkyeYoung Jul 22, 2025 •

edited

Loading

bzp2010 Jul 22, 2025 •

edited

Loading

SkyeYoung Jul 23, 2025 •

edited

Loading

SkyeYoung Jul 22, 2025 •

edited

Loading

bzp2010 Jul 22, 2025 •

edited

Loading

bzp2010 Jul 22, 2025 •

edited

Loading

bzp2010 Jul 22, 2025 •

edited

Loading

SkyeYoung Jul 23, 2025 •

edited

Loading