bug: limit-count rejections are not captured in the metrics

[indrekj]

Current Behavior

When the limit-count plugin rejects a request, there will be a log message, e.g.

2025/02/25 08:16:10 [warn] 49#49: *42708330 [lua] plugin.lua:1153: run_plugin(): limit-count exits with http status code 429, client: [redacted], server: _, request: "POST /some-address HTTP/1.1", host: "[redacted]", referrer: "[redacted]"

However, these "429" response status codes aren't in the metrics. I've tried it with both the datadog plugin and the prometheus plugin.

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

1. Enable limit-count plugin
2. Set up a route with strict rate limiting (set rejected_code to 429)
3. Enable prometheus plugin
4. Send a bunch of requests to the route so that it gets limited
5. Query the response metrics (/apisix/prometheus/metrics). Expected: 429 responses, actual: no 429 responses

Environment

• Installed through the Apache APISIX Helm Chart
• APISIX version (run apisix version): 3.7.0
• Operating system (run uname -a): Linux apisix-76f84f94f7-blqrb 5.10.228-219.884.amzn2.x86_64 #1 SMP Wed Oct 23 17:17:00 UTC 2024 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V):

nginx version: openresty/1.21.4.2
built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
built with OpenSSL 1.1.1s  1 Nov 2022 (running with OpenSSL 1.1.1w  11 Sep 2023)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_RUNTIME_VER=1.0.1 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.2 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.25 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.34 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.13 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/stream --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/meta --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../wasm-nginx-module-0.6.5 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../grpc-client-nginx-module-v0.4.4 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module

[indrekj]

This is not a bug actually, finally figured it out. We were overriding the error page handler:

      httpSrv: |
        access_log /dev/stdout main buffer=16384 flush=3 if=$loggable;

        location @error {
          default_type 'text/html';
          echo '<html><head><title>$status $status_text</title></head><body><center><h1>$status $status_text</h1></center></html>';
        }
        error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 421 422 423 424 425 426 428 429 431 451 500 501 502 503 504 505 506 507 508 510 511 @error;

[indrekj]

I solved it by writing a custom plugin:

  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: apisix-custom-plugins
    data:
      glia-errors.lua: |
        local core = require("apisix.core")
        local schema = {
          type = "object",
          properties = {},
          required = {},
        }
        local _M = {
          version = 0.1,
          priority = 10,
          name = "glia-errors",
          schema = schema,
        }

        function _M.check_schema(conf)
          return core.schema.check(schema, conf)
        end

        local function is_match()
          return ngx.status == 429
        end

        function _M.header_filter(conf, ctx)
          if is_match() then
            core.response.clear_header_as_body_modified()
          end
        end

        function _M.body_filter(conf, ctx)
          if is_match() then
            local body = core.response.hold_body_chunk(ctx)
            if not body then
              return
            end

            ngx.arg[1] = "<html><head><title>" .. ngx.status .. " " .. ngx.var.status_text .. "</title></head><body><center><h1>" .. ngx.status .. " " .. ngx.var.status_text ..  "</h1></center></html>\n"
          end
        end

        return _M

Note that this is just the preliminary code. It doesn't distinguish between apisix errors and upstream errors. But for 429 it is fine for us. Also it doesn't handle other >= 400 errors.