bug: event worker failed: failed to receive the header , attempt to index upvalue 'ev' (a nil value)

[hebbaa]

Current Behavior

Seeing below error in our production apisix data-plane logs

2025/11/23 10:46:52 [error] 54#54: *643711487 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer 2025/11/23 10:46:52 [error] 116#116: *643711540 lua entry thread aborted: runtime error: /usr/local/openresty/lualib/resty/events/compat/init.lua:47: attempt to index upvalue 'ev' (a nil value) 2025/11/23 10:46:52 [error] 52#52: *643711538 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer

Expected Behavior

These error logs should not be there.

Error Logs

2025/11/23 10:46:52 [error] 54#54: *643711487 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer 2025/11/23 10:46:52 [error] 116#116: *643711540 lua entry thread aborted: runtime error: /usr/local/openresty/lualib/resty/events/compat/init.lua:47: attempt to index upvalue 'ev' (a nil value) 2025/11/23 10:46:52 [error] 52#52: *643711538 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer

Steps to Reproduce

Dont know. Happening in production deployment.

Environment

• APISIX version (run apisix version):3.14.1

• Operating system (run uname -a):Linux data-plane-7c4b955754-dp552 5.10.230-223.885.amzn2.x86_64 change: added doc of how to load plugin. #1 SMP Tue Dec 3 14:36:00 UTC 2024 x86_64 GNU/Linux

• OpenResty / Nginx version (run openresty -V or nginx -V):nginx version: openresty/1.27.1.2 (x86_64-pc-linux-gnu)
  built by gcc 15.2.0 (Wolfi 15.2.0-r3)
  built with OpenSSL 3.6.0 1 Oct 2025
  TLS SNI support enabled

[hebbaa]

Looks like the above issue is occurring from killing of one or multiple worker nodes.
dmesg | grep -i "killed"
[1742140.134417] Memory cgroup out of memory: Killed process 1454261 (openresty) total-vm:637692kB, anon-rss:377044kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:904kB oom_score_adj:970
[1743469.967848] Memory cgroup out of memory: Killed process 1454264 (openresty) total-vm:637220kB, anon-rss:375624kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:904kB oom_score_adj:970
[1748701.318216] Memory cgroup out of memory: Killed process 1454262 (openresty) total-vm:636236kB, anon-rss:375532kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:896kB oom_score_adj:970
[1763367.552343] Memory cgroup out of memory: Killed process 198704 (openresty) total-vm:633328kB, anon-rss:373304kB, file-rss:10348kB, shmem-rss:20792kB, UID:636 pgtables:888kB oom_score_adj:970

[hebbaa]

mem request and mem limit for the conatiner is set at 2GB

[Baoyuantop]

Hi @hebbaa, could you please provide your APISIX configuration file? What is your APISIX deployment structure?

[hebbaa]

Our deployment is decoupled.

We have data-plane, ingress control plane, etcd.

Please find the overriden conf. A

deployment:                                                                                                                                                                                                                                                                        
  role: data_plane                                                                                                                                                                                                                                                                 
  role_data_plane:                                                                                                                                                                                                                                                                 
    config_provider: etcd                                                                                                                                                                                                                                                          
  etcd:                                                                                                                                                                                                                                                                            
    host:                                                                                                                                                                                                                                                                          
      - "https://etcd-0.etcd-headless.apisix.svc.cluster.local:2379"                                                                                                                                                                                                               
      - "https://etcd-1.etcd-headless.apisix.svc.cluster.local:2379"                                                                                                                                                                                                               
      - "https://etcd-2.etcd-headless.apisix.svc.cluster.local:2379"                                                                                                                                                                                                               
    prefix: /apisix                                                                                                                                                                                                                                                                
    tls:                                                                                                                                                                                                                                                                           
      cert: "/usr/local/apisix/certs/client.crt"                                                                                                                                                                                                                                   
      key: "/usr/local/apisix/certs/client.key"                                                                                                                                                                                                                                    
nginx_config:                                                                                                                                                                                                                                                                      
  worker_processes: 4                                                                                                                                                                                                                                                              
  worker_connections: 4096                                                                                                                                                                                                                                                         
  worker_rlimit_nofile: 65535                                                                                                                                                                                                                                                      
  http:                                                                                                                                                                                                                                                                            
    keepalive_timeout: "60s"                                                                                                                                                                                                                                                       
    send_timeout: "300s"                                                                                                                                                                                                                                                           
    proxy_connect_timeout: "90s"                                                                                                                                                                                                                                                   
    proxy_send_timeout: "300s"                                                                                                                                                                                                                                                     
    proxy_read_timeout: "300s"                                                                                                                                                                                                                                                     
    access_log_format: |                                                                                                                                                                                                                                                           
         { "time": "$time_iso8601", "remote_addr": "$remote_addr", "request_id": "$request_id", "host": "$http_host", "request_uri": "$request_uri", "request_method": "$request_method", "status": "$status", "body_bytes_sent": "$body_bytes_sent", "request_time": "$request_tim
  http_server_configuration_snippet: |                                                                                                                                                                                                                                             
    error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 421 422 423 424 425 426 428 429 431 451 500 501 502 503 504 505 506 507 508 510 511 @custom_error;                                                                                      
                                                                                                                                                                                                                                                                                   
    location @custom_error {                                                                                                                                                                                                                                                       
      internal;                                                                                                                                                                                                                                                                    
      default_type text/html;                                                                                                                                                                                                                                                      
      content_by_lua_block {                                                                                                                                                                                                                                                       
        local status = ngx.status or 500                                                                                                                                                                                                                                           
        local messages = {                                                                                                                                                                                                                                                         
          [400] = "Bad Request",                                                                                                                                                                                                                                                   
          [401] = "Unauthorized",                                                                                                                                                                                                                                                  
          [403] = "Forbidden",                                                                                                                                                                                                                                                     
          [404] = "Not Found",                                                                                                                                                                                                                                                     
          [500] = "Internal Server Error",                                                                                                                                                                                                                                         
          [502] = "Bad Gateway",                                                                                                                                                                                                                                                   
          [503] = "Service Unavailable",                                                                                                                                                                                                                                           
          [504] = "Gateway Time-out"                                                                                                                                                                                                                                               
        }                                                                                                                                                                                                                                                                          
        local msg = messages[status] or "Unexpected Error"                                                                                                                                                                                                                         
        ngx.say("<body><center><h1>" .. status .. " " .. msg .. "</h1></center></body>")                                                                                                                                                                                           
      }                                                                                                                                                                                                                                                                            
    }                                                                                                                                                                                                                                                                              
  stream_configuration_snippet: |                                                                                                                                                                                                                                                  
      upstream redis {                                                                                                                                                                                                                                                             
          server apisix-redis.apisix.svc.cluster.local:6380;                                                                                                                                                                                                                       
      }                                                                                                                                                                                                                                                                            
      server {                                                                                                                                                                                                                                                                     
          listen 127.0.0.1:6379;                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                   
          proxy_pass redis;                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                   
          proxy_ssl on;                                                                                                                                                                                                                                                            
          proxy_ssl_certificate /usr/local/apisix/certs/redis-client.crt;                                                                                                                                                                                                          
          proxy_ssl_certificate_key /usr/local/apisix/certs/redis-client.key;                                                                                                                                                                                                      
          proxy_ssl_trusted_certificate /usr/local/apisix/certs/ca.crt;                                                                                                                                                                                                            
      }                                                                                                                                                                                                                                                                            
apisix:                                                                                                                                                                                                                                                                            
  node_listen: 9080                                                                                                                                                                                                                                                                
  enable_heartbeat: true                                                                                                                                                                                                                                                           
  enable_report: false                                                                                                                                                                                                                                                             
  enable_debug: false                                                                                                                                                                                                                                                              
  enable_admin: false                                                                                                                                                                                                                                                              
  extra_lua_path: /usr/local/apisix/custom_plugins/?.lua;                                                                                                                                                                                                                          
  proxy_mode: "http&stream"                                                                                                                                                                                                                                                        
  stream_proxy:                                                                                                                                                                                                                                                                    
    tcp:                                                                                                                                                                                                                                                                           
      - addr: "127.0.0.1:6379"                                                                                                                                                                                                                                                     
      - addr: "0.0.0.0:8883"                                                                                                                                                                                                                                                       
      - addr: "0.0.0.0:60000"                                                                                                                                                                                                                                                      
  proxy_protocol:                                                                                                                                                                                                                                                                  
    listen_http_port: 9081                                                                                                                                                                                                                                                         
    listen_https_port: 9082                                                                                                                                                                                                                                                        
  ssl:                                                                                                                                                                                                                                                                             
    enable: true                                                                                                                                                                                                                                                                   
    listen:                                                                                                                                                                                                                                                                        
      - port: 9443                                                                                                                                                                                                                                                                 
    fallback_sni: <removed intentionally >                                                                                                                                                                                                                                       
    ssl_trusted_certificate: /usr/local/apisix/certs/ca.crt                                                                                                                                                                                                                        
    ssl_ciphers:  <removed intentionally >                
                                                                                                                                                                                                                                                                                   
plugins:                                                                                                                                                                                                                                                                           
    - prometheus                                                                                                                                                                                                                                                                   
    - fault-injection                                                                                                                                                                                                                                                              
    - serverless-pre-function                                                                                                                                                                                                                                                      
    - serverless-post-function                                                                                                                                                                                                                                                     
    - file-logger                                                                                                                                                                                                                                                                  
    - redirect                                                                                                                                                                                                                                                                     
    - workflow                                                                                                                                                                                                                                                                     
    - mobiheaders-plugin                                                                                                                                                                                                                                                           
    - mobiredirect-plugin                                                                                                                                                                                                                                                          
    - mi-xsrf-token-filter                                                                                                                                                                                                                                                         
    - request-id                                                                                                                                                                                                                                                                   
    - forward-auth                                                                                                                                                                                                                                                                 
    - mi-x509-details                                                                                                                                                                                                                                                              
    - mi-auth-cookie-to-header                                                                                                                                                                                                                                                     
    - mi-forward-by-cookie                                                                                                                                                                                                                                                         
    - mi-forward-by-ddm-request                                                                                                                                                                                                                                                    
    - mi-path-decoder-filter                                                                                                                                                                                                                                                       
    - cors                                                                                                                                                                                                                                                                         
    - mi-signed-cookie-validate                                                                                                                                                                                                                                                    
    - proxy-rewrite                                                                                                                                                                                                                                                                
    - response-rewrite                                                                                                                                                                                                                                                             
plugin_attr:                                                                                                                                                                                                                                                                       
  prometheus:                               # Plugin: prometheus attributes                                                                                                                                                                                                        
    export_uri: /apisix/prometheus/metrics  # Set the URI for the Prometheus metrics endpoint.                                                                                                                                                                                     
    metric_prefix: apisix_                  # Set the prefix for Prometheus metrics generated by APISIX.                                                                                                                                                                           
    enable_export_server: true              # Enable the Prometheus export server.                                                                                                                                                                                                 
    export_addr:                            # Set the address for the Prometheus export server.                                                                                                                                                                                    
      ip: 0.0.0.0                        # Set the IP.                                                                                                                                                                                                                             
      port: 9091                                                                                                                                                                                                                                                                   
- conf/config.yaml 115/115 100%

[hebbaa]

The issue is because of OOM one of the worker got killed

dmesg | grep -i "killed"
[1742140.134417] Memory cgroup out of memory: Killed process 1454261 (openresty) total-vm:637692kB, anon-rss:377044kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:904kB oom_score_adj:970
[1743469.967848] Memory cgroup out of memory: Killed process 1454264 (openresty) total-vm:637220kB, anon-rss:375624kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:904kB oom_score_adj:970
[1748701.318216] Memory cgroup out of memory: Killed process 1454262 (openresty) total-vm:636236kB, anon-rss:375532kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:896kB oom_score_adj:970
[1763367.552343] Memory cgroup out of memory: Killed process 198704 (openresty) total-vm:633328kB, anon-rss:373304kB, file-rss:10348kB, shmem-rss:20792kB, UID:636 pgtables:888kB oom_score_adj:970

from apisix data-plane logs
"2025/11/23 14:29:08 [alert] 1#1: worker process 214 exited on signal 9".
After the above looks like master is trying to start new worker process . During its startup the below errors seen

2025/11/23 10:46:52 [error] 54#54: *643711487 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer 2025/11/23 10:46:52 [error] 116#116: *643711540 lua entry thread aborted: runtime error: /usr/local/openresty/lualib/resty/events/compat/init.lua:47: attempt to index upvalue 'ev' (a nil value) 2025/11/23 10:46:52 [error] 52#52: *643711538 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer

Issue to investigate is why the new worker process didn't start cleanly .

[Baoyuantop]

Hi @hebbaa, have you configured the following settings? This error log is related to this event module. If this problem can be consistently reproduced, please provide the steps to reproduce it.

apisix:
  events:
    module: "lua-resty-events"

[hebbaa]

No I have not configured .

I have already mentioned the the cause for the issue. You can consider that as a step to reproduce.

Simple step.

1. Make OOM error "[1742140.134417] Memory cgroup out of memory: Killed process 1454261 (openresty) total-vm:637692kB, anon-rss:377044kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:904kB oom_score_adj:970
   "
2. master will kill the process
3. New process will not come up cleanly .