Skip to content

Conversation

gruceo
Copy link
Contributor

@gruceo gruceo commented Mar 13, 2025

We have identified a ~5% RPS performance regression in the Kong EE PR 11452

Upon further investigation, we found that a big chunk of CPU time is spent on ngx.var variables access.

We add upstream timing variables to the default indexed variables list, allowing efficient access through Kong's FFI interface. These variables track detailed timing information for upstream connections:

  • upstream_start_timestamp_us
  • upstream_connect_timestamp_us
  • upstream_request_timestamp_us
  • upstream_header_timestamp_us
  • upstream_response_timestamp_us

These microsecond-precision timestamps enable detailed upstream latency analysis and troubleshooting of connection issues.

@gruceo gruceo force-pushed the add-more-upstream-vars-to-default-index branch from c4b5566 to fde2099 Compare March 13, 2025 23:31
@chronolaw
Copy link
Contributor

chronolaw commented Mar 13, 2025

A temporary solution is using directive lua_kong_load_var_index in nginx.conf manually.

@gruceo
Copy link
Contributor Author

gruceo commented Mar 13, 2025

After we merged this nginx patch in OSS, a test is failing consistently: https://github.com/Kong/lua-kong-nginx-module/actions/runs/13846604773/job/38746240223#step:8:42

#   Failed test 't/011-get_last_peer_connection_cached.t TEST 1: sanity - grep_error_log_out (req 0)'
#   at /usr/local/share/perl/5.34.0/Test/Nginx/Socket.pm line 1220.
# @@ -1,3 +1,2 @@
#  last_peer_connection_cached 0
#  last_peer_connection_cached 0
# -last_peer_connection_cached 1
# Looks like you failed 1 test of 2.

It seems related to the patch in the sense that the new timestamp variables we added are causing the upstream state to be reset or modified in a way that affects connection caching behavior because the test expects to see a cached connection being used. @chronolaw @samugi

@samugi
Copy link
Member

samugi commented Mar 14, 2025

looks to be unrelated to the nginx patch, probably related to the OpenResty bump - moving the conversation internally

@gruceo gruceo force-pushed the add-more-upstream-vars-to-default-index branch from fde2099 to 1c7d91d Compare March 14, 2025 12:07
We have identified a ~5% RPS performance regression in the PR Kong/kong-ee#11452

Upon further investigation, we found that a big chunk of CPU time is
spent on ngx.var variables access.

We add upstream timing variables to the default indexed variables list,
allowing efficient access through Kong's FFI interface. These variables
track detailed timing information for upstream connections:

* upstream_start_timestamp_us
* upstream_connect_timestamp_us
* upstream_request_timestamp_us
* upstream_header_timestamp_us
* upstream_response_timestamp_us

These microsecond-precision timestamps enable detailed upstream latency
analysis and troubleshooting of connection issues.
@samugi samugi force-pushed the add-more-upstream-vars-to-default-index branch from 1c7d91d to b7b0bda Compare March 14, 2025 13:39
@gruceo gruceo merged commit fc29d26 into master Mar 15, 2025
6 checks passed
@gruceo gruceo deleted the add-more-upstream-vars-to-default-index branch March 15, 2025 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants