feat(benchmarking): add Bayesian model fitting and CI workflow#6089
Draft
mfleader wants to merge 2 commits into
Draft
feat(benchmarking): add Bayesian model fitting and CI workflow#6089mfleader wants to merge 2 commits into
mfleader wants to merge 2 commits into
Conversation
mfleader
force-pushed
the
api-latency-analysis
branch
2 times, most recently
from
1631e38 to
c66502c
Compare
This was referenced Jun 12, 2026
mfleader
force-pushed
the
api-latency-analysis
branch
from
c66502c to
c3d0510
Compare
Adds a data collection pipeline under benchmarking/api_latency_comparison/ for comparing per-request API latency between two OGX versions. The orchestrator sets up git worktrees for each version, generates a randomized complete block design experiment matrix, starts servers with CPU pinning via mirakuru, runs Locust against each version, and records per-request response times. A third "comparison control" group runs the same code as comparison to catch false positives from environmental noise. First of two commits. Follow-up adds model fitting and a CI workflow. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew F Leader <mleader@redhat.com>
Fit a Wald (Inverse Gaussian) latency model with HSGP (Hilbert Space Gaussian Process) temporal adjustment via nutpie. Detect regressions using posterior predictive quantile decisions with false positive calibration. Add MCMC diagnostics and a daily CI workflow comparing latest release against main. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Matthew F Leader <mleader@redhat.com>
mfleader
force-pushed
the
api-latency-analysis
branch
from
c3d0510 to
05d9a6a
Compare
cdoern
added a commit
that referenced
this pull request
Jun 15, 2026
…sults (backport #6093) (#6117) ## Summary - The catch-all `except Exception` in `openai_search_vector_store` was silently swallowing backend errors and returning empty results with HTTP 200 - Clients had no way to distinguish "no matching documents" from "the search failed" - This masked bugs like the milvus-lite 3.0 `chunk_content` KeyError (#6089), where file_search appeared to work but retrieval silently returned nothing - Now re-raises the exception so it surfaces as a 500 to the client - Also fixes `test_openai_vector_store_with_chunks` which was passing `filters={"topic": "ai"}` (the old shorthand format) instead of the typed format `{"type": "eq", "key": "topic", "value": "ai"}` required since #4471. The error was silently swallowed, and the assertion loop over empty results never executed, so the test appeared to pass. Closes #6092 ## Test plan - Added unit test `test_search_vector_store_propagates_backend_errors` that verifies a `KeyError` from the backend propagates to the caller - Fixed `test_openai_vector_store_with_chunks` filter format so it actually exercises the filter path - Reproduced locally with milvus-lite 3.0: before the fix, search returns 200 with empty results; after, returns 500<hr>This is an automatic backport of pull request #6093 done by [Mergify](https://mergify.com). Signed-off-by: Derek Higgins <derekh@redhat.com> Co-authored-by: Derek Higgins <derekh@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Charlie Doern <cdoern@redhat.com>
mfleader
pushed a commit
to mfleader/llama-stack
that referenced
this pull request
Jun 15, 2026
…sults (ogx-ai#6093) ## Summary - The catch-all `except Exception` in `openai_search_vector_store` was silently swallowing backend errors and returning empty results with HTTP 200 - Clients had no way to distinguish "no matching documents" from "the search failed" - This masked bugs like the milvus-lite 3.0 `chunk_content` KeyError (ogx-ai#6089), where file_search appeared to work but retrieval silently returned nothing - Now re-raises the exception so it surfaces as a 500 to the client - Also fixes `test_openai_vector_store_with_chunks` which was passing `filters={"topic": "ai"}` (the old shorthand format) instead of the typed format `{"type": "eq", "key": "topic", "value": "ai"}` required since ogx-ai#4471. The error was silently swallowed, and the assertion loop over empty results never executed, so the test appeared to pass. Closes ogx-ai#6092 ## Test plan - Added unit test `test_search_vector_store_propagates_backend_errors` that verifies a `KeyError` from the backend propagates to the caller - Fixed `test_openai_vector_store_with_chunks` filter format so it actually exercises the filter path - Reproduced locally with milvus-lite 3.0: before the fix, search returns 200 with empty results; after, returns 500 --------- Signed-off-by: Derek Higgins <derekh@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Charlie Doern <cdoern@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds Bayesian model fitting, diagnostics, and a CI workflow on top of the experiment pipeline from #6088. Estimates changes in API response latency (mean, p50, p95, p99) with uncertainty intervals. The false positive detection runs a negative control (same code as comparison, run independently) to verify the experiment isn't producing spurious differences.
analysis/fit_resp_latency_model.py— Wald (Inverse Gaussian) model with HSGP (Hilbert Space Gaussian Process) temporal adjustment, fitted via nutpieanalysis/diagnostics.py— analysis quality assessment: estimation reliability, model assumptions, data fitanalysis/decisions.py— quantile decisions and false positive detectionanalysis/wald_numba.py— numba dispatch shim so nutpie can compile the Wald distributionanalysis/smoke_test.py— smoke test for the analysis dependencies (nutpie, NetCDF, LOO)analysis/MODEL.md— model specification.github/workflows/response-latency-regression-benchmark.yml— daily CI comparing latest release vs mainTest Plan
test_benchmark.pyruns the full pipeline (experiment + model fitting) and verifies analysis artifacts:decisions.csv,fp-results.json, andidata.nc.