FX pairs modelling: make 06_linear and 07_gbm reproduce the populations they publish - #590
Merged
Conversation
case_studies/fx_pairs/config/setup.yaml declared modeling.gbm.device: gpu. The installed LightGBM 4.6.0 has no CUDA tree learner, so resolving any FX GBM request raised "LightGBM CUDA was requested but is unavailable" before a single model was planned. The adapter deliberately excludes OpenCL (device="gpu"), which does build here, because it is far slower and makes benchmarks misleading - so cpu is the backend this machine actually has. Same defect and same fix as crypto_perps_funding in ce91dfa (#533). old -> new -> cause: FX GBM planning raised RuntimeError -> plans and fits 12 checkpoints across ols, ridge_a0.001 and default_mse -> the declared device named a LightGBM build that was never installed, not a model change. Verified with a reduced real-data preview (1 fold, 6 pairs, isolated preview root): fresh fit 5.8s, cache replay reused fitted state, every checkpoint complete and identity_status=current, and the signal backtest composed from those catalog rows.
11_causal_dml resolved one label - setup.yaml's primary - while causal_dml is
configured for all three horizons and the released registry carries published
runs for all three. A canonical run therefore produced one member of a
three-member population, and nothing compared what it produced against what the
case study configures: the notebook agreed with itself, exited 0, and was short.
The same defect is in all nine case studies from the same template; 29 labels are
configured across the fleet and 9 would have been produced.
Also migrates the notebook to the shared research boundary alongside 06-10:
visible causal requests, resolution before estimation, per-label identity and
population reporting, and a check that the estimation covered every configured
label and that no two labels collapsed to one identity.
old -> new -> cause: one causal row for fwd_ret_1d -> three rows, one per
configured label -> the label was read from setup.yaml's primary rather than from
the configured menu, so two of three were never requested.
Produced and validated at this source:
fwd_ret_1d 6e17a9b4644c n_rows 59560
fwd_ret_5d e9623aa44d9a n_rows 59500
fwd_ret_21d f53540351b6b n_rows 59280
each with numerics {deterministic_reduction: true, thread_limit: 1} recorded by
#548, identity_version 3, resolved-spec/v1, provenance stamped, 5 of 5 cells
executed, no error output.
07_gbm resolved one label and registered one IC-chosen checkpoint per configuration. The menu configures 15 GBM presets for each of three horizons and each preset declares ten tree checkpoints, so the population is 45 trainings and 450 checkpoint prediction sets - each checkpoint being its own downstream configuration. Selecting one by IC collapsed 450 candidates to 45 before validation backtest Sharpe had seen any of them, and IC selects nothing. Migrated to the shared boundary alongside 06 and 08-11: visible model requests, plan_models resolving every training and checkpoint identity before the first fit, the official population written before fitting so a failed member is a gap rather than a shorter result, and the published catalog compared against the planned population rather than against a count. The plan is also compared against the configured menu as a set of (label, config_name) pairs, failing closed with the missing and unexpected members named. A count check passes on a right-sized plan with the wrong labels, which is how the single-label shape would have survived. Drops this notebook's KNOWN_BLOCKED entry in the OpenMP import-order gate. The rewritten notebook imports case_studies.utils.gbm and never imports scikit-learn, so the exemption no longer describes it, and the gate reports a stale entry as a failure precisely because a stale exemption is how a gate stops covering a file. Its recorded reason was a feature-artifact problem parked in an import-order list, which this run does not reproduce: all 450 checkpoints completed. old -> new -> cause: 45 prediction sets for fwd_ret_1d -> 450 across all three configured horizons -> the label came from setup.yaml's primary rather than the configured menu, and the checkpoint was chosen by IC rather than published whole. Produced and validated at this source: 450 prediction sets, official population 183ef5c80011 complete at 450 of 450, provenance stamped, no error output.
08, 09 and 10 each resolved one label and, for the deep families, chose one
checkpoint by IC. The menus configure three TabM presets and three sequence
architectures for each of three horizons, and every preset declares its epoch
checkpoint schedule, so the populations are 9 TabM trainings at 8 checkpoints
and 3 trainings at 20 checkpoints for each sequence architecture. A checkpoint
is part of a configuration, and IC selects nothing.
Migrated to the shared boundary alongside 06, 07 and 11: visible model requests,
plan_models resolving every identity before the first fit, the official
population written before fitting, and the published catalog compared against
the planned population rather than a count.
09 and 10 read the resolved specification through spec.get("computation", spec),
which works either side of the sequence family's move onto ResolvedSpec in #545,
and each names the configured architectures it does not run so a population that
is short is short by declaration rather than silently. lstm_h64 is configured for
all three labels, has a published training run for fwd_ret_1d in the released
registry, and no scoped fx notebook produces it; that is recorded in the cohort
gate evidence and in 12_model_analysis's exclusion set, not resolved here.
old -> new -> cause: one label and one IC-chosen checkpoint per configuration ->
72 TabM and 60 per sequence architecture across all three horizons -> the label
came from setup.yaml's primary rather than the configured menu, and the
checkpoint was selected rather than published whole.
Produced and validated at this source, populations complete rather than exit 0:
08_tabular_dl 72 of 72 7:14.64 uncontended, load median 2.85, GPU p50 38%
09_dl_tcn 60 of 60 23:35.14 uncontended, load median 4.45, GPU p50 72%
10_dl_nlinear 60 of 60 17:24.34 CONTENDED, not a bankable width
06_linear resolved one label, so a canonical run published 28 of the 84 configurations the menu declares across the three configured horizons. The allocation and cost stages select per label, so a one-label population starves them, and nothing compared what the notebook produced against what the case study configures. Migrated to the shared boundary alongside 07-11: visible model requests, plan_models resolving every training and checkpoint identity before the first fit, the official population written before fitting, and the published catalog compared against the planned population. The plan is also compared against the configured menu as a set of (label, config_name) pairs and fails closed with the missing and unexpected members named. Adds SUPERSEDES_POPULATION, empty by default. A population is written before the first fit, so an interrupted run leaves a snapshot with no artifacts under it; re-running is silent while the identities hold, and once they move the registry refuses to overwrite the earlier snapshot and names it. The value declares which snapshot this one replaces. It is deliberately reader-supplied, because supersedes is inside the hashed snapshot and so is part of what the population is, and because a hash baked into the source would make a reader's first run against a fresh registry fail at population.py:86. It is coerced to text: a population hash is a string, and an all-digit hash arriving through papermill's typed -p would be an integer that never matches the stored value. old -> new -> cause: 28 prediction sets for fwd_ret_1d -> 84 across all three configured horizons -> the label came from setup.yaml's primary rather than from the configured menu. Produced and validated at this source, population complete rather than exit 0: 84 of 84 in population 32480db48558, which supersedes 342446006141 - a snapshot left by a run killed before fitting. Stamped production=True with the parameter declared, under the allowlist added in #553.
12_model_analysis checked that every configured label was present and that rows
carried complete identities, but nothing compared the assembled population
against the menu the case study configures. A population short a configured
model is indistinguishable from a complete one: each notebook verifies it
produced what it requested, and no check asks whether the requests covered what
is configured.
The population is now compared to the configured menu minus an explicitly
declared exclusion set, failing closed with the missing and unexpected members
named. Exclusions are derived per label from that label's own menu rather than
listed, so a label that never declares a member does not gain a phantom
exclusion for it. deep_learning is the one family split across two notebooks, so
lstm_h64 - configured for all three labels, with a published training run for
fwd_ret_1d in the released registry, and produced by no scoped fx notebook - is
excluded by name with its reason. The run prints it rather than passing over it.
Also fixes a silent shape defect in the backtest handoff. `select(list, str,
str)` does not expand the list: polars collapses it to a single `literal`
column, the select SUCCEEDS, and the failure surfaces later at `.sort("label")`
with a column-not-found error that names the wrong cause. Corrected to
`select([*identity_columns, ...])`, verified against the three call forms.
old -> new -> cause: a population verified only against itself -> compared
against the configured menu, 147 configured, 3 excluded, 144 expected, 144
present, none missing and none unexpected -> nothing had compared plan to
configuration at any stage.
Produced and validated at this source: 726 catalog rows, all complete, all three
configured labels, causal DML absent from the predictive catalog, and the
declared exclusion reported by name.
Found by the pre-push review and by re-running what it forced. The prediction-agreement matrix rendered as a 4x4 grid of NaN, diagonal included, in a notebook that had been produced and validated. `actual` was in the pivot index, and the representative artifacts do not agree on its float representation, so the pivot split 41,260 keys into 82,473 rows in two disjoint groups - each score column populated on a different half, no overlapping observations, every correlation undefined. The index is now the canonical eligibility key alone, which yields the real matrix, and two guards were added that would have caught it: the pivot must align every representative on the same keys, and no representative may be missing scores where the others have them. NaN is not an error, so nothing in the run reported this. The checkpoint scatter silently dropped a whole family. Linear's checkpoint kind is `final` with no numeric value, so plotly discarded those rows without saying so. It now excludes them by name and states the count in the title. Three guards asserted things that could not be false, or could not be true: - `handoff.height != catalog.height` compared a frame against the frame it was sliced from, so it never fired. Removed; the menu-versus-population check does the real work. - `set(results) != set(labels)` in 11 iterated a dict built from `labels`, so it never fired. Removed. - `result.spec != resolved.spec` in 11 compared FULL specifications including `provenance`, which records the git commit of the registering run. It passes on a first run and fails on every re-run made after any commit - it asserted that nothing had been committed since, which is not a property of a causal estimate. Re-anchored to the identity-bearing `computation` block, which can differ for a reason that matters. This is why 11 failed on its first re-run. The review's fourth finding, that 08's parameter prose describes epochs and batch size as equivalent preview controls when only epochs select a preview, is NOT fixed here. Correcting a comment in 08 changes its .py, which stales its stamp and forces a re-run; 08's fitting cell carries 686 stream outputs that an idempotent re-run would discard, since a cache hit cannot re-emit stdout it never produced. Losing the record of nine trainings to correct a sentence is the wrong trade. It is recorded for the next occasion 08 legitimately re-runs. Produced and validated at this source: 12 re-run with a real agreement matrix (diagonal 1.0), 11 re-run clean across all three labels.
PR #560 moved every case study's model stages onto one shared boundary and deleted the private config-selection copies. FX 06_linear and 07_gbm had hand-rolled the same intent on this branch - "fit the menu across every configured label" - so both sides rewrote the notebooks and the conflict spans their whole body. main's version wins for 06 and 07. #560 generalises what the branch did: declared_labels defaults to every label whose training menu declares the family, so the population follows the menus rather than a per-case-study constant. Keeping the branch side would put FX back on the copy #560 removed, and #560 also fixes silent under-fitting - a mistyped configuration name raised nothing and quietly fit a smaller population. Kept from the branch: 08_tabular_dl through 12_model_analysis, which main does not have, and config/setup.yaml's modeling.gbm.device: cpu. That last one is still needed and is not superseded - LightGBM 4.6.0 here has no CUDA tree learner, so device: gpu raised before a single model was planned. 06 and 07 now need re-running under the merged boundary; their committed outputs were produced by the version this merge replaces.
`06_linear` and `07_gbm` fit one label per run, but published under a name
fixed per case study: `fx_pairs-linear-validation-v1`. A population is
immutable, and `OfficialPopulation.create` refuses a different member set
under an existing name unless the caller names the snapshot it supersedes,
which `run_model_population` has no parameter to pass. Running the notebook
for a second label therefore raised
a changed population named 'fx_pairs-linear-validation-v1'
must explicitly supersedes 6fbcf59c8c52
rather than adding to the population. The FX menus declare linear and GBM
configurations for all three labels, so two thirds of each declared population
could not be registered at all - and the same holds for the other eight case
studies, which share this code.
The name now carries the label. It is derived below the parameters cell rather
than in it: a parameterized run replaces `LABEL` in a cell papermill inserts
*after* the tagged one, so a name composed alongside `LABEL` would have carried
the default label into every other label's run - the failure would have been
one population silently overwritten by the next, not an error.
Passing `labels=None` to fit every declared label in a single run is not the
alternative: the catalog is joined to `configs` on `config_name` alone, so a
three-label catalog fanned 84 configurations into 252 rows, showed each name
three times, and computed `full_coverage` against the longest horizon's day
count, marking every row false. Measured, not reasoned about.
`pre_run_gate.py` no longer leaves `ML4T_OUTPUT_DIR` pointing at the preview
workspace it deletes. `check_the_run_is_costed` runs after it and reads the
modeling dataset on the no-recorded-cost branch, so the gate crashed with
`FileNotFoundError` on exactly the path that branch exists to serve. Verified
against `crypto_perps_funding`, which has no recorded GBM cost: the
extrapolation now completes.
Runs on a registry reset for the rebuild: linear 28 configurations registered
for `fwd_ret_1d`, gbm 15 configurations over 150 checkpoints.
All seven phase-1 notebooks run end to end on the registry reset for the
rebuild: `06_linear`, `07_gbm`, `08_tabular_dl`, `09_dl_tcn`, `10_dl_nlinear`,
`11_causal_dml`, `12_model_analysis`.
`12_model_analysis` is what made the label question concrete. It assembles the
population and compares it against the configured menu across every label, and
on the first attempt it stopped at cell 4:
RuntimeError: the assembled population does not match the configured menu;
missing [('fwd_ret_21d','gbm','leaves_15_huber'), ... 24 entries ...]
`07_gbm` had fitted `fwd_ret_1d` only. The other two horizons were then filled
by parameterized runs, which `nb-run.sh` executes on a copy outside the
worktree so the committed notebook stays the page a reader sees. The registry
now holds every declared configuration: linear 28, gbm 15, tabular_dl 3 and
deep_learning 2, each across all three labels.
The population name carries the label, which is what lets a second horizon
register at all - under one fixed name per family, `OfficialPopulation.create`
refuses the second run rather than adding to the first. Nothing downstream here
resolves a model population by name, so the rename has no consumer; `cme_futures`
and `sp500_options` do have one and keep their fixed names. The open question
that spans both is recorded in `work/2026-08-21-stocktake/PRE-RUN-BLOCKER.md`.
The `POPULATION_NAME` explanation moved from a comment block into a markdown
cell: the pre-run gate refuses a code cell that opens with three lines of prose.
06_linear fitted one label per run, under a population named for that label. Three labels therefore meant three runs and three populations, the variants were declared and never fitted unless someone remembered to run them, and nothing in the notebook compared the horizons the case study defines. The label menus exist to be fitted; this fits all of them in one run, under one population. What changes in the notebook: - LABEL becomes LABELS, defaulting to every label whose menu declares linear models. load_model_configs already defaults that way; the notebook was overriding it. - One population, fx_pairs-linear-validation-v1, covering all 84 configurations. The label-scoped name existed only to work around one-label-per-run. - The config join is on (config_name, label). On config_name alone the 28-name menu fanned 84 catalog rows into 252. - full_coverage compares ic_n_days against the label's own maximum. The three horizons offer 2,063, 2,059 and 2,043 scorable validation dates, so one global maximum marked every 5d and 21d configuration incomplete and emptied their panels. - The IC bar chart becomes one panel per label, configurations held in the primary label's order so the panels can be read against each other. The Ridge sweep becomes one curve per label on shared axes. - A new results cell reports the rank correlation between each pair of horizons. What it shows: all three horizons have the same penalty-response shape, negative at weak penalties, crossing zero near alpha=1e4 and peaking at 1e5-1e6. The size differs by much more than the penalty moves it - best IC 0.0065 at one day, 0.028 at five days and 0.039 at 21 days - so the traded horizon is the weakest of the three. Rank correlation between horizons runs 0.71 to 0.82 over the 26 configurations with full coverage everywhere, and the top three configurations differ at every horizon. No fit was recomputed: 84 configurations, 0 folds fitted, 672 reused. The identities were already registered from the per-label runs; only the population is new. primary_label(study) is added to case_studies/research so a notebook that fits every label can order a comparison by the one the strategy trades without re-parsing setup.yaml.
…read against Same change as 06_linear, for the same reason: the label menus declare three horizons and the notebook fitted one, under a population named for it. - LABEL becomes LABELS, defaulting to every label whose menu declares gbm. - One population, fx_pairs-gbm-validation-v1, 45 configurations and 450 checkpoints. - full_coverage and the final iteration are both computed per label, not against a global maximum, because the horizons offer different numbers of scorable dates. - Learning curves and the final-iteration bar chart become one panel per label, configurations in the primary label's order. The curve legend is one entry per loss function rather than fifteen names repeated three times. - The checkpoint-versus-model comparison becomes a frame with one row per label, reporting the ratio directly instead of two printed numbers. The new results cell reports each label's excess kurtosis, and it contradicts the explanation the section around it used to give. The loss ordering - Huber closest to zero, squared error furthest below - is clear on the five-day and 21-day labels and absent on the one-day label. The one-day label has excess kurtosis 59; the 21-day label 2.6. So the ordering is clearest where the tails are lightest, which is the opposite of what the heavy-tails argument predicts, and the previous text asserted that argument without measuring it. The section now says what is there: at one day every configuration sits within a hundredth of zero on the same side of it, so nothing separates them on any axis, and whether the loss function or merely the presence of some signal produces the separation at longer horizons is left open rather than answered. Deciding it needs a trimmed-tail comparison this grid does not run. 45 configurations fitted, all reused from the per-label runs; only the population is new.
06 and 07 now publish one population each covering all three labels rather than three populations covering one. The catalog 12 reads is unchanged in content - the same 84 linear and 450 gbm prediction sets - so this is the render catching up, not a change in what it found. The menu check passes: every configured (label, family, config_name) is present.
…chart Push review found three claims in the prose that the rendered results did not support, all of them written from looking at a panel rather than from a number: - The five-day best IC was given as 0.028. That belongs to two configurations the notebook itself excludes from the chart for scoring only 1,543 of the label's 2,059 dates; the charted five-day best is 0.024. The horizon comparison was mixing a partial-coverage number with two full-coverage ones. - The rank-correlation frame pivoted all three labels and dropped nulls once, so every pair was intersected down to the 26 configurations charted everywhere. fwd_ret_1d against fwd_ret_21d shares all 28 and now reports 28 (rho 0.681); the pairs involving five days report 26. - 'the top three configurations are a different three at each horizon' is now read from best_config in the frame rather than asserted. A new horizons frame carries charted, scorable_dates, best_config, best_ic, worst_ic and above_zero per label, and the section reads from it. Two robustness fixes to LABELS, which the change to all-labels introduced: - A run that narrows LABELS fits a different member set than the canonical population declares, and populations are immutable, so it now raises naming the parameter instead of being refused by the registry several cells later. - panel_labels is derived from the labels actually charted, with the primary first only when it is among them. It previously prepended the primary unconditionally, so a subset run without it produced an empty config_order and a pivot referencing a column that does not exist.
…t the tails frame at the development boundary The push review found the kurtosis frame reading each label's full parquet, which runs to 2025-12-30 and so includes the sealed 2024-2025 holdout: 10,360 of the 77,460 one-day rows. A validation-stage notebook was publishing a statistic measured partly on holdout outcomes, and section 5 argued from it. The rows are now cut at validation_end taken from the plan, which is the same development boundary the fits were resolved against. Development-only kurtosis is 61.3 / 10.0 / 2.4 against the 59.1 / 9.9 / 2.6 that included the holdout, so the argument is unchanged and the number a reader sees now comes from development data. Three prose claims contradicted the rendered results, each written from looking at a chart rather than from a number: - 'one configuration out of fifteen ends above zero' at one day: two do. - 'the ones that reach above zero are Huber and absolute error': at 21 days a squared-error configuration finishes at +0.0031. - 'more trees make it worse at every horizon': at five days the median configuration ends where it started, and about half the configurations peak at an interior checkpoint at every horizon. Two frames now carry those facts. trees_effect reports, per label, the median last-minus-first IC, how many configurations ended lower and how many peaked in the interior. objective_summary reports mean and best IC and the above-zero count per loss function per label. Reading them changes the loss-function conclusion. Squared error is last at every horizon, which is the ordering the heavy-tailed case studies report. But Huber leads at one day and 21 days while absolute error leads at five, and the gap between best and worst objective is widest at five days and narrowest at 21 - while tail weight orders one day far heaviest, five days next, 21 days lightest. The ordering survives; its size does not track tail weight the way the mechanism predicts. The section now says so and leaves it open, rather than asserting the tails explanation. Also: spread.head(15) showed only the primary label's block of a 45-row frame, so it now takes the five widest-ranging configurations per label; LABELS gets the same subset guard as 06_linear; panel ordering derives from the labels actually present.
… whole member set The blocking finding: the tails frame took a single global max of validation_end and cut all three labels there. The boundaries differ - 2023-12-28 at one day, 2023-12-21 at five, 2023-11-29 at 21 - because a longer forward window has to stop earlier to keep its outcome inside the development period. Cutting the two variant labels at the one-day date left decision dates whose forward window ends inside the sealed holdout, which is the leak the previous commit set out to close, and the prose claiming a per-label cut was false for two of three. Each label is now filtered by its own boundary: 67,080 / 67,000 / 66,680 rows, kurtosis 61.4 / 10.0 / 2.4. Four more from the same review: - The subset guard covered LABELS and not CONFIG_NAMES, which narrows the member set the same way. It now compares the loaded configurations against the complete declared catalog, so either knob is caught. - 'The horizon changes the size of the result by more than the penalty does' was refuted by the frame the paragraph pointed at: the 21-day grid spans 0.050 from worst to best, wider than the 0.033 between the best 21-day and best one-day configuration. The heading now claims only what the numbers support - the best result rises several-fold with the horizon - and the paragraph says outright that this does not rank the two choices. - The 1,543-of-2,059 figure appeared in no rendered output. The two partial-coverage configurations now have their own displayed frame carrying ic_n_days against the label's maximum. - 'Roughly half peak at an interior checkpoint, which is what noise produces' read against the wrong null: with ten checkpoints and no trend the argmax lands interior four times in five, so noise predicts about twelve of fifteen. Seven, eight and eight is below that, meaning peaks sit at the ends more often than noise would put them - which supports the drift the median change already reports. The text now says that. - The line-chart alt text said no one-day curve finishes above zero; two do, as the bar-chart alt text three cells later already said. Both alt texts drop the hardcoded panel count.
…ar finding forward as stated Two from the push review, both correct. The intro still told the reader that 06_linear had established the horizon matters more than anything chosen within a family. The previous commit withdrew exactly that claim there. It now forwards what 06_linear does support: the best result a grid reaches and the count above zero both rise with the horizon. The interior-peak baseline assumed the ten checkpoint readings were exchangeable, so the argmax would be uniform over positions and land interior four times in five. They are not exchangeable - checkpoint k+1 is the same model with more trees - and the maximum of a serially correlated path concentrates at its ends even with no drift at all. Seven, eight and eight of fifteen is therefore not evidence against that null. The count stays in the frame as a description, the paragraph says why it is not read as evidence, and the conclusion rests on the median change and the two-thirds that end lower, which measure drift directly.
… actually supports The intro said the linear notebook's horizon finding was carried forward 'to be checked' and nothing checked it. A horizons frame at the final iteration now does: 21 days reaches 0.0056 with five configurations above zero, while one day reaches 0.0015 and five days 0.0012, both with two above zero. So the linear notebook's pattern holds at the long horizon and not between one and five days, and section 5 says so rather than leaving a reader to discover the mixed answer. The stopping-rule sentence survived the retirement of the argument that supported it. Drift points the other way: if two thirds of configurations end below where they started, stopping at the first checkpoint would have beaten training to the last, which is something a rule could act on. What the drift does support is that 500 trees is not a tuned quantity - the grid fixes training length so capacity and loss are comparable, and the extra trees do not help that comparison. Choosing a stopping point on these curves would be choosing after seeing the validation folds, which is why the checkpoint travels into 13_backtest as part of what gets selected.
main carries this branch's own 06/07 work through the #554 squash, so every fx conflict is main holding the version the branch has since replaced. Resolved to the branch side throughout, and checked: for 08-12 the .py files are identical either way and only rendered outputs conflicted, and every line 06/07 take from the branch is a line the label change rewrote.
The three FX deep-learning notebooks defaulted to DEVICE = "cuda" and passed it through as an override. resolve_torch_device and _sequence_runtime_spec both refuse cuda on a host without it rather than falling back - deliberately, since a silent fallback would register results from a computation nobody asked for - so the default made these notebooks fail at the request-building cell for any reader without an NVIDIA card, and on a CPU CI runner. tests/overrides.yaml does not mark them gpu: true, unlike the CME entries, so that is exactly where CI runs them. DEVICE now defaults to empty and resolves to cuda when torch reports it available and cpu otherwise. An explicit "cuda" still raises where there is none, so the no-silent-fallback contract is intact for anyone who states a device; what changes is that not stating one no longer means demanding a GPU. The resolved value is printed, so no run leaves it implicit. All three re-run on this machine and print Device: cuda, so the production render is unchanged and every fit was reused.
…ersedes A population is a set of prediction identities, so anything that moves a training identity produces a different population under the same name. Correcting `max_bin` from the GPU-inherited 63 to LightGBM's CPU default is exactly that: the configuration menu is untouched and every member name is unchanged, but every prediction hashes differently. `OfficialPopulation.create` already refused such a snapshot unless told which one it replaces, and already recorded the answer in `supersedes_hash`. Nothing could reach that argument. The nine GBM sweeps therefore could not be re-run at all - `sp500_options` failed at the guard with "a changed population named 'sp500-options-gbm-validation-v1' must explicitly supersedes 6f061b802c3f" and no notebook parameter could supply it. `supersedes` now threads from `run_model_population` through `run_official_models` and `snapshot_official_models` to the create call. A preview is refused it: a preview population is discarded with its workspace, so accepting the argument would let a caller believe a snapshot was superseded when nothing was written. The other half was that superseding broke every reader. `OfficialPopulation.one` resolved a name by requiring exactly one snapshot to carry it, so the second generation made the name ambiguous and raised - and `12_model_analysis`, the backtest workflows and `run_official_models` itself all resolve populations by name. It now returns the one snapshot in the chain that nothing supersedes, which is the generation in force. Earlier snapshots stay readable by hash, which is what makes recording the lineage worth anything. Both are covered by what they let you do rather than by the argument existing: a second snapshot under a changed parameter is refused without the predecessor and accepted with it, the superseded members stay readable, and a three-generation chain still resolves by name to its head.
…it them unexecuted
13, 14, 15, 16 and 17 opened the study through Study.regenerate (13-16) or
Study.open (17). regenerate needs the features/, labels/ and run_log/ symlinks and
raises PermissionError where they are absent, so the notebooks could not run in CI
or in any workspace that is not the maintainer's. They now declare EXECUTION_TIER
and WORKSPACE in the parameters cell and call
study = open_study(CASE_STUDY_ID, execution_tier=EXECUTION_TIER, workspace=WORKSPACE or None)
which keeps the canonical path identical and falls back to Study.open(workspace=...)
for a preview run. tests/pm_helpers.py injects both parameters automatically when a
notebook declares them, so no overrides entry is needed.
07_gbm gains SUPERSEDES_POPULATION as a parameter, defaulting to the empty string
and so behaving exactly as before. The default still has to become the hash the
published population actually supersedes, the way etfs does it in 9765afa; that is
the remaining half of ml4t/agent-workspace#879 and is left for the run that fixes it.
The five phase-2 notebooks are committed CLEARED - outputs and stamps dropped - so
the source correction lands now and the execution follows. Before the gate change in
public#589 this commit was impossible: keeping the stale stamps read STALE, dropping
them read DE-STAMPED, and clearing either needed the production run this correction
is a prerequisite for.
… row count ml4t/agent-workspace#868. The canonical-population guard compared `configs.height` against the full declared catalog's height. A run that narrowed `LABELS` or `CONFIG_NAMES` to a different set of the same size would pass it and publish an entirely different member list under `fx_pairs-linear-validation-v1`. `narrows_declared_catalog` compares `(label, config_name)` pairs instead, which is what the markdown above the cell already claimed the check did. No such subset exists for fx as the menu stands - three labels, 28 linear configurations each, so any proper subset gives 28 or 56 against 84 - so nothing published under the old guard is in question. Re-executed: 84 configurations, 0 folds fitted, 672 reused, population resolves to the row already on record, d1b0c5a302f8. No number moved.
… and guard by member set Two fixes, one execution. ml4t/agent-workspace#879. SUPERSEDES_POPULATION defaulted to empty while the published population `fx_pairs-gbm-validation-v1` carries supersedes_hash 06e9ea03f2f2. `OfficialPopulation.create` hashes `supersedes` into the snapshot, so a run passing None computes a different hash and is refused against the row on record with "a changed population named ... must explicitly supersedes 06e9ea03f2f2": the committed notebook could not reproduce what it published. The lineage had been supplied at run time and never written into the source. It now defaults to 06e9ea03f2f2, with the markdown above the call saying why the default carries a hash. Confirmed against the supersedes_hash column of official_populations before applying. ml4t/agent-workspace#868. The canonical-population guard compared `configs.height` against the declared catalog's height, so a run narrowing LABELS or CONFIG_NAMES to a same-sized different set would pass it. `narrows_declared_catalog` compares (label, config_name) pairs, which is what the markdown above the cell already claimed. No such subset exists for fx as the menu stands - three labels, 150 gbm label-config pairs in total - so nothing published under the old guard is in question. Re-executed: 45 configurations fitted from cache, 450 prediction sets, population resolves to 23abc9ef1009, the row already on record. No number moved.
…correct the shared-fold claim Both markdown; no computed value moves. Synced with --update, so the executed outputs and the provenance stamp are unchanged. 1. THE DEFAULT LINEAGE BLOCKS A PREVIEW RUN. `SUPERSEDES_POPULATION` now defaults to 06e9ea03f2f2 and is passed unconditionally, but a preview population is discarded with its workspace, so it has no lineage to extend and research/execution.py:746 refuses it with "preview populations cannot supersede a snapshot". That is the gate before production. The routing fix belongs in `research_preview_parameters`, which already forces EXECUTION_TIER and WORKSPACE for a notebook declaring both, and etfs is landing it there rather than nine notebooks each branching on EXECUTION_TIER. What is this notebook's to carry is the page saying it: an operator running the gate by hand passes `SUPERSEDES_POPULATION=` alongside the reductions, learned here rather than from a traceback. 2. THE SHARED-FOLD CLAIM WAS FALSE ON THE PATH THIS NOTEBOOK TAKES. It said preparation happens once per fold and is shared by every configuration, and that the run walks folds outside and configurations inside. That describes the batch path taken by unresolved requests. research/execution.py:225-226 routes a ResolvedModelRequest to request.run() one at a time, and this notebook resolves every request first so the plan can be shown against the real data, so each configuration prepares its own folds. Now says which path shares, which path this takes, and why resolving first is the right trade on a cross-section this size and the wrong one on a large panel. Found by the sp500_equity_option_analytics session, reported through etfs; the same sentence is in six other 07_gbm notebooks and each owner corrects their own. fx 06_linear does not carry it.
…he canonical population Two defects, neither of which changes a number today. The notebook restated the sweep from setup.yaml's primary and variants rather than asking which labels declare a tabular_dl menu. Those are different questions - setup.yaml says which labels the case study carries, a training menu says what to fit for one of them - and they agree here only by coincidence, so a menu that dropped its tabular_dl section would have gone on being fitted silently. Membership now comes from declared_labels(). The order does not. declared_labels() returns menu-file order (1d, 21d, 5d) while setup.yaml's is 1d, 5d, 21d, and the population is named after its labels and hashed over its members as an ordered list: the published population is fx_pairs:fwd_ret_1d+fwd_ret_5d+fwd_ret_21d:tabular_dl, so taking menu order would have renamed it and demanded a supersedes for a run fitting the same nine models. Membership from the menus, order from the sweep, is what leaves the identity where it is. The notebook also had no way to publish a narrowed run. PRIMARY_LABEL fits one label of three, and the population name derived from it happened not to collide, but nothing stated the rule the other model notebooks enforce. POPULATION_NAME is now a parameter and narrows_declared_catalog() refuses a narrowed run that does not pass one, at the menu rather than several cells later in a message about hashes. Verified unchanged: labels ['fwd_ret_1d', 'fwd_ret_5d', 'fwd_ret_21d'], nine label-configuration pairs, narrows_declared_catalog False, population name fx_pairs:fwd_ret_1d+fwd_ret_5d+fwd_ret_21d:tabular_dl - identical to the published population, so the nine registered training rows still reuse. Outputs cleared pending re-execution; the fits reload from cache.
…ical population The same two defects the TabM notebook had, fixed the same way and with the same result: no number moves. Both restated the sweep from setup.yaml's primary and variants rather than asking which labels declare a deep_learning menu, which agrees here only by coincidence. Membership now comes from declared_labels(); the order stays setup.yaml's, because the population is named after its labels and hashed over them as an ordered list, and menu-file order would rename it. Neither could publish a narrowed run. narrows_declared_catalog() does not fit these two - each fits one architecture out of the deep_learning catalog, so it would report every canonical run as narrowing - so the check is on the knob that actually narrows here, the label set, with POPULATION_NAME as the way to publish one under its own name. Verified unchanged: labels ['fwd_ret_1d', 'fwd_ret_5d', 'fwd_ret_21d'], not narrowed, populations fx_pairs:fwd_ret_1d+fwd_ret_5d+fwd_ret_21d:tcn and :nlinear - identical to the published ones, so the registered training rows still reuse. Outputs cleared pending re-execution; the fits reload from cache.
…lare Six entries in tests/overrides.yaml named parameters that no longer exist: MAX_SYMBOLS on 13, 14, 15 and 16, and TOP_N_COMBOS on 15 and 16. The phase-2 rewrite renamed the knobs and the overrides file was not followed through. A binding that names nothing is not a dead line. Papermill injects the unused variable, nothing raises, and the reduced-tier run proceeds at full scale against the canonical registry while the entry reads as a reduction that was applied - which is how a preview passes and the production run then fails. MAX_SYMBOLS drops from all four: the rewritten notebooks have no symbol-count knob. TOP_N_COMBOS becomes MAX_COST_POINTS on 15_costs, which is the parameter that truncates the cost grid, and drops from 16_risk_management, where MAX_RISK_VARIANTS already binds the axis it was meant to cut. All four still route to the preview tier after the edit, since each retains at least one non-zero knob and include_preview is inferred from them. The binding sets are not finished. Every one of these notebooks iterates over labels and concatenates, and 14, 15 and 16 rebuild upstream backtest identities by convention rather than from what 13_backtest registered, so a preview run still reconstructs a TOP_K the upstream run did not use. That is the phase-2 fix pass; what changes here is only that no entry claims a reduction it is not making.
…ties rather than rebuild them Two defects, both of which made a reduced run look for results nothing wrote. `include_preview` was inferred from whether any reduction knob was non-zero, conflating how much of the registry a run covers with which namespace it reads and writes. Any reduced run therefore went looking for preview predictions, so a reduction applied over a canonical upstream - which is what the test suite exercises - resolved no rows at all. It now comes from `EXECUTION_TIER`, the same form `crypto_perps_funding/12_model_analysis.py` already uses. Because a canonical run may now legitimately be narrowed, `13_backtest` gains `POPULATION_NAME` and refuses to publish the canonical population from a run that narrows the sweep, matching the guard the model notebooks carry. `_preview_baselines` in `14` and `_preview_leader` in `15` and `16` rebuilt the upstream backtest identity from their own parameters instead of reading what the upstream run registered. `14` restated `13_backtest`'s `top_k`; `15` and `16` restated both a `top_k` and which allocator came first. A preview reduces each notebook independently, so those were guesses about another run's parameters, and a wrong guess does not report a disagreement - it computes a hash that was never written and fails looking for it. All three now read the registered rows by stage and prediction identity, which is the same read the canonical branches already do against their published populations, and raise a message naming the upstream notebook when nothing is there. The fix is deliberately not a matching `TOP_K: 2` in the stage-14 overrides entry. Two literals that agree today are what breaks the next time one is tuned. Not yet verified end to end: the seeded CI registry predates this schema - no `execution_tier` on `training_runs`, no `prediction_coverage` table - so every row resolves with a null tier and `complete = false` and all four notebooks still stop at the catalog filter. That is `tests/fixtures/seed_results.py`, outside this case study, and is filed separately.
… failing there The notebook raises unless RESEARCH_LOCK_HASH is supplied, its overrides entry was empty, and nothing in the repo supplies one, so test_case_study_pipeline failed on its first executable cell. The notebook is right to demand it. A research lock is the authorization token the one-shot holdout transaction produces, and the checks below it - that the lock names this candidate set, that its state is HOLDOUT_EVALUATED, that the deterministic validation selection matches what the lock recorded - are the point of the notebook. Seeding a fixture lock would hand CI a fabricated authorization and make those checks pass against something no holdout produced. So the entry says so: skip with the reason, which is the mechanism the file already uses for notebooks CI cannot supply the inputs for. It runs after the holdout workflow, in production, where a real lock exists.
…rrowed run Taking the tier from EXECUTION_TIER rather than from whether a reduction knob is set made a narrowed canonical run reachable, and these three publish their populations under `if not include_preview`. So a run reduced by TOP_K, TOP_N_PREDICTIONS, TOP_N_CONFIGS, MAX_COST_POINTS or MAX_RISK_VARIANTS would have registered a partial snapshot under the canonical name - allocation-backtests, cost-sensitivity-backtests, risk-overlay-backtests - and a population is immutable once written. This is the defect the previous commit guarded in 13_backtest and did not carry to the three notebooks downstream of it. Same guard, same escape: POPULATION_NAME is a parameter, a narrowed canonical run without one is refused at the top rather than several cells later in a message about hashes, and the preflight overrides entries now name their own populations. Found by looking for High-shaped defects on the branch before pushing it, rather than during a twenty-minute review.
…rites
POPULATION_NAME gave a narrowed run its own name for the population it publishes.
Nothing gave it its own name for the population it resolves, so 14, 15 and 16 read
their upstream through a hardcoded f"{CASE_STUDY_ID}:..." while publishing under the
isolated name. The two outcomes differ in the wrong direction: in a fresh workspace
the run raises, and in a workspace that already holds the canonical populations it
allocates over the full baselines and freezes that under a name saying it is
narrowed. The version that corrupts is the one that only occurs where the data
already exists, and generate_intermediates.py runs this chain at canonical tier.
research_name(case_study_id, suffix, scope=...) is now the single place a published
name is formed, and both the writing stage and the reading stage call it with the
same scope. With no scope it reproduces today's string exactly, so no published
population moves; with one, every name in the chain moves together.
Three further defects in the same four files, all of the same kind - a guard that
sees less than the run does:
LABEL narrows the catalog in all four notebooks and appeared in none of the four
narrowing guards, so a canonical run with LABEL set and no other knob published a
one-label population under the canonical name. Populations are immutable per name,
so the later full run could not publish at all.
POPULATION_NAME isolated the population and not the candidate sets. A narrowed run
still created fx_pairs:holdout-candidates and fx_pairs:{label}:* from reduced
members, a later full run registered a second current identity under those names,
and 17_strategy_analysis could no longer resolve them. Those names now scope too.
The upstream reads in 14, 15 and 16 filtered on stage, execution_tier and
prediction_hash but not on complete or identity_status, both of which the backtest
catalog carries. 15 and 16 take the first row of the sort, so one incomplete or
superseded row aborted a run with complete siblings; 14 let incomplete baselines
into baseline_results and could advance a configuration whose baseline never
finished.
tests/overrides.yaml sets research_preview: false on the four entries. The harness
injects EXECUTION_TIER="preview" and a fresh WORKSPACE for any notebook declaring
both, which is right for a model notebook - it fits something small and registers it
where it was told to. These four only read at that tier, and the per-notebook suite
runs each notebook alone, so they filtered for preview rows nobody had written.
9 tests. The cross-notebook one is the one that matters: a unit test of the helper
cannot see this defect, because both sides called a correct function and one of them
was not calling it at all. Reverting 14's read to its previous form fails it by name.
…lares At canonical tier the baseline sweep runs over every current prediction in the catalog, and TOP_K alone only limits the top_k values swept, not the predictions they are swept over. The preflight run therefore timed out at 600s having done real work rather than having failed - which is what the tier fix in bb98ec1 turned it into. TOP_N_PREDICTIONS: 2 binds the axis TOP_K does not. POPULATION_NAME is already set on this entry, so the narrowing guard is satisfied and the run publishes under its own scope. The timeout goes to 900s because the remaining work is real backtest execution, not because the run is expected to use it.
The four overrides entries each carried their own POPULATION_NAME, which was correct while POPULATION_NAME named only the population a notebook publishes and wrong the moment it also named the population a notebook reads. 13 published fx_pairs:preflight-baselines:equal-weight-baselines and 14 looked for fx_pairs:preflight-allocations:equal-weight-baselines, so the chain broke at the first hand-off - the same class of mismatch bb98ec1 set out to remove, moved one level up into the configuration. All four now take fx_pairs:preflight. A scope isolates a run of the chain from the canonical names; it is not a per-notebook namespace. The helper's docstring says so now, because nothing in the signature does. Measured: 13_backtest passes. 14, 15 and 16 fail on exactly the three names above, which is what identified this.
…the local catalog The canonical branch of each notebook resolved its members from the upstream population and then iterated over the labels in its own prediction catalog. Those are the same set only when the upstream run covered everything, so a narrowed preflight raised 'no equal-weight baselines resolved for fwd_ret_21d' - a label the catalog holds and the upstream run was configured not to produce. Rebuilding the label list locally is the same guess that reading the registered population exists to avoid: it reproduces the upstream narrowing by convention, and convention breaks the moment one side is tuned. The list now comes from the results actually resolved. Where the run is unscoped it is publishing under canonical names, and there a missing label is a real defect rather than a configured reduction. Both notebooks now check that case explicitly and name the two sets when they disagree, so the narrowed path is permissive and the canonical path is not.
14_portfolio_management timed out at 300s having done real work. Its grid is labels x top_k values x allocators, and the preflight entry bound none of the three: TOP_N_PREDICTIONS appears only in this notebook's narrowing guard and never reaches its catalog, so it declared a reduction it was not making - the same defect 4f1b8c7 removed for MAX_SYMBOLS, reintroduced by me when I carried the binding across from 13. TOP_K and TOP_N_CONFIGS are the knobs this notebook applies. TOP_N_PREDICTIONS is dropped: since 9ce2f11 the label and configuration coverage is inherited from the upstream population rather than recomputed here, so there is no local axis for it to cut. Timeouts go to 900s on 14, 15 and 16. The runs are executing backtests now instead of failing in the first cell, and 13 already needed the same.
…ied against 13_backtest filtered the prediction catalog on identity_status, split and complete, froze that as an official population, and verified it on the next line with require_complete. Those are two different definitions of complete. The catalog's is a five-way conjunction over registry state; require_complete opens each member and asks PredictionResult.complete, which additionally requires predictions.parquet on disk. A prediction that is current and complete in the registry but has no artifact therefore passed selection and failed verification - a population frozen and immediately rejected, with the reason two lines apart. Measured: 'official population a62cce2acde1 is incomplete: 3e849448c5a7:partial, 0c1a7042e325:partial', on exactly the two rows TOP_N_PREDICTIONS selected. artifact_available is the catalog column that carries this, and 12_model_analysis:78 already refuses a catalog row without it. 13 now filters on it, and 14, 15 and 16 do the same where they build a prediction catalog, since each of them hands its selection to a population or a backtest that will open the artifact. This is not a test-fixture accommodation. In production every prediction carries its artifact, so the filter changes nothing there; where an artifact is missing, freezing the row was never going to work.
…ot from the catalog Restoring every checkpoint of a selected configuration is deliberate - the markdown above it says so, and advancing only the checkpoint that ranked highest would be wrong. But the checkpoints were taken from this notebook's own prediction catalog, which under a narrowed upstream holds rows 13_backtest never produced a baseline for. The allocation grid then planned over predictions with no equal-weight sibling, and the pairing check twelve cells later failed with 'allocation c82856432177 resolved to 0 equal-weight siblings' - a mismatch introduced at selection and reported long after, in terms that do not name it. The restore is now over the predictions the resolved baselines actually carry. In a full canonical run the two sets are identical, so the filter removes nothing; where the run is unscoped and publishing canonical names, the equality is asserted rather than assumed, and names the count when it does not hold. Third instance of one mistake today: 14 rebuilding upstream state from local parameters rather than reading what was registered. First was the strategy identity, then the label set, now the checkpoint set.
…input The pairing check compared each allocation result against its equal-weight sibling under a projection that strips strategy.allocation, and treated everything left as a field the allocation must not move. input_identity.prices is not such a field. A moment allocator declares a warmup, load_backtest_prices_for leaves the start of the window unconstrained by that many periods, and the frame it returns digests differently from the baseline's - by design, with the extra prefix consumed by the rolling window and excluded from return aggregation. So the check reported the warmup as a changed input and would fire in production on inverse_vol, risk_parity, hrp, mvo and mvo_ledoit_wolf, not only under a narrowed preflight. The digest is now excluded exactly when the allocation declares a warmup, read from strategy_warmup_periods rather than by restating which allocators those are. Where the allocator declares none, both sides load the same window and the comparison is a real check that the allocation did not move the price input, so it still happens. The error also named no field. It now names the backtest, its sibling, and every dotted path that disagrees with both values - which is how this was diagnosed at all: 'input_identity.prices: baseline=a9ac1aff57b412c4 allocation=0f04e1082cb658b6' took one run, against three spent guessing at the previous message.
9bda5fe asked each spec about its own allocation. The allocation result has one and declares a warmup, so its projection dropped input_identity.prices; the equal-weight baseline has none, so its projection kept the key. The two then differed on whether the key was present rather than on its value, and the check failed with allocation=None - a comparison made unequal by the step meant to make it fair. The decision is taken once, from the allocation under test, and passed to both projections. Where the allocator declares no warmup neither side drops it and the digest is still compared, which is the case the check is for.
…preview branch Two findings from the gate, both correct. The artifact_available conjunct added in fb3869a is redundant and its comment is wrong. catalog.py:308-313 already defines complete as identity_status == current and c_status == complete and a prediction hash and the fold counts agreeing AND artifact.is_file() - the same expression artifact_available reports at :339. So the conjunct cannot remove a row, and the comment's claim that require_complete requires a parquet 'which the catalog's complete conjunction does not' is false. The real gap between the two definitions is the artifact-digest comparison at results.py:417-420, which was the fixture defect and is fixed there. I said when landing it that it was right on its own terms; it was not, and a wrong explanation in book source is worse than the redundant filter it justified. Both are gone from all four notebooks. The preview branches of 15 and 16 still enumerated their own catalog's labels and called _preview_leader for each. With 13 narrowed by TOP_N_PREDICTIONS the upstream covers fewer labels than the catalog holds, so each uncovered label raised 'no preview allocation backtests are registered' - reporting a reduction the run was configured to make as a missing upstream. They now take the labels from the predictions an upstream preview actually backtested, which is the rule the canonical branch already follows. That is the fourth place in this chain where one notebook rebuilt another's coverage locally, and the second where I fixed the canonical branch and left the preview branch holding the same defect.
The covered block restated the predicate _preview_leader uses, differing only by the prediction_hash term. That makes the two required to agree from here on: tighten one and covered admits a label whose backtests the other rejects, failing with the same 'no preview allocation backtests are registered' message the block was added to prevent. It also rebuilt the backtest catalog once per label plus once more. _registered_preview_allocations is now the one definition, read once and passed in. Measured: 4 passed on 13-16, production registry unchanged.
…e preview/workspace split
All three notebooks derived an execution tier from whether any reduction was set,
printed it, passed it to the request builders - and then opened the study with
Study.regenerate(CASE_STUDY_ID) regardless. That is the in-place production path.
A reduced run therefore read and wrote the case study's own artifacts, and in a
maintainer worktree, where case_studies/fx_pairs/{features,labels,run_log} link
to the shared artifacts, it wrote the published registry. It is the same shape as
the harness defect #602 fixed one level up, still present in the notebook source.
CI caught it from the other side: with no symlinks to regenerate into,
Study.regenerate refuses, and cs-fx_pairs failed on 08, 09, 10 and 11 with
"canonical regeneration requires generated-artifact directory symlinks", taking
12_model_analysis with it for want of registered validation predictions. Measured
on run 32775518868 at e913745: 5 failed, 11 passed, 1 skipped.
fx_pairs was the only case study left on that path. cme_futures and
crypto_perps_funding already take open_study in their 08-12; fx did not.
The tier is now a parameter rather than an inference, with WORKSPACE beside it,
which is the pair the harness reads to route a notebook to an isolated workspace.
Inferring the tier from the reductions was what allowed the two to disagree: the
notebook could know it was a preview and still open the production study. The two
are now checked against each other in both directions - a preview declaring no
reduction and a canonical run carrying one each raise, naming what is wrong.
The reduction dict moves above the open, because which study to open now depends
on it. No number moves: 08, 09 and 10 are committed unexecuted, so there are no
outputs to invalidate and no run to repeat.
11_causal_dml and 12_model_analysis carry outputs and are left for the
re-execution already queued behind the corrected causal holdout cutoff.
3 passed under tests/test_case_studies.py. All nine published registries compared
before and after: fx_pairs unchanged at 23/1/269/144/726/3, and the only delta
anywhere was etfs, from that case study's own concurrent papermill run.
…repo links to 11_causal_dml was the last Study.regenerate in this case study. It took the same shape as 08-10 - derive a tier from whether any reduction is set, pass that tier to the request builders, then open the production study regardless - so a reduced run wrote where the published run writes. It now takes open_study with the tier as a parameter and WORKSPACE beside it, and refuses the two ways the pair can disagree. 12_model_analysis had a different version of the same fault. Study.open(CASE_STUDY) with no workspace resolves through the repo case directory, which holds a registry only where a maintainer worktree has linked one there. That is why it passed here and failed in CI with "no current canonical validation predictions are registered": locally it was reading the published registry, and in CI it was reading nothing. Neither run measured what the notebook claims to check. With the workspace injected it reads the fixture registry, and that surfaces why it cannot run under the fixture at all: _trim_label_configs trims the seeded config directory while the seeded registry carries the full production population, so the guard reports 117 unexpected members - three labels over 13 gbm and 26 linear configurations - and nothing missing. The two are inconsistent by construction, so no reduction fixes it. It is skipped with that reason, the same class as 17_strategy_analysis. A preview tier is not the alternative: it reads no preview rows. The source change stands independently of the skip. Reading whatever the repo case directory happens to link to is wrong whether or not this notebook runs in CI. Both notebooks carried outputs from a run of the previous source, so both are cleared: they claim nothing about a run until one happens. 11_causal_dml has to be re-executed regardless - its committed numbers came from the calendar-subtracted holdout cutoff that #606 corrected - and 12_model_analysis reads what it publishes. Measured: CI run 32778564473 at 86bd1bc went from 5 failed to 2 failed, 14 passed, after 08-10 moved. Locally 11 passes and 12 skips. The full fx suite here is 4 failed, 11 passed, 2 skipped, and all four failures are the same defect reported in ml4t/agent-workspace#912: a canonical-tier study with an isolated workspace still reads the published registry through case_studies/fx_pairs/run_log. All four pass in CI, which has no such symlink. 07_gbm is the clearest instance - its SUPERSEDES_POPULATION default is applied only when the registry carries that supersedes_hash, so CI leaves it None and passes while the leaked read here applies it and the preview branch refuses it. All nine published registries compared before and after: identical.
Second merge of main tonight: the first was clean and the pre-push review rejected it on findings that were already on main, from #597, #601 and #607. Those are fixed in #610, so the range is clean now. One conflict, case_studies/fx_pairs/11_causal_dml.py, and it is a union rather than a choice. This branch made the execution tier an explicit parameter instead of inferring it from whether a reduction happens to be set; #610 added SUPERSEDES_CAUSAL and the supersedes_for() parser to the same cell and the same import line. Both are kept. The paired .ipynb auto-merged into invalid JSON - two conflict markers inside the notebook's own text - so it is rebuilt from the .py and cleared.
stefan-jansen
added a commit
that referenced
this pull request
Aug 25, 2026
#590 (fx_pairs) landed between merges and collided on case_studies/research/__init__.py: this branch's import line names supersedes_for_run, main's names research_name, and the merge kept only one. Both functions exist in population.py and both are already in __all__, so the resolution imports both.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The fx_pairs modelling stage, phase 1 of the stage-06 rebuild.
08_tabular_dlthrough
12_model_analysisare already onmain(#554, #566); what is left ofphase 1 is
06_linearand07_gbm, and both are here.The branch also carries the five phase-2 notebooks,
13_backtestthrough17_strategy_analysis, committed unexecuted with outputs and provenance stampcleared. They are not part of this stage and they are not finished - see
"What is still open" below. They are on the branch because that is where the
phase-2 work continues, and the register rows bind to their commits, so they are
not rebased or cherry-picked out.
06_linear and 07_gbm
Both published populations could not be reproduced by the notebooks that
published them. Both are re-executed here, and no number moved: every fold came
back from cache and each population resolved to the row already on record.
07_gbmcould not re-run at all (ml4t/agent-workspace#879).SUPERSEDES_POPULATIONdefaulted to empty while the published populationfx_pairs-gbm-validation-v1carriessupersedes_hash = 06e9ea03f2f2.OfficialPopulation.createhashessupersedesinto the snapshot, so a runpassing
Nonecomputes a different hash and is refused against the row onrecord with
a changed population named ... must explicitly supersedes 06e9ea03f2f2. The lineage had been supplied at run time and never written intothe source. It now defaults to that hash, confirmed against the
supersedes_hashcolumn before applying, with the markdown above the callsaying why a default carries a hash at all. Re-executed: 45 configurations from
cache, 450 prediction sets, population resolves to
23abc9ef1009.The canonical-population guard could be bypassed (ml4t/agent-workspace#868).
Both notebooks compared
configs.heightagainst the declared catalog's height,so a run narrowing
LABELSorCONFIG_NAMESto a different set of the samesize would pass the guard and publish a non-declared member set under the
canonical name.
narrows_declared_catalogcompares(label, config_name)pairs, which is what the markdown above the cell already claimed the check did.
No such subset exists for fx as the menu stands - three labels, 28 linear
configurations each - so nothing published under the old guard is in question.
06_linearre-executed: 84 configurations, 0 folds fitted, 672 reused,population resolves to
d1b0c5a302f8.Two false statements in
07_gbm. It said preparation happens once per foldand is shared by every configuration, and that the run walks folds on the
outside and configurations on the inside. That describes the batch path taken by
unresolved requests;
research/execution.py:225-226routes aResolvedModelRequesttorequest.run()one at a time, and this notebookresolves every request first so the plan can be shown against real data, so each
configuration prepares its own folds. It now says which path shares, which path
it takes, and what the trade is. And the page now tells an operator running a
reduced-scale run to pass
SUPERSEDES_POPULATION=alongside the reductions,because a preview population is discarded with its workspace and has no lineage
to extend. Found by the sp500_equity_option_analytics session and reported
through etfs; the same sentence is in six other
07_gbmnotebooks and eachowner corrects their own.
Markdown-only changes were synced with
--update, so the executed outputs andthe provenance stamps are untouched.
Phase 2: what the branch now carries
13_backtestthrough16_risk_managementare still committed unexecuted, butthey now run. They pass
tests/test_case_studies.pyat reduced scale, 4 passed,on three consecutive runs. Eleven commits,
bb98ec19todd16c703, and thefindings behind them:
A narrowed run isolated only what it wrote. Population and candidate-set
names were scoped on the write side and hard-coded on the read side, so a
reduced-scale run wrote to an isolated name and then read the canonical one.
research_name(case_study_id, suffix, *, scope="")incase_studies/research/population.pyis now the single place a name is built,and every read and write in
13-16goes through it.LABELjoined the fournarrowing guards.
tests/test_research_name_scoping.pyincludes a regex testacross the four notebooks that fails when a name bypasses it.
A scope names a run of the chain, not a notebook. The first version gave
each of the four its own
POPULATION_NAME, which reintroduced the samemismatch one level up:
14looked for what13had written under a differentscope. All four now share
fx_pairs:preflight.Three notebooks rebuilt state the upstream population already carried.
14-16derived their label set from the local catalog rather than from theresolved upstream population, and
14restored checkpoints from the catalograther than from the predictions the upstream baselines were actually built on.
Both now read from upstream; an unscoped run asserts the two agree.
An allocator's warmup read as a changed price input.
14compares abacktest against its sibling and refused when a non-allocation field differed.
strategy_warmup_periodschanges the price frame and thereforeinput_identity.prices, which is a consequence of the allocation, not adivergence. The exclusion is decided once from the allocation and applied to
both sides through
_non_allocation_projection(spec, *, drop_prices); anasymmetric first version made the two sides differ on the key's presence.
The error message was fixed after three guesses, which was two too many.
an allocation result changed a non-allocation strategy fieldnamed no field.Printing every differing dotted path found the real bug on the next run and
caught an incomplete fix on the one after.
One commit was reverted.
fb3869a4added anartifact_availablefilter anda comment justifying it, on the claim that catalog
completedoes not requirethe parquet.
catalog.py:312endsand artifact.is_file(). The claim wasinferred, not read; review disproved it by opening the file.
0cbadaf3revertsboth. A wrong explanation in book source is worse than the redundant filter it
justified.
Every notebook now opens the study its tier asks for
08_tabular_dl,09_dl_tcn,10_dl_nlinearand11_causal_dmleach derived anexecution tier from whether any reduction was set, printed it, passed it to the
request builders - and then called
Study.regenerate(CASE_STUDY_ID)regardless.That is the in-place production path, so a reduced run read and wrote the case
study's own artifacts, and in a maintainer worktree it wrote the published
registry. fx_pairs was the last case study on that path;
cme_futuresandcrypto_perps_fundingwere already onopen_study.CI found it from the other side. With no symlinks to regenerate into,
Study.regeneraterefuses, andcs-fx_pairsfailed on all four withcanonical regeneration requires generated-artifact directory symlinks, taking12_model_analysiswith it for want of registered validation predictions.The tier is now a parameter with
WORKSPACEbeside it, and the two are checkedagainst each other in both directions: a preview declaring no reduction and a
canonical run carrying one each raise, naming what is wrong. Inferring the tier
from the reductions was exactly what let the notebook know it was a preview and
still open the production study.
12_model_analysishad a quieter version of the same fault.Study.open(CASE_STUDY)with no workspace resolves through the repo case directory, which holds a registry
only where a maintainer worktree has linked one there. It passed locally by reading
the published registry and failed in CI by reading nothing, so neither run measured
what the notebook claims to check. It now takes the workspace, and that surfaced why
it cannot run under the fixture at all - see the skip below.
Measured across three CI runs on this branch: 6 failed at
dd16c703, 5 failed ate9137455, 2 failed at86bd1bc5, andcs-fx_pairsis green at2cbc4fd0:14 passed, 3 skipped.
What is still open, and what it blocks
17_strategy_analysisisskip: true: it needsRESEARCH_LOCK_HASHfrom theone-shot holdout transaction, and a lock exists only after that workflow has run
in production.
12_model_analysisisskip: true. It asserts the assembled canonicalpopulation equals the configured menu, and the fixture cannot present both
consistently:
_trim_label_configstrims the seeded config directory while theseeded registry carries the full production population, so the guard reports 117
unexpected members - three labels over 13 gbm and 26 linear configurations - and
nothing missing. Not a reduction that can be made to work, and a preview tier is
not the alternative because it reads no preview rows.
11_causal_dmland12_model_analysisare committed cleared and need oneexecution each - and that execution is currently blocked, see
ml4t/agent-workspace#916.
CausalResult.one(research/causal.py:52) keeps everycausal_runsrow for a label at the currentidentity_versionand tier andrequires exactly one, with no recency rule and no tiebreak.
causal_runshasnineteen columns and no
supersedes, and11_causal_dmlhas no parameter to passone, so a refit writes a second row and nothing downstream can select from that
point on - not recoverable by re-running. All three fx rows carry
identity_version = 3, which is current, so causal: count the pre-holdout buffer in observations, not calendar days #606 did not move them out of theway; refitting would break all three labels, and
12_model_analysis:390callsCausalResult.onedirectly. Confirmed against the fx registry, not inferred.11's previous numbers came from the calendar-subtracted holdoutcutoff that causal: count the pre-holdout buffer in observations, not calendar days #606 corrected; against the real fx label artifacts all three labels
leaked, 20, 60 and 160 rows for
fwd_ret_1d,fwd_ret_5dandfwd_ret_21d, thelast being 0.24% of 66,840. Contained by construction - the catalog is built from
prediction_setsand causal rows live incausal_runs, so a causal estimatecannot reach a selection.
production scale.
Four fx notebooks still fail locally and pass in CI:
07_gbm,14,15and16. All four are one defect, reported as ml4t/agent-workspace#912 - acanonical-tier study with an isolated workspace still reads the published registry
through
case_studies/fx_pairs/run_log, which CI does not have.07_gbmis theclearest instance: its
SUPERSEDES_POPULATIONdefault is applied only when theregistry carries that
supersedes_hash, so CI leaves itNoneand passes, whilethe leaked read here applies it and the preview branch refuses it. Nothing in this
PR works around that; the issue is the record.
Register
case_studies/fx_pairs/06_linear->fixedatf283a243.case_studies/fx_pairs/07_gbm->fixedat24d05f9e.All five phase-1 cohorts are
frozen=yes;model_analysiswas the last, and itfailed no check - a preview is not applicable to a canonical-only notebook that
asserts the assembled population against the configured menu.
Phase-2 notebooks are not registered as
fixed: they pass at reduced scale buthave not been executed at production scale.