`InitContext`, part 2 - Move `hasvalue` and `getvalue` to AbstractPPL; enforce key type of `AbstractDict` #980

penelopeysm · 2025-07-09T21:32:48Z

Part 1: Adding hasvalue and getvalue to AbstractPPL

This is Part 2/N of splitting #967 up into manageable bits.

In Part 1 we defined the following methods in AbstractPPL:

hasvalue(::NamedTuple, ::VarName[, ::Distribution])
hasvalue(::AbstractDict{<:VarName}, ::VarName[, ::Distribution])
getvalue(::NamedTuple, ::VarName[, ::Distribution])
getvalue(::AbstractDict{<:VarName}, ::VarName[, ::Distribution])

This PR removes them (and their helper functions) from the DynamicPPL codebase.

In DynamicPPL, their type bounds for Dict arguments were too lax: they used to take AbstractDict{<:Any} as the first argument. The AbstractPPL versions have tighter bounds. This PR therefore switches all overly-loosely-typed Dicts to use a key type of VarName.

Closes #791

github-actions · 2025-07-09T21:34:32Z

Benchmark Report for Commit `9b87e02`

Computer Information

Julia Version 1.11.6
Commit 9615af0f269 (2025-07-09 12:58 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Benchmark Results

|                 Model | Dimension |  AD Backend |      VarInfo Type | Linked | Eval Time / Ref Time | AD Time / Eval Time |
|-----------------------|-----------|-------------|-------------------|--------|----------------------|---------------------|
| Simple assume observe |         1 | forwarddiff |             typed |  false |                 12.7 |                 1.4 |
|           Smorgasbord |       201 | forwarddiff |             typed |  false |                736.1 |                37.3 |
|           Smorgasbord |       201 | forwarddiff | simple_namedtuple |   true |                526.8 |                45.8 |
|           Smorgasbord |       201 | forwarddiff |           untyped |   true |               1228.0 |                28.5 |
|           Smorgasbord |       201 | forwarddiff |       simple_dict |   true |               7802.3 |                26.8 |
|           Smorgasbord |       201 | reversediff |             typed |   true |               1124.4 |                36.6 |
|           Smorgasbord |       201 |    mooncake |             typed |   true |               1096.8 |                12.1 |
|    Loop univariate 1k |      1000 |    mooncake |             typed |   true |               6731.2 |                52.3 |
|       Multivariate 1k |      1000 |    mooncake |             typed |   true |               1009.0 |                 8.7 |
|   Loop univariate 10k |     10000 |    mooncake |             typed |   true |              80977.4 |                50.7 |
|      Multivariate 10k |     10000 |    mooncake |             typed |   true |               8580.8 |                 9.6 |
|               Dynamic |        10 |    mooncake |             typed |   true |                140.6 |                23.9 |
|              Submodel |         1 |    mooncake |             typed |   true |                 17.4 |                26.4 |
|                   LDA |        12 | reversediff |             typed |   true |               1192.3 |                 1.8 |

codecov · 2025-07-09T21:46:46Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.91%. Comparing base (05cd886) to head (9b87e02).
Report is 1 commits behind head on breaking.

Additional details and impacted files

@@             Coverage Diff              @@
##           breaking     #980      +/-   ##
============================================
- Coverage     82.07%   81.91%   -0.17%     
============================================
  Files            38       38              
  Lines          4066     4041      -25     
============================================
- Hits           3337     3310      -27     
- Misses          729      731       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-07-09T21:48:14Z

DynamicPPL.jl documentation for PR #980 is available at:
https://TuringLang.github.io/DynamicPPL.jl/previews/PR980/

This reverts commit 709dc9e.

mhauru

Looks good except I don't understand the need for Pkg.update.

test/runtests.jl

penelopeysm · 2025-07-25T13:38:44Z

(Will merge once CI passes as agreed on Slack)

* Bump minor version to 0.37.0 * Accumulators, stage 1 (#885) * Release 0.36 * AbstractPPL 0.11 + change prefixing behaviour (#830) * AbstractPPL 0.11; change prefixing behaviour * Use DynamicPPL.prefix rather than overloading * Remove VarInfo(VarInfo, params) (#870) * Unify `{untyped,typed}_{vector_,}varinfo` constructor functions (#879) * Unify {Untyped,Typed}{Vector,}VarInfo constructors * Update invocations * NTVarInfo * Fix tests * More fixes * Fixes * Fixes * Fixes * Use lowercase functions, don't deprecate VarInfo * Rewrite VarInfo docstring * Fix methods * Fix methods (really) * Draft of accumulators * Fix some variable names * Fix pointwise_logdensities, gut tilde_observe, remove resetlogp!! * Map rather than broadcast Co-authored-by: Tor Erlend Fjelde <[email protected]> * Start documenting accumulators * Use Val{symbols} instead of AccTypes to index * More documentation for accumulators * Link varinfo by default in AD testing utilities; make test suite run on linked varinfos (#890) * Link VarInfo by default * Tweak interface * Fix tests * Fix interface so that callers can inspect results * Document * Fix tests * Fix changelog * Test linked varinfos Closes #891 * Fix docstring + use AbstractFloat * Fix resetlogp!! and type stability for accumulators * Fix type rigidity of LogProbs and NumProduce * Fix uses of getlogp and other assorted issues * setaccs!! nicer interface and logdensity function fixes * Revert back to calling the macro @addlogprob! * Remove a dead test * Clarify a comment * Implement split/combine for PointwiseLogdensityAccumulator * Switch ThreadSafeVarInfo.accs_by_thread to be a tuple * Fix `condition` and `fix` in submodels (#892) * Fix conditioning in submodels * Simplify contextual_isassumption * Add documentation * Fix some tests * Add tests; fix a bunch of nested submodel issues * Fix fix as well * Fix doctests * Add unit tests for new functions * Add changelog entry * Update changelog Co-authored-by: Hong Ge <[email protected]> * Finish docs * Add a test for conditioning submodel via arguments * Clean new tests up a bit * Fix for VarNames with non-identity lenses * Apply suggestions from code review Co-authored-by: Markus Hauru <[email protected]> * Apply suggestions from code review * Make PrefixContext contain a varname rather than symbol (#896) --------- Co-authored-by: Hong Ge <[email protected]> Co-authored-by: Markus Hauru <[email protected]> * Revert ThreadSafeVarInfo back to Vectors and fix some AD type casting in (Simple)VarInfo * Improve accumulator docs * Add test/accumulators.jl * Docs fixes * Various small fixes * Make DynamicTransformation not use accumulators other than LogPrior * Fix variable order and name of map_accumulator!! * Typo fixing * Small improvement to ThreadSafeVarInfo * Fix demo_dot_assume_observe_submodel prefixing * Typo fixing * Miscellaneous small fixes * HISTORY entry and more miscellanea * Add more tests for accumulators * Improve accumulators docstrings * Fix a typo * Expand HISTORY entry * Add accumulators to API docs * Remove unexported functions from API docs * Add NamedTuple methods for get/set/acclogp * Fix setlogp!! with single scalar to error * Export AbstractAccumulator, fix a docs typo * Apply suggestions from code review Co-authored-by: Penelope Yong <[email protected]> * Rename LogPrior -> LogPriorAccumulator, and Likelihood and NumProduce * Type bound log prob accumulators with T<:Real * Add @addlogprior! and @addloglikelihood! * Apply suggestions from code review Co-authored-by: Penelope Yong <[email protected]> * Move default accumulators to default_accumulators.jl * Fix some tests * Introduce default_accumulators() * Go back to only having @addlogprob! * Fix tilde_observe!! prefixing * Fix default_accumulators internal type * Make unflatten more type stable, and add a test for it * Always print all benchmark results * Move NumProduce VI functions to abstract_varinfo.jl --------- Co-authored-by: Penelope Yong <[email protected]> Co-authored-by: Tor Erlend Fjelde <[email protected]> Co-authored-by: Hong Ge <[email protected]> * Replace PriorExtractorContext with PriorDistributionAccumulator (#907) * Implement values_as_in_model using an accumulator (#908) * Implement values_as_in_model using an accumulator * Make make_varname_expression a function * Refuse to combine ValuesAsInModelAccumulators with different include_colon_eqs * Fix nested context test * Bump DynamicPPL versions * Fix merge (1) * Add benchmark Pkg source * [no ci] Don't need to dev again * Disable use_closure for ReverseDiff * Revert "Disable use_closure for ReverseDiff" This reverts commit 3cb47cd. * Fix LogDensityAt struct * Try not duplicating * Update comment pointing to closure benchmarks * Remove `context` from model evaluation (use `model.context` instead) (#952) * Change `evaluate!!` API, add `sample!!` * Fix literally everything else that I broke * Fix some docstrings * fix ForwardDiffExt (look, multiple dispatch bad...) * Changelog * fix a test * Fix docstrings * use `sample!!` * Fix a couple more cases * Globally rename `sample!!` -> `evaluate_and_sample!!`, add changelog warning * Mark function as Const for Enzyme tests (#957) * Move submodel code to submodel.jl; remove `@submodel` (#959) * Move submodel code to submodel.jl * Remove `@submodel` * Fix missing field tests for 1.12 (#961) * Remove 3-argument `{_,}evaluate!!`; clean up submodel code (#960) * Clean up submodel code, remove 3-arg `_evaluate!!` * Remove 3-argument `evaluate!!` as well * Update changelog * Improve submodel error message * Fix doctest * Add error hint for three-argument evaluate!! * Improve API for AD testing (#964) * Rework API for AD testing * Fix test * Add `rng` keyword argument * Use atol and rtol * remove unbound type parameter (?) * Don't need to do elementwise check * Update changelog * Fix typo * DebugAccumulator (plus tiny bits and pieces) (#976) * DebugContext -> DebugAccumulator * Changelog * Force `conditioned` to return a dict * fix conditioned implementation * revert `conditioned` bugfix (will merge this to main instead) * fix show * Fix doctests * fix doctests 2 * Make VarInfo actually mandatory in check_model * Re-implement `missing` check * Revert `combine` signature in docstring * Revert changes to `Base.show` on AccumulatorTuple * Add TODO comment about VariableOrderAccumulator Co-authored-by: Markus Hauru <[email protected]> * Fix doctests --------- Co-authored-by: Markus Hauru <[email protected]> * VariableOrderAccumulator (#940) * Turn NumProduceAccumulator into VariableOrderAccumulator * Add comparison methods * Make VariableOrderAccumulator use regular Dict * Use copy rather than deepcopy for accumulators * Minor docstring touchup * Remove unnecessary use of NumProduceAccumulator * Fix split(VariableOrderAccumulator) * Remove NumProduceAcc from Debug * Fix set_retained_vns_del! --------- Co-authored-by: Penelope Yong <[email protected]> * Accumulators stage 2 (#925) * Give LogDensityFunction the getlogdensity field * Allow missing LogPriorAccumulator when linking * Trim whitespace * Run formatter * Fix a few typos * Fix comma -> semicolon * Fix `LogDensityAt` invocation * Fix one last test * Fix tests --------- Co-authored-by: Penelope Yong <[email protected]> * Implement more consistent tracking of logp components via `LogJacobianAccumulator` (#998) * logjac accumulator * Fix tests * Fix a whole bunch of stuff * Fix final tests * Fix docs * Fix docs/doctests * Fix maths in LogJacobianAccumulator docstring * Twiddle with a comment * Add changelog * Fix accumulator docstring * logJ -> logjac * Fix logjac accumulation for StaticTransformation * Fix behaviour of `set_retained_vns_del!` for `num_produce == 0` (#1000) * `InitContext`, part 2 - Move `hasvalue` and `getvalue` to AbstractPPL; enforce key type of `AbstractDict` (#980) * point to unmerged AbstractPPL branch * Remove code that was moved to AbstractPPL * Remove Dictionaries with Any key type * Fix bad merge conflict resolution * Fix doctests * Point to [email protected] This reverts commit 709dc9e. * Fix doctests * Fix docs AbstractPPL bound * Remove stray `Pkg.update()` * Accumulator miscellanea: Subset, merge, acclogp, and LogProbAccumulator (#999) * logjac accumulator * Fix tests * Fix a whole bunch of stuff * Fix final tests * Fix docs * Fix docs/doctests * Fix maths in LogJacobianAccumulator docstring * Twiddle with a comment * Add changelog * Simplify accs with LogProbAccumulator * Replace + with accumulate for LogProbAccs * Introduce merge and subset for accs * Improve acc tests * Fix docstring typo. Co-authored-by: Penelope Yong <[email protected]> * Fix merge --------- Co-authored-by: Penelope Yong <[email protected]> * Minor tweak to changelog wording --------- Co-authored-by: Penelope Yong <[email protected]> Co-authored-by: Tor Erlend Fjelde <[email protected]> Co-authored-by: Hong Ge <[email protected]>

penelopeysm added 2 commits July 9, 2025 22:13

point to unmerged AbstractPPL branch

709dc9e

Remove code that was moved to AbstractPPL

a19c9a6

github-actions bot assigned penelopeysm Jul 9, 2025

penelopeysm changed the base branch from main to breaking July 9, 2025 21:33

This was referenced Jul 9, 2025

InitContext, part 3 - Introduce InitContext #981

Open

The big InitContext PR. Do not merge! #967

Closed

InitContext, part 4 - Use init!! to replace evaluate_and_sample!!, predict, returned, and initialize_values #984

Draft

penelopeysm added 3 commits July 10, 2025 12:59

Remove Dictionaries with Any key type

f00879b

Fix bad merge conflict resolution

af7c6fc

Fix doctests

21cf568

penelopeysm force-pushed the py/hasgetvalue branch from 90e95e1 to 21cf568 Compare July 10, 2025 11:59

penelopeysm changed the title ~~Move hasvalue and getvalue to AbstractPPL; enforce key type of AbstractDict{VarName} instances~~ InitContext II - Move hasvalue and getvalue to AbstractPPL; enforce key type of AbstractDict{VarName} instances Jul 10, 2025

penelopeysm changed the title ~~InitContext II - Move hasvalue and getvalue to AbstractPPL; enforce key type of AbstractDict{VarName} instances~~ InitContext, part 2 - Move hasvalue and getvalue to AbstractPPL; enforce key type of AbstractDict{VarName} instances Jul 10, 2025

penelopeysm changed the title ~~InitContext, part 2 - Move hasvalue and getvalue to AbstractPPL; enforce key type of AbstractDict{VarName} instances~~ InitContext, part 2 - Move hasvalue and getvalue to AbstractPPL; enforce key type of AbstractDict Jul 10, 2025

This was referenced Jul 10, 2025

Release v0.37 #901

Merged

InitContext, part 5 - Remove SamplingContext, SampleFrom{Prior,Uniform}, {tilde_,}assume #985

Draft

penelopeysm added 3 commits July 16, 2025 17:54

Point to [email protected]

f0e686f

This reverts commit 709dc9e.

Merge branch 'breaking' into py/hasgetvalue

936f9a1

Fix doctests

e406068

penelopeysm requested a review from mhauru July 16, 2025 16:57

penelopeysm marked this pull request as ready for review July 16, 2025 17:09

penelopeysm added 2 commits July 16, 2025 18:10

Fix docs AbstractPPL bound

1bc303b

Merge branch 'breaking' into py/hasgetvalue

93c1dd5

mhauru reviewed Jul 25, 2025

View reviewed changes

test/runtests.jl Outdated Show resolved Hide resolved

penelopeysm added 2 commits July 25, 2025 14:34

Remove stray Pkg.update()

4b66e73

Merge branch 'breaking' into py/hasgetvalue

9b87e02

mhauru approved these changes Jul 25, 2025

View reviewed changes

penelopeysm merged commit c6c0cbc into breaking Jul 25, 2025
19 of 21 checks passed

penelopeysm deleted the py/hasgetvalue branch July 25, 2025 14:22

penelopeysm mentioned this pull request Aug 7, 2025

key type of SimpleVarInfo is underfined #791

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

`InitContext`, part 2 - Move `hasvalue` and `getvalue` to AbstractPPL; enforce key type of `AbstractDict` #980

`InitContext`, part 2 - Move `hasvalue` and `getvalue` to AbstractPPL; enforce key type of `AbstractDict` #980

Uh oh!

penelopeysm commented Jul 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 9, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jul 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

mhauru left a comment

Uh oh!

Uh oh!

penelopeysm commented Jul 25, 2025

Uh oh!

Uh oh!

Uh oh!

InitContext, part 2 - Move hasvalue and getvalue to AbstractPPL; enforce key type of AbstractDict #980

InitContext, part 2 - Move hasvalue and getvalue to AbstractPPL; enforce key type of AbstractDict #980

Uh oh!

Conversation

penelopeysm commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Report for Commit 9b87e02

Computer Information

Benchmark Results

Uh oh!

codecov bot commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

mhauru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

penelopeysm commented Jul 25, 2025

Uh oh!

Uh oh!

Uh oh!

`InitContext`, part 2 - Move `hasvalue` and `getvalue` to AbstractPPL; enforce key type of `AbstractDict` #980

`InitContext`, part 2 - Move `hasvalue` and `getvalue` to AbstractPPL; enforce key type of `AbstractDict` #980

penelopeysm commented Jul 9, 2025 •

edited

Loading

github-actions bot commented Jul 9, 2025 •

edited

Loading

Benchmark Report for Commit `9b87e02`

codecov bot commented Jul 9, 2025 •

edited

Loading