Make particle Gibbs deterministic across architectures

[penelopeysm]

Fixes #108

The problem was in this function split:

AdvancedPS.jl/src/rng.jl

Lines 33 to 41 in 379336b

	"""
	split(key::Integer, n::Integer=1)
	Split `key` into `n` new keys
	"""
	function split(key::Integer, n::Integer=1)
	T = typeof(key) # Make sure the type of `key` is consistent on W32 and W64 systems.
	return T[hash(key, i) for i in UInt(1):UInt(n)]
	end

Here, UInt(1) gives UInt64 on 64-bit systems., and UInt32 on 32-bit. Thus, the result of hash was different, despite ensuring that we convert back to the same type at the end.

The 'simple' solution of replacing UInt with UInt64 doesn't work, because on 32-bit systems, hash(::UInt64, ::UInt64) is not defined.

This PR therefore uses a pseudo-random number generator to deterministically generate new values of type T from the original value.

[coveralls]

Pull Request Test Coverage Report for Build 12891599012

Details

• 3 of 3 (100.0%) changed or added relevant lines in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.5%) to 96.098%

---

Totals
Change from base Build 9928521956:	0.5%
Covered Lines:	394
Relevant Lines:	410

---

💛 - Coveralls

[penelopeysm]

I had to bump this to get the test to pass.

[penelopeysm]

It's not possible to put in a test that checks that it's the same result across architectures (we could compare to a fixed value, but that would be very brittle). But this at least demonstrates that sampling a Turing model with PG() now gives the same results across architectures: https://github.com/penelopeysm/Shaymin.jl/actions/runs/12891706831

[mhauru]

Great work figuring this out! This has been bugging me, too, for a while now.

[yebai]

@FredericWantiez implemented this splittable pseudo-random number generator (PRNG). The splitting step requires some attention since not all splitting is valid. See ¹.

There is a package for this splittable PRNGs, which we could depend on: https://github.com/Julia-Tempering/SplittableRandoms.jl

See, also: https://github.com/janestreet/splittable_random

Footnotes

1. Steele, G. L., Lea, D., & Flood, C. H. (2014). Fast splittable pseudorandom number generators. Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications, 453–472. ↩

[penelopeysm]

Thank you, @yebai, I was reading this over lunch :)

For context, the RNG stream being used right now is Philox which is described here: https://www.thesalmons.org/john/random123/papers/random123sc11.pdf

I'm not really certain that our original implementation of split is correct either, though - is it? It seems to me that (unlike, e.g., Numpy, https://numpy.org/doc/2.1/reference/random/bit_generators/philox.html) the Random123.jl package doesn't include the necessary functionality to split a Philox RNG, and this split function seems to me to be quite ad hoc too as the algorithm isn't mentioned in the paper.

So it seems to me that this is at least no worse, and maybe we could rework the RNG system in another PR, perhaps? Let me know what you think.

[yebai]

Does SplittableRandoms.jl provide enough functionality for our purpose? If so, it's okay to depend on it -- it is a very lightweight package without any dependencies other than the Julia standard library.

[FredericWantiez]

Thanks @penelopeysm for the fix. I don't think there was an implementation of split at the time, probably best to switch to SplittableRandoms.jl now

[penelopeysm]

I'm in favour of switching over, and I think that it does have exactly the right functionality, but looking at AdvancedPS right now, it seems to me that that would require a fairly substantial reworking of the whole rng.jl file.

I'd prefer to merge this and open another issue for that (unless there are concerns over this PR itself, of course).

[FredericWantiez]

We could also tweak split here to be somewhat closer to the jax implementation for example, but that's probably going to abuse the Random123 interface.
https://github.com/jax-ml/jax/blob/main/jax/_src/prng.py#L1114

[yebai]

@penelopeysm feel free to merge and open an issue for it.