Add DI integration tests

[gdalle]

Proposed answer to #2528 for DifferentiationInterface

• Remove existing integration tests from CI.yml (Bijectors, DynamicExpressions) and put them inside a new Integration.yml.
• Add tests for DifferentiationInterface, along with a documentation on how to interpret them.
• Make sure that all tests pass:
  • The failure in the second-order tests is known and reported (Enzyme v0.13.69 broke second-order differentiation with constant matrix #2523)
  • The failure in the closures tests is also known and reported (Duplicated function failure with matrix view #2532)
• Increase time limit (to see how long the first run takes), pass -O1 option to the julia command to reduce compilation time (the main culprit for the duration of Enzyme's DI tests).

[github-actions]

Your PR no longer requires formatting changes. Thank you for your contribution!

[wsmoses]

@gdalle I think this is too long to be included. Different machines will have different limits, but is this doable within 20 minutes

[gdalle]

I understand, I was setting this to a high value for prototyping, hoping to see how long it takes to run on your custom runner.
In the DI test suite with the default runners, the Enzyme part takes close to an hour, but maybe here the machine is more bulky? Or we can leverage multiprocessing to speed things up (how many cores are there)?
Once I have an order of magnitude of how expensive this is, it will tell me what I need to cut to bring it down under 20 min.

[giordano]

how many cores are there

32, but they help only with precompilation, if tests are fully serial.

[gdalle]

If I parallelize tests, will the JIT compilation be parallelized too? In my understanding, this is only true if I distribute the tasks to separate worker processes (using Distributed), and not just to separate threads (using Base.Threads)?

[gdalle]

I have started working on reducing compilation time in DI tests with SnoopCompile, there are probably gains to be made there too

[giordano]

Yeah, probably a Distributed-based test setup like CUDA.jl would make great use of the parallelism we have available here, it's very unfortunate there isn't anything standardised for parallel testing at the moment.

[giordano]

In the meantime I don't think 20 minutes are enough for all jobs at the moment, maybe something like 45 (or 60, just be safe) would be better than 120.

[avik-pal]

it's very unfortunate there isn't anything standardised for parallel testing at the moment.

I have been using ReTestItems.jl in Lux and a few SciML repos for over a year now. it generally works great. (https://github.com/LuxDL/Lux.jl/actions/runs/17257270161/job/48971604321#step:9:644). Though it requires pretty much rewriting your entire existing test suite

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.92%. Comparing base (54de339) to head (08484a1).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2531   +/-   ##
=======================================
  Coverage   74.92%   74.92%           
=======================================
  Files          56       56           
  Lines       17428    17428           
=======================================
  Hits        13058    13058           
  Misses       4370     4370           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[giordano]

I think the testing setup is pretty good as a starting point (which means my job is done, yay), but there are testing failures (should be marked as broken for the time being?)

[gdalle]

I'll take a look at the test failures