Skip to content

Conversation

@desertaxle
Copy link
Member

@desertaxle desertaxle commented Dec 22, 2025

Summary

  • Fixes an issue where the TASK_SOURCE cache policy fails when used with remote execution decorators like @ecs that use cloudpickle to ship code to remote environments
  • inspect.getsource() fails on cloudpickled functions because the original source file doesn't exist on the remote machine
  • The previous fallback to hashing __code__.co_code (bytecode) led to inconsistent cache keys because bytecode varies across operating systems and architectures
  • Stores the function's source code on the Task object during initialization so it survives cloudpickle serialization
Changes

src/prefect/tasks.py

  • Added source_code attribute to Task.__init__() that captures inspect.getsource(fn) during task initialization
  • Gracefully handles TypeError and OSError when source code is unavailable (e.g., callable objects)

src/prefect/cache_policies.py

  • Updated TaskSource.compute_key() to check for stored source_code attribute first
  • Falls back to inspect.getsource() for backward compatibility when source_code is not available

Tests

  • Added TestTaskSourceCode class in tests/test_tasks.py with tests for source code capture and cloudpickle survival
  • Updated tests/test_cache_policies.py with tests for the new stored source code behavior

🤖 Generated with Claude Code

@desertaxle desertaxle changed the title Fix TASK_SOURCE cache policy for remote execution with cloudpickle Fix TASK_SOURCE cache policy for remote execution with cloudpickle Dec 22, 2025
@codspeed-hq
Copy link

codspeed-hq bot commented Dec 22, 2025

CodSpeed Performance Report

Merging #19926 will degrade performance by 22.92%

Comparing fix/task-source-cache-policy-cloudpickle (84cd604) with main (7d73faa)

Summary

❌ 1 (👁 1) regression
✅ 1 untouched

Benchmarks breakdown

Benchmark BASE HEAD Efficiency
👁 bench_task_decorator 463.3 µs 601.1 µs -22.92%

@desertaxle desertaxle force-pushed the fix/task-source-cache-policy-cloudpickle branch from 71988fe to 904ffaa Compare December 22, 2025 17:08
This PR fixes an issue where the `TASK_SOURCE` cache policy fails when
used with remote execution decorators like `@ecs` that use cloudpickle
to ship code to remote environments.

The problem: `inspect.getsource()` fails on cloudpickled functions
because the original source file doesn't exist on the remote machine.
The previous fallback to hashing `__code__.co_code` (bytecode) led to
inconsistent cache keys because bytecode varies across Python versions
and the code object contains unstable attributes like `co_locals` that
change between serialization/deserialization cycles.

The solution: Store the function's source code on the `Task` object
during initialization so it survives cloudpickle serialization. The
`TaskSource.compute_key()` method now checks for this stored source
code first before falling back to `inspect.getsource()`. This ensures
stable, deterministic cache keys regardless of execution environment.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@desertaxle desertaxle force-pushed the fix/task-source-cache-policy-cloudpickle branch from 904ffaa to 02c13aa Compare December 22, 2025 17:21
@desertaxle desertaxle marked this pull request as ready for review December 22, 2025 17:46
Copy link
Collaborator

@zzstoatzz zzstoatzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you think the codspeed report is just flapping or a real regression?

@desertaxle
Copy link
Member Author

do you think the codspeed report is just flapping or a real regression?

I think it's a real regression casued by the new inspect.getsource call, but it's a difference of <150 microseconds, so I think it's acceptable.

@desertaxle desertaxle merged commit 0d60bc1 into main Dec 22, 2025
89 of 90 checks passed
@desertaxle desertaxle deleted the fix/task-source-cache-policy-cloudpickle branch December 22, 2025 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants