[RFC]: Support asynchronous put and get

[amy-why-3459]

Motivation.

Synchronous put and get may cause a performance bottleneck in high concurrency scenarios.

Proposed Change.

Please provide the detailed design document of the RFC using the template.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Shirley125]

Code Refactor:

1. Refactor SharedMemoryConnector

• Remove all request-specific state, including:put_requests, get_requests, request_payload, and other request-bound metadata

• After refactoring, the connector: Only performs data transport operations (put / get)

2. Introduce a new class: OmniChunkManager
If use async chunk feature, it manages the chunk lifecycle.

• chunk ID management,
• chunk key construction
• assembling chunked business data.
• async puting chunk/getting chunk

The following logic will be extracted from Scheduler:

• process_pending_chunks
• restore_queues
• filter_scheduler_output

In Scheduler.schedule(), only interface calls are retained. For example:

# Before super().schedule()
self.chunk_processor.process_pending_chunks(self.waiting, self.running)

# After super().schedule()
self.chunk_processor.restore_queues(self.waiting, self.running)
self.chunk_processor.filter_scheduler_output(scheduler_output)

Transfer Manager Comparison

This document summarizes the commonalities and differences between the
Chunk Transfer Manager and the KV Transfer Manager.

Commonalities (Shared Points)

Dimension	Chunk Transfer Manager	KV Transfer Manager
Connector usage	Uses connector based on config	Same
Transfer key	Builds key from stage_id / req_id / etc.	Same
Core flow	Has both save & load flows	Same
Final interface	Calls connector.put / connector.get	Same

Differences

Dimension	Chunk Transfer Manager	KV Transfer Manager
Pre-processing	Requires model input processor to prepare payload	Requires KV cache reshape/extract
Transfer mode	Asynchronous (threads + pending queue)	Synchronous (direct call)
Scheduler trigger	Must ensure recv/send before every scheduling cycle	Only when prefill finishes or special token triggers

Sequence Diagram:

flowchart：

[Bhanu068]

Hello @Shirley125 ,

I noticed you are working on refactoring the chunk handling mechanism this overlaps with the work I am doing. I have a PR submitted, which not only refactors the code but also introduces several performance improvements and abstractions to easily support new models for chunk stream.

This is my PR #1132

Would love to hear your thoughts and collaborate to reduce the duplicate work.