append_to_context=False on TextFrame not propagated through TTS service

[jamsea]

Summary

When a TextFrame (or LLMTextFrame) with append_to_context=False passes through the TTS service, the flag is silently dropped. The resulting TTSTextFrame always has append_to_context=True, so the text still gets added to the LLM context by the assistant aggregator.

The TTSSpeakFrame path works correctly since it explicitly passes frame.append_to_context through to _push_tts_frames().

Steps to reproduce

1. Create a custom FrameProcessor positioned between the LLM and TTS
2. Emit a TextFrame or LLMTextFrame with append_to_context = False
3. Observe the text still appears in the LLM assistant context

Root cause

In tts_service.py, _process_text_frame() (line 803) creates new AggregatedTextFrame objects from the incoming text but does not forward frame.append_to_context to _push_tts_frames():

async def _process_text_frame(self, frame: TextFrame):
    async for aggregate in self._text_aggregator.aggregate(frame.text):
        # ...
        await self._push_tts_frames(
            AggregatedTextFrame(aggregate.text, aggregate.type),
            includes_inter_frame_spaces,
            # append_tts_text_to_context is not passed, defaults to True
        )

By contrast, the TTSSpeakFrame path correctly propagates the flag:

await self._push_tts_frames(
    AggregatedTextFrame(frame.text, AggregationType.SENTENCE),
    append_tts_text_to_context=frame.append_to_context,  # properly forwarded
    push_assistant_aggregation=push_assistant_aggregation,
)

Suggested fix

Pass frame.append_to_context through in _process_text_frame():

async def _process_text_frame(self, frame: TextFrame):
    async for aggregate in self._text_aggregator.aggregate(frame.text):
        includes_inter_frame_spaces = (
            frame.includes_inter_frame_spaces
            if aggregate.type == AggregationType.TOKEN
            else False
        )
        if aggregate.type != AggregationType.TOKEN:
            await self.stop_text_aggregation_metrics()
        await self._push_tts_frames(
            AggregatedTextFrame(aggregate.text, aggregate.type),
            includes_inter_frame_spaces,
            append_tts_text_to_context=frame.append_to_context,
        )

Versions

• pipecat-ai 0.0.104

[markbackman]

@jamsea my memory is a bit hazy on this, but I think this may be working how we expect. There is no idea of transference of the append_to_context property between a TextFrame type as it travels through the TTSService.

The point of the append_to_context property is to allow processors to create TextFrames (e.g. AggregatedTextFrame) for the purpose of having this text appear in a client UI ASAP, as there's an RTVIObserver that monitors the existence of this frame. The idea being that both AggregatedTextFrame and TTSTextFrame are both TextFrames, meaning that both could potentially end up in the context. That would be bad, so we give the developer a way to make the AggregatedTextFrame omitted from the context.

There is no concept of omitting TTS generated TTSTextFrames from the context—essentially what the bot says is what generally goes into the context.

As a rule of thumb, what the LLM generates should always end up back in the context. If you break this rule, then you'll teach the LLM how to respond, as the context, itself, acts as a few shot example of what the LLM should do. So, it would be bad practice to omit text generated by the LLM from the context.

Note: the TTSSpeakFrame is different. It's user generated context that may or may not make sense to include in the context.

To omit text from the context, you would need a filter after the TTS service to block the TTSTextFrames from entering the context. This isn't a common or recommended use case as it can poison the context, so we don't provide a way to make it easy.

@mattieruth worked on this as well and may remember better than me, so I'll phone a friend.

[virendra-dhakhada]

Thanks @markbackman for the detailed explanation. I completely agree with the rule of thumb that what the LLM generates should always end up back in context. Our use case actually follows that principle, but we need more granular control over which frame carries the context.

Our setup: We use OpenAI structured output (response_format: json_schema) with a Pydantic model like:

{
  "coverage_note": "Covered: preferences. Next: ask about frequency.",
  "response": "Thanks for sharing! How often do you listen to music?"
}

coverage_note is internal model reasoning (not user-facing). response is what the participant hears via TTS.

What we want:

• Full JSON → added to LLM context (preserves the rule of thumb, the LLM sees its complete output)
• Only the response field → spoken via TTS

How we're doing it: We have a custom FrameProcessor between the LLM and TTS that:

1. Buffers the raw JSON tokens and uses from langchain_core.utils.json import parse_partial_json to extract the response field incrementally
2. Emits LLMTextFrame(delta, skip_tts=True) for the raw JSON tokens, these flow to the context aggregator but skip TTS ✓
3. Emits TTSTextFrame(response_delta) for just the spoken response, TTS speaks this ✓

The problem: The TTSTextFrame from step 3 also gets captured by the assistant context aggregator, so the response text appears twice in context, once as part of the full JSON (correct) and once from the TTSTextFrame (duplicate).

We tried setting append_to_context=False on the TTSTextFrame, but as this issue describes, the flag isn't propagated. We also tried TTSSpeakFrame, but it bypasses Cartesia's sentence aggregation and causes fragmented speech with Invalid transcript errors on punctuation-only chunks.

What would help: A mechanism to emit text that reaches TTS but is excluded from the context aggregator.