Enable Token Caching with Vercel AI and Anthropic

[planger]

Vercel doesn't handle caching in a generic way, so we would need to set caching explicitly if we access an Anthropic model with Vercel.

Additional Info

To control caching with Vercel AI SDK when using Anthropic models, you can utilize the providerOptions to set cache control for specific message parts. Here's how you can do it:

Setting Cache Control

You can specify cache control for individual message parts by including the providerOptions with the cacheControl setting. For example:

import { anthropic } from '@ai-sdk/anthropic';
import { generateText } from 'ai';

const result = await generateText({
  model: anthropic('claude-3-5-sonnet-20240620'),
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'You are a JavaScript expert.',
        },
        {
          type: 'text',
          text: 'Explain the error message.',
          providerOptions: {
            anthropic: {
              cacheControl: { type: 'ephemeral' },
            },
          },
        },
      ],
    },
  ],
});

This setup marks the specific message part for ephemeral caching.

Important Considerations

• Minimum Token Threshold: Ensure that the content marked for caching meets the minimum token requirement. For instance, Claude 3.5 Haiku requires at least 2048 tokens for caching to be effective.

• Structured Content (parts): If you're using structured content with parts, be aware that there have been issues where providerOptions are not correctly applied. It's advisable to use plain content strings in such cases.

• System Messages: You can also apply cache control to system messages by including them at the beginning of your messages array and specifying the providerOptions.

If you need further assistance or examples, feel free to ask!

[planger]

I also observed the following error related to Vercel with Anthropic:

025-06-06T08:17:09.922Z root ERROR TypeError [ERR_INVALID_STATE]: Invalid state: ReadableStream is locked
    at InternalReadableStream.cancel (node:internal/webstreams/readablestream:326:9)
    at VercelAiStreamTransformer.transform (/.../theia/examples/browser/lib/backend/main.js:8137:37)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /.../theia/examples/browser/lib/backend/main.js:5160:34