Skip to content

Commit 9ff222a

Browse files
committed
add context-size prop
1 parent 967f88f commit 9ff222a

File tree

2 files changed

+22
-6
lines changed

2 files changed

+22
-6
lines changed

content/manuals/ai/model-runner/_index.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -380,3 +380,7 @@ The Docker Model CLI currently lacks consistent support for specifying models by
380380
## Share feedback
381381
382382
Thanks for trying out Docker Model Runner. Give feedback or report any bugs you may find through the **Give feedback** link next to the **Enable Docker Model Runner** setting.
383+
384+
## Related pages
385+
386+
- [Use Model Runner with Compose](/manuals/compose/how-tos/model-runner.md)

content/manuals/compose/how-tos/model-runner.md

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -40,15 +40,27 @@ services:
4040
type: model
4141
options:
4242
model: ai/smollm2
43+
context-size: 1024
4344
```
4445
45-
Notice the dedicated `provider` attribute in the `ai_runner` service.
46-
This attribute specifies that the service is a model provider and lets you define options such as the name of the model to be used.
46+
Notice the following:
4747
48-
There is also a `depends_on` attribute in the `chat` service.
49-
This attribute specifies that the `chat` service depends on the `ai_runner` service.
50-
This means that the `ai_runner` service will be started before the `chat` service to allow injection of model information to the `chat` service.
48+
- In the `ai_runner` service:
5149

50+
- `provider.type`: Specifies that the service is a `model` provider.
51+
- `provider.options`: Specifies the options of the model. In our case, we want to use
52+
`ai/smollm2`, and we set the context size to 1024 tokens.
53+
54+
> [!NOTE]
55+
> Each model has its own maximum context size. When increasing the context length,
56+
> consider your hardware constraints. In general, try to use the smallest context size
57+
> possible for your use case.
58+
59+
- In the `chat` service:
60+
61+
- `depends_on` specifies that the `chat` service depends on the `ai_runner` service. The
62+
`ai_runner` service will be started before the `chat` service, to allow injection of model information to the `chat` service.
63+
5264
## How it works
5365

5466
During the `docker compose up` process, Docker Model Runner automatically pulls and runs the specified model.
@@ -61,6 +73,6 @@ In the example above, the `chat` service receives 2 environment variables prefix
6173

6274
This lets the `chat` service to interact with the model and use it for its own purposes.
6375

64-
## Reference
76+
## Related pages
6577

6678
- [Docker Model Runner documentation](/manuals/ai/model-runner.md)

0 commit comments

Comments
 (0)