server : remove system prompt support

[ggerganov]

The "system_prompt" related functionality is quite outdated and is introducing unnecessary complexity. It only sort of makes sense for non-finetuned models in order to save the computation of a common prefix when there are multiple parallel slots. But in practice, only finetuned models are utilized for this use case and they always require a chat template, which is incompatible with the current implementation of the system prompt. So in order to simplify the code a bit, we should remove the system prompt related functionality from the server.

[GuillaumeBruand]

I am using system_prompt along with parallel slots and the built-in chat template of each model. What will be the preferred way to keep the current behaviour, a conditionned chatbot for all my users ?

[ggerganov]

What does your context currently look? Is it:

[system prompt tokens]
<|im_start|>system
[another system prompt here?]<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant

[GuillaumeBruand]

Kind of, for llama3.1 it is looking like

<|start_header_id|>system<|end_header_id|>
{system prompt here}<|eot_id|><|start_header_id|>user<|end_header_id|>
Hello<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Hi there<|eot_id|><|start_header_id|>user<|end_header_id|>

My understanding is that {system prompt} is updated at launch or through the api endpoint, and to replicate the current behaviour I would need to update the template myself.

[ggerganov]

There are 2 types of system prompts:

• The one implemented in llama-server that I would like to remove. The way it works is it is prefixed to all other tokens. Regardless if there is a chat template or not, the system prompt tokens of this kind will be at the start of the context (see my message earlier)
• The second type of of system prompt is the one that can be passed from the client through the chat template by adding a message with role system for example. This continues to work as always - the client is responsible to send the appropriate system prompt

@GuillaumeBruand I'm afraid your context likely looks like this:

{system prompt here}
<|start_header_id|>system<|end_header_id|>
<|eot_id|><|start_header_id|>user<|end_header_id|>
Hello<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Hi there<|eot_id|><|start_header_id|>user<|end_header_id|>

Which is technically incorrect. The system prompt configured at launch or passed through the API is not applied inside the chat templates - this is stated in the llama-server README, but it's easy to get confused, so one more reason to remove this functionality.

[GuillaumeBruand]

@ggerganov Thanks for the clarification, I will update my workflow according your recommandations so that I will not rely anymore on this (deprecated?) feature.