Exclude thinking generation from python parsing

[sbucaille]

Hey,

Is your feature request related to a problem? Please describe.
Sometimes, my LLM steps ends with final_answer(...) not being included in the required ``py `` code block syntax, so smolagents raises an error and starts a new step telling the LLM that the syntax was wrong.
The problem comes from the fact that I was using a reasoning LLM (qwen 14b in my case), which first reflects on the problem by generating its reasoning between <think> </think> tags, at some point, it plans to use the required syntax by generating it in its thoughts, but without code in it.

Step 1 not including the python block syntax :

<think>                                                                                                                                                                                                           
Okay, the user wanted to add a component with the text "hello world" to their Gradio interface. I used the interface_agent tool to handle that. The agent was instructed to add a text component with the label   
"hello world". The response from the agent said the task was completed. Now I need to confirm that the component was added successfully.                                                                          
                                                                                                                                                                                                                  
I should check if there's any feedback or confirmation needed. Since the user didn't specify further actions, maybe they just want to know it's done. The final answer should be a simple confirmation. Let me    
make sure the response is clear and concise.                                                                                                                                                                      
</think>                                                                                                                                                                                                          
                                                                                                                                                                                                                  
The text component with the label "hello world" has been successfully added to the Gradio interface.                                                                                                              
                                                                                                                                                                                                                  
final_answer("The 'hello world' text component has been added to the Gradio interface.")

Step 2 thinking about adding it but stopped because the python code block syntax got detected by the parser :

<think>                                                                                                                                                                                                           
Okay, the user wants me to add a component with the text "hello world" to their Gradio interface. Let me think about how to do this correctly.                                                                    
                                                                                                                                                                                                                  
First, I remember that the interface_agent tool is used for modifying the Gradio interface. So I should use that. The task needs to be clearly described. The user mentioned adding a component with the text     
"hello world", so the label should be "hello world".                                                                                                                                                              
                                                                                                                                                                                                                  
Wait, in the previous attempt, I called the interface_agent with the correct instruction, and the agent responded that the task was completed. But the user is now showing an error about code parsing. Maybe the 
way I structured the code was incorrect. The user's system might be expecting a different format for the code snippet.                                                                                            
                                                                                                                                                                                                                  
Let me check the rules again. Oh right, the code must be in a specific format with ```py and end with                                                                                                             
Error in code parsing:
Your code snippet is invalid, because the regex pattern ```(?:py|python)?\s*\n(.*?)\n``` was not found in it.

Here is the full example for more clarity : https://app.warp.dev/block/embed/hUU6K5aEKJKoJZJMpKuPcR

Describe the solution you'd like

Prevent parsing code generation if a <think> tag appeared without a closing </think> tag
Adding a new rule in fix_final_answer_code may solve the problem.
I am willing to contribute if you want.

[albertvillanova]

Hi @sbucaille,

Step 2 thinking about adding it but stopped because the python code block syntax got detected by the parser

I think the error here is not that "the python code block syntax got detected by the parser", but that there is NO code snippet in the model response.

Please note that the code parsing expects a code snippet in the LLM output with this format:

```py
...
```<end_code>

However, your model output was:

<think>                                                                                                                                                                                                           
Okay, the user wants me to add a component with the text "hello world" to their Gradio interface. Let me think about how to do this correctly.                                                                    
                                                                                                                                                                                                                  
First, I remember that the interface_agent tool is used for modifying the Gradio interface. So I should use that. The task needs to be clearly described. The user mentioned adding a component with the text     
"hello world", so the label should be "hello world".                                                                                                                                                              
                                                                                                                                                                                                                  
Wait, in the previous attempt, I called the interface_agent with the correct instruction, and the agent responded that the task was completed. But the user is now showing an error about code parsing. Maybe the 
way I structured the code was incorrect. The user's system might be expecting a different format for the code snippet.                                                                                            
                                                                                                                                                                                                                  
Let me check the rules again. Oh right, the code must be in a specific format with ```py and end with ```<end_code>

You see there is no code snippet, so the parsing error is raised.

Please, feel free to reopen the issue if there is more context to be considered.

[sbucaille]

Hi @albertvillanova ,
Just to be sure I understand correctly the way smolagents is implemented, does the parser stop the generation as soon as it detects the python code snippet syntax ?

[albertvillanova]

Oh, my mistake, @sbucaille!

I understood you were referring to the parsing of the model output, but indeed the issue lies in the generation of the model output itself.

In fact, the <end_code> is passed as a stop_sequence to the model, which causes the generation to stop as soon as that token appears (which is the case in your model output: see comment above: #1404 (comment)):

smolagents/src/smolagents/agents.py

Lines 1495 to 1499 in 8c8bf2e

	chat_message: ChatMessage = self.model.generate(
	input_messages,
	stop_sequences=["<end_code>", "Observation:", "Calling tools:"],
	**additional_args,
	)

On the other hand, I don't see a straightforward solution to this issue... Any ideas?

[sbucaille]

I see that there are checks whether the generated message contains <end_code> or not, if not it is appended.
What if we perform a check there whether a <think> was opened and never closed with a </think>, if that's the case, then we continue the generation, because it means we are still in the thinking process and that isn't part of our model's actions.
I get it may not generalize to all models, but maybe adding a new attribute to the CodeAgent class to let it know that the LLM is a reasoning one and that, in its thoughts, it is possible it outputs such tokens but it should be discarded until it is not in its thinking generation anymore ?