Skip to content

Commit 5dfd3fc

Browse files
committed
Update tokenizer usage
1 parent 912fcd3 commit 5dfd3fc

File tree

1 file changed

+11
-4
lines changed

1 file changed

+11
-4
lines changed

ch05/11_qwen3/README.md

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,8 +45,14 @@ pip install llms_from_scratch tokenizers
4545
Specify which model to use:
4646

4747
```python
48-
USE_REASONING_MODEL = False # The base model
49-
USE_REASONING_MODEL = True # The "thinking" model
48+
USE_REASONING_MODEL = True
49+
# Uses the base model if USE_REASONING_MODEL = False
50+
51+
USE_INSTRUCT_MODEL = False
52+
# Uses the instruct mode (without reasoning) if
53+
# USE_REASONING_MODEL = True
54+
# USE_INSTRUCT_MODEL = False
55+
# This setting does have no effect if USE_REASONING_MODEL = False
5056

5157

5258
# Use
@@ -187,10 +193,11 @@ else:
187193
tok_filename = "tokenizer-base.json"
188194

189195
tokenizer = Qwen3Tokenizer(
190-
tokenizer_file_path=tok_filename,
196+
tokenizer_file_path=tokenizer_file_path,
191197
repo_id=repo_id,
198+
apply_chat_template=USE_REASONING_MODEL,
192199
add_generation_prompt=USE_REASONING_MODEL,
193-
add_thinking=USE_REASONING_MODEL
200+
add_thinking=not USE_INSTRUCT_MODEL
194201
)
195202
```
196203

0 commit comments

Comments
 (0)