MIPROv2 crashes with AssertionError: No input variables found in the example #1506
Description
Description
Hey guys,
I created an extractor using CoT to extract entities and relationships out of text, and tried to run it with MIPROv2 to generate optimized prompt instructions. So far, I am not able to get far. I am also using the recommended dspy.settings.configure(experimental=True) because the program keeps asking me about format_handler so I had to switch to using the experimental feature, hence getting stuck with this new problem. Note, this error happens for v2.4.14 and v2.4.16 so far.
The following logs show that I am able to process the 4 training examples without problems, before hitting a problem after bootstrapping with AssertionError: No input variables found in the example:
Code:
Full code in jupyter notebook and logs: https://github.com/gusye1234/nano-graphrag/blob/be09b227e018ee56faa5b3d0e4f74d0ba7d04f63/examples/finetune_entity_relationship_dspy.ipynb
Logs
WARNING: Projected Language Model (LM) Calls
Please be advised that based on the parameters you have set, the maximum number of LM calls is projected as follows:
- Prompt Model: 10 data summarizer calls + 4 * 2 lm calls in program + (3) lm calls in program aware proposer = 21 prompt model calls
- Task Model: 25 examples in minibatch * 10 batches + 4 examples in train set * 1 full evals = 254 task model calls
Estimated Cost Calculation:
Total Cost = (Number of calls to task model * (Avg Input Token Length per Call * Task Model Price per Input Token + Avg Output Token Length per Call * Task Model Price per Output Token)
+ (Number of calls to prompt model * (Avg Input Token Length per Call * Task Prompt Price per Input Token + Avg Output Token Length per Call * Prompt Model Price per Output Token).
For a preliminary estimate of potential costs, we recommend you perform your own calculations based on the task
and prompt models you intend to use. If the projected costs exceed your budget or expectations, you may consider:
- Reducing the number of trials (`num_batches`), the size of the trainset, or the number of LM calls in your program.
- Using a cheaper task model to optimize the prompt.
Error getting source code: unhashable type: 'list'.
Running without program aware proposer.
INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
summary: Prediction(
summary='The dataset comprises news articles covering diverse topics such as political events, economic data, and corporate news, with each example detailing entities and their relationships. This structured content, presented in formal journalistic syntax, is highly relevant to current events and suitable for training models in entity recognition, relationship extraction, and summarization tasks.'
)
DATA SUMMARY: The dataset comprises news articles covering diverse topics such as political events, economic data, and corporate news, with each example detailing entities and their relationships. This structured content, presented in formal journalistic syntax, is highly relevant to current events and suitable for training models in entity recognition, relationship extraction, and summarization tasks.
0%| | 0/4 [00:00<?, ?it/s]INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
WARNING:nano-graphrag:Received an empty JSON string
DEBUG:nano-graphrag:Entities: 29 | Missed Entities: 0 | Total Entities: 29
DEBUG:nano-graphrag:Relationships: 19 | Missed Relationships: 7 | Total Relationships: 26
DEBUG:nano-graphrag:Direct Relationships: 26 | Second-order: 0 | Third-order: 0 | Total Relationships: 26
25%|██▌ | 1/4 [03:32<10:37, 212.60s/it]INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
WARNING:nano-graphrag:Received an empty JSON string
DEBUG:nano-graphrag:Entities: 6 | Missed Entities: 0 | Total Entities: 6
DEBUG:nano-graphrag:Relationships: 5 | Missed Relationships: 0 | Total Relationships: 5
DEBUG:nano-graphrag:Direct Relationships: 5 | Second-order: 0 | Third-order: 0 | Total Relationships: 5
50%|█████ | 2/4 [04:26<03:58, 119.06s/it]INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
WARNING:nano-graphrag:Received an empty JSON string
DEBUG:nano-graphrag:Entities: 20 | Missed Entities: 0 | Total Entities: 20
DEBUG:nano-graphrag:Relationships: 22 | Missed Relationships: 6 | Total Relationships: 28
DEBUG:nano-graphrag:Direct Relationships: 28 | Second-order: 0 | Third-order: 0 | Total Relationships: 28
75%|███████▌ | 3/4 [07:26<02:27, 147.10s/it]INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
WARNING:nano-graphrag:Received an empty JSON string
DEBUG:nano-graphrag:Entities: 6 | Missed Entities: 0 | Total Entities: 6
DEBUG:nano-graphrag:Relationships: 6 | Missed Relationships: 2 | Total Relationships: 8
DEBUG:nano-graphrag:Direct Relationships: 7 | Second-order: 1 | Third-order: 0 | Total Relationships: 8
100%|██████████| 4/4 [08:32<00:00, 128.14s/it]
Bootstrapped 4 full traces after 4 examples in round 0.
0%| | 0/4 [00:00<?, ?it/s]INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
WARNING:nano-graphrag:Received an empty JSON string
DEBUG:nano-graphrag:Entities: 17 | Missed Entities: 0 | Total Entities: 17
DEBUG:nano-graphrag:Relationships: 12 | Missed Relationships: 8 | Total Relationships: 20
DEBUG:nano-graphrag:Direct Relationships: 17 | Second-order: 3 | Third-order: 0 | Total Relationships: 20
25%|██▌ | 1/4 [02:32<07:36, 152.01s/it]INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
WARNING:nano-graphrag:Received an empty JSON string
DEBUG:nano-graphrag:Entities: 29 | Missed Entities: 0 | Total Entities: 29
DEBUG:nano-graphrag:Relationships: 18 | Missed Relationships: 10 | Total Relationships: 28
DEBUG:nano-graphrag:Direct Relationships: 28 | Second-order: 0 | Third-order: 0 | Total Relationships: 28
50%|█████ | 2/4 [06:17<06:30, 195.10s/it]WARNING:nano-graphrag:Received an empty JSON string
DEBUG:nano-graphrag:Entities: 6 | Missed Entities: 0 | Total Entities: 6
DEBUG:nano-graphrag:Relationships: 5 | Missed Relationships: 0 | Total Relationships: 5
DEBUG:nano-graphrag:Direct Relationships: 5 | Second-order: 0 | Third-order: 0 | Total Relationships: 5
INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://api.deepseek.com/chat/completions "HTTP/1.1 200 OK"
WARNING:nano-graphrag:Received an empty JSON string
DEBUG:nano-graphrag:Entities: 7 | Missed Entities: 0 | Total Entities: 7
DEBUG:nano-graphrag:Relationships: 7 | Missed Relationships: 1 | Total Relationships: 8
DEBUG:nano-graphrag:Direct Relationships: 5 | Second-order: 3 | Third-order: 0 | Total Relationships: 8
100%|██████████| 4/4 [07:18<00:00, 109.53s/it]
Bootstrapped 4 full traces after 4 examples in round 0.
Using a randomly generated configuration for our grounded proposer.
Selected tip: description
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[13], line 10
1 optimizer = MIPROv2(
2 prompt_model=lm,
3 task_model=lm,
(...)
7 verbose=True
8 )
9 kwargs = dict(num_threads=os.cpu_count(), display_progress=True, display_table=0)
---> 10 miprov2_model = optimizer.compile(
11 model,
12 trainset=trainset[:4],
13 valset=valset,
14 requires_permission_to_run=False,
15 num_batches=10,
16 max_labeled_demos=5,
17 max_bootstrapped_demos=4,
18 eval_kwargs=kwargs
19 )
20 miprov2_model
File /opt/homebrew/Caskroom/miniconda/base/envs/nano-graphrag/lib/python3.10/site-packages/dspy/teleprompt/mipro_optimizer_v2.py:291, in MIPROv2.compile(self, student, trainset, valset, num_batches, max_bootstrapped_demos, max_labeled_demos, eval_kwargs, seed, minibatch, program_aware_proposer, requires_permission_to_run)
289 proposer.use_instruct_history = False
290 proposer.set_history_randomly = False
--> 291 instruction_candidates = proposer.propose_instructions_for_program(
292 trainset=trainset,
293 program=program,
294 demo_candidates=demo_candidates,
295 N=self.n,
296 prompt_model=self.prompt_model,
297 T=self.init_temperature,
298 trial_logs={},
299 )
300 for i, pred in enumerate(program.predictors()):
301 instruction_candidates[i][0] = get_signature(pred).instructions
File /opt/homebrew/Caskroom/miniconda/base/envs/nano-graphrag/lib/python3.10/site-packages/dspy/propose/grounded_proposer.py:302, in GroundedProposer.propose_instructions_for_program(self, trainset, program, demo_candidates, prompt_model, trial_logs, N, T, tip)
299 if pred_i not in proposed_instructions:
300 proposed_instructions[pred_i] = []
301 proposed_instructions[pred_i].append(
--> 302 self.propose_instruction_for_predictor(
303 program=program,
304 predictor=predictor,
305 pred_i=pred_i,
306 prompt_model=prompt_model,
307 T=T,
308 demo_candidates=demo_candidates,
309 demo_set_i=demo_set_i,
310 trial_logs=trial_logs,
311 tip=selected_tip,
312 ),
313 )
314 return proposed_instructions
File /opt/homebrew/Caskroom/miniconda/base/envs/nano-graphrag/lib/python3.10/site-packages/dspy/propose/grounded_proposer.py:349, in GroundedProposer.propose_instruction_for_predictor(self, program, predictor, pred_i, prompt_model, T, demo_candidates, demo_set_i, trial_logs, tip)
347 with dspy.settings.context(lm=prompt_model):
348 prompt_model.kwargs["temperature"] = T
--> 349 proposed_instruction = instruction_generator.forward(
350 demo_candidates=demo_candidates,
351 pred_i=pred_i,
352 demo_set_i=demo_set_i,
353 program=program,
354 data_summary=self.data_summary,
355 previous_instructions=instruction_history,
356 tip=tip,
357 ).proposed_instruction
358 prompt_model.kwargs["temperature"] = original_temp
360 # Log the trace used to generate the new instruction, along with the new instruction itself
File /opt/homebrew/Caskroom/miniconda/base/envs/nano-graphrag/lib/python3.10/site-packages/dspy/propose/grounded_proposer.py:203, in GenerateModuleInstruction.forward(self, demo_candidates, pred_i, demo_set_i, program, previous_instructions, data_summary, max_demos, tip)
200 modules = [match[0].strip() for match in matches]
201 module_code = modules[pred_i]
--> 203 module_description = self.describe_module(
204 program_code=self.program_code_string,
205 program_description=program_description,
206 program_example=task_demos,
207 module=module_code,
208 max_depth=10,
209 ).module_description
211 # Generate an instruction for our chosen module
212 print(f"task_demos {task_demos}")
File /opt/homebrew/Caskroom/miniconda/base/envs/nano-graphrag/lib/python3.10/site-packages/dspy/predict/predict.py:91, in Predict.__call__(self, **kwargs)
90 def __call__(self, **kwargs):
---> 91 return self.forward(**kwargs)
File /opt/homebrew/Caskroom/miniconda/base/envs/nano-graphrag/lib/python3.10/site-packages/dspy/predict/predict.py:126, in Predict.forward(self, **kwargs)
124 completions = v2_5_generate(lm, config, signature, demos, kwargs)
125 elif dsp.settings.experimental:
--> 126 completions = new_generate(lm, signature, dsp.Example(demos=demos, **kwargs), **config)
127 else:
128 completions = old_generate(demos, signature, kwargs, config, self.lm, self.stage)
File /opt/homebrew/Caskroom/miniconda/base/envs/nano-graphrag/lib/python3.10/site-packages/dspy/predict/predict.py:179, in new_generate(lm, signature, example, max_depth, **kwargs)
177 # Generate and extract the fields.
178 template = signature_to_template(signature, adapter=dsp.ExperimentalAdapter)
--> 179 prompt = template(example)
180 completions = lm(prompt, **kwargs)
181 completions = [template.extract(example, p) for p in completions]
File /opt/homebrew/Caskroom/miniconda/base/envs/nano-graphrag/lib/python3.10/site-packages/dsp/adapters/experimental_adapter.py:191, in ExperimentalAdapter.__call__(self, example, show_guidelines)
187 rdemos = rdemos_
189 example["augmented"] = True
--> 191 query = self.query(example)
192 parts = [self.instructions, *rdemos, self.guidelines(show_guidelines), *ademos, query,]
194 prompt = "\n\n---\n\n".join([p.strip() for p in parts if p])
File /opt/homebrew/Caskroom/miniconda/base/envs/nano-graphrag/lib/python3.10/site-packages/dsp/adapters/experimental_adapter.py:25, in ExperimentalAdapter.query(self, example, is_demo)
17 has_value = [
18 field.input_variable in example
19 and example[field.input_variable] is not None
20 and example[field.input_variable] != ""
21 for field in self.fields
22 ]
24 if not any(has_value):
---> 25 assert False, "No input variables found in the example"
27 for i in range(1, len(has_value)):
28 if has_value[i - 1] and not any(has_value[i:]):
AssertionError: No input variables found in the example