Gave instructions to correct citation on answer #929
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
key should be used
There was a problem hiding this comment.
Pull Request Overview
This PR clarifies the required citation format by preventing the use of non-citation-key formats in answers.
- Updates the test suite to reject answers containing a citation with a name and year
- Enhances the prompt instructions to specify allowed citation formats and provide examples
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tests/test_agents.py | Adds a test assertion to ensure answers contain valid citation keys |
| paperqa/prompts.py | Updates prompt instructions to enforce strict citation key formats |
dosubot
bot
added
size:S
jamesbraza
reviewed
Apr 1, 2025
jamesbraza
reviewed
Apr 1, 2025
tests/test_agents.py
Outdated
| @@ -973,6 +973,30 @@ async def test_clinical_tool_usage(agent_test_settings) -> None: | |||
| ), "No clinical trials were put into contexts" | |||
|
|
|||
|
|
|||
| @pytest.mark.asyncio | |||
| async def test_citation_formatting(agent_test_settings): | |||
There was a problem hiding this comment.
What do you think of collapsing this test into a subtest of test_agent_types, and it would actually:
- Decrease CI runtime (since we're already running the agent in those tests)
- Increase the coverage of the assertions, since more agents types are used there
tests/test_agents.py
Outdated
Comment on lines
986
to
988
| assert not re.search( | ||
| name_year_regex, response.session.answer | ||
| ), "Answer contains citation with name and year instead of citation key" |
There was a problem hiding this comment.
This assertion fails on current main branch, nice work here!! 🚀
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The LLM sometimes broke the pattern by using citation styles other than the citation key, which caused problems on the platform downstream. This PR gives more instructions on how the citation should be included in the answer.
A few examples of wrong citations:
1 - Task: What is the molecule known to have the greatest solubility in water?
Answer: (...) Similarly, studies on aqueous solubility in drug discovery (e.g., Ishikawa and Hashimoto, 2011) discuss extremely low solubility values for particular organic compounds, but these systems are not directly comparable to the gas-phase solubility measurements in Hildebrand’s work. (...)
Issue: Ishikawa and Hashimoto, 2011 is not a citation key
2 - Task: What is the molecule known to have the greatest solubility in water?
Answer: (...) Therefore, based solely on the provided context, ammonia (NH₃) is the molecule known to have the greatest solubility in water (hildebrand1916solubility pages 14–17).
Issue: * Hildebrand’s 1916 paper (pages 14–17) * and hildebrand1916solubility should be this paper. It should have used "hildebrand1916solubility. pages 17-19". The difference is the period.
We need to use whatever the citation key is and avoid the LLM from changing it