💬 Share your feedback here!

[alozowski]

This issue serves as a centralised place for general feedback, suggestions, and comments about Yourbench

Use this issue if you:

• have general thoughts or suggestions about the project;
• want to share feedback that doesn't need a separate issue;
• have questions or ideas you'd like to discuss;
• encountered something that's not quite a bug but worth mentioning.

For specific issues, please create a dedicated issue instead:

• 🐛 Bug reports
• ✨ Feature requests
• 📚 Documentation improvements

Feel free to comment below with any feedback, ideas, or general thoughts about the project. All constructive input is welcome and appreciated!

[patrickfleith]

Hi, I really like YourBench, which is why I started contributing. I appreciate that it provides an all-in-one pipeline: you start with raw documents and it outputs a benchmark dataset. I think it has strong potential for teams initiating a project who need an evaluation/test dataset quickly from their own data. I may be able to try it out on a real-world use case very soon, and I also hope to have more time to contribute.

Can you tell if it supports LLMs served by Ollama?

[alozowski]

Hi @patrickfleith!

Thank you for your feedback!

Yes, you should be able to work with Ollama. I decided to verify myself and slightly changed client and response in yourbench/utils/inference/inference_core.py to this to be on the safe side:

client = AsyncInferenceClient(
    base_url=model.base_url,
    timeout=GLOBAL_TIMEOUT,
    headers={"X-Request-ID": request_id},
)

response = await client.chat_completion(
    model=model.model_name,
    messages=inference_call.messages,
    temperature=inference_call.temperature or 0.7, # just in case
)

And in the yaml config I set up model_list like this:

model_list:
  - model_name: llama3.2
    base_url: http://127.0.0.1:11434/v1
    max_concurrent_requests: 5

It works! If you have any questions when running the project yourself, don't hesitate to open an issue

[patrickfleith]

Thanks indeed it works with ollama. There maybe a few tweaks necessary (like updating the default size of the context window. If I ever come back to creating a dataset with ollama I'll make sure to document the process as much as possible).

Hi @patrickfleith!

Thank you for your feedback!

Yes, you should be able to work with Ollama. I decided to verify myself and slightly changed client and response in yourbench/utils/inference/inference_core.py to this to be on the safe side:

client = AsyncInferenceClient(
    base_url=model.base_url,
    timeout=GLOBAL_TIMEOUT,
    headers={"X-Request-ID": request_id},
)

response = await client.chat_completion(
    model=model.model_name,
    messages=inference_call.messages,
    temperature=inference_call.temperature or 0.7, # just in case
)

And in the yaml config I set up model_list like this:

model_list:
  - model_name: llama3.2
    base_url: http://127.0.0.1:11434/v1
    max_concurrent_requests: 5

It works! If you have any questions when running the project yourself, don't hesitate to open an issue