Add tool completion to batch inference

[yunfeng-scale]

Pull Request Summary

• Partial open source of tool_completion
• Adapt batch inference to run tool completion loop (generations -> tool run)
• Add tool config to batch inference API

for notekeeping, some things to fix:

1. vLLM stop sequence does not cut token logprobs so there's extra stop sequence token logprobs
2. tool should not use a hardcoded tokenizer, should use whatever provided together with model weights
3. token log prob not captured for tool output

Test Plan and Usage Guide

some local test
tested running jobs in training cluster
added unit tests
vllm batch image deployed

[Georgepu1]

overall lgtm and it's nice that tool_completion currently mirrors the latest version, just a couple of nits. thanks for adding this feature in for batch complete calls, we'll add it into our inference pipeline

[saiatmakuri]

nit: move each tool to own file for future if there are more tools

[saiatmakuri]

is this safe for the batch job? seems that potentially harmful generations could break this

[saiatmakuri]

(synced offline) known risk - for now since main use is internal, risk should be low and this is okay. will have follow-ups to mitigate issues

[yunfeng-scale]

that's true. please take a look how we think about this problem: https://docs.google.com/document/d/1zKfhizd0FpwoupF_jYmWkz4naTxkInI0g-rS6xzNWso/edit#heading=h.2hej6pszhyrd

[yunfeng-scale]

i think for batch jobs the risk is more manageable