Support for connecting to models running via Llamafile

[phildougherty]

Hey Steven!

Very cool project! I have been having some fun trying to get a multi-modal agent setup going using hosted models alongside open source models running locally on my 3090 setup. I have been surprised by how much is already possible!

I am not sure if you have heard of the Mozilla foundation's Llamafile project but it builds on llama-cpp and it bundles weights as executable programs and allows for easier sharing/running of models locally. I was curious to get your thoughts on integrating the use of these? It seems like each Llamafile has a OAI API completions endpoint it runs by default. I thought perhaps this might be useful to avoid having to convert models to support the API as you describe in the README. Would love to pick your brain on so much!

[stevenlu137]

Hi, thank you very much for your suggestion. A project like llamafile is significant in reducing installation difficulties!

Since Llamafile provides an API compatible with OpenAI, I tried modifying the config.json to include the relevant configurations and found that AIlice can run well on Llamafile in non-multimodal mode. However, in multimodal mode, it cannot work properly due to some existing issues with Llamafile's multimodal interface, but I believe that solving these issues is just a matter of time.

In the long run, integrating AIlice directly with Llamafile to automatically download and run models can further simplify installation. I will consider making Llamafile the default choice!