docs: Added Deploying LLMs into production + a new ecosystem

[kouroshHakha]

No description provided.

[kouroshHakha]

One possible extension to this PR:

• Add an example of self-hosted chains (e.g. gpt2 from hf) to serve.

[skcoirz]

ah, this is a good idea. Conventionally, inference services are hosted by LLM providers. I was thinking of LangChain as a part of product logic, but after reading your PR, I realized we are a valuable inference host too especially after so many customization through multi-api combination, (prefix) prompt engineering and complex agent features.

Thank you for sharing with us your great ideas!

[hwchase17]

this is more of a ray integration than a comprehensive guide on deploying LLMs in production

this seems more suitable to go in the ecosystem section

[kouroshHakha]

Hey @hwchase17 , thanks for your input. I'm considering revising the section on "Deployment of LLMs in production." Instead of the current story about how Ray serve can assist with deployment, I'm thinking of creating a more general tutorial on the key concepts to consider when deploying LLMs (autoscaling, spot-instance serving, defining end-points, etc). Then I can link it to the ray integration page for follow-ups. What do you think of this approach? Do you have any other suggestions or ideas to improve this section?

[richardliaw]

Hey there - also from Ray team here -- your feedback makes sense @hwchase17 !

Probably better for this PR to actually start the "comprehensive guide for deploying LLMs in production". We could use both the Ray and the BentoML (https://github.com/ssheng/BentoChain) example as a starting point, so that it doesn't just look like a basic Ray integration.

And the sections of the guide would include the parts that @kouroshHakha mentioned.

Thoughts?

[kamil-kaczmarek]

I think it makes sense. we can keep the comprehensive guide for deploying LLMs in production focused more on concept understanding and provide solution starters. From there we can link to more detailed examples.

[kouroshHakha]

Hey @hwchase17, I updated the main section to be more general and more in the lines of a comprehensive guide for deploying LLMs in production. I am thinking of linking different serving ecosystems (ray serve, bentochain, etc) to this main doc and this should live somewhere in the main doc pages that is super visible (maybe under LLMs?) I can polish the text / figures a bit more if the outline sounds reasonable to you. Thanks.

[hwchase17]

ray serve notebook looks good!

Deploying LLMs notebook is also good in content, but i think in wrong place. modules is very specific to code in the library, this is more of (very good) general purpose documentation.

i would suggest we move this to the additional resources section (and maybe make it a markdown file, no reason for it to be ipynb)

[kamil-kaczmarek]

Hi @hwchase17 thanks for the suggestions.

@kouroshHakha implemented requested the changes. Please have a look.

[kouroshHakha]

@hwchase17 I think this PR is ready to be merged. Can someone from your team do a final pass? Thanks.

[hwchase17]

lgtm - thanks! sorry for dealy