(EAI-993): deprecate framework

README.md

@@ -1,47 +1,32 @@
-# MongoDB Chatbot Framework
+# MongoDB Knowledge Service
 
-The MongoDB Chatbot Framework is a set of libraries that you can use to build
-full-stack intelligent chatbot applications using MongoDB and [Atlas Vector Search](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/).
-The MongoDB Chatbot Framework includes first class support for
-retrieval-augmented generation (RAG).
+This repo contains the work of the MongoDB Education AI team. 
 
-The framework can take your chatbot application from prototype to production.
+## MongoDB Knowledge Service
 
-You can quickly get an AI chatbot enhanced with your data up and running using
-the framework's built-in data ingest process, chatbot server, and web UI. As you
-refine your application and scale to more users, you can modify the chatbot's
-behavior to meet your needs.
+The MongoDB Knowledge Service lets you learn about MongoDB using generative AI. To learn more about it, refer to the [MongoDB Knolwedge Service documentation](https://mongodb.github.io/chatbot)
 
-The framework is flexible and customizable. It supports multiple AI models and
-complex prompting strategies. It also includes tools for programmatic evaluation of your chatbot's AI components.
+## MongoDB Chatbot Framework (deprecated)
 
-## Documentation
+The team building the MongoDB Knowledge Service previous developed the MongoDB Chatbot Framework. This consisted of the npm packages:
 
-To learn how to use the MongoDB Chatbot Framework, refer to the documentation:
-<https://mongodb.github.io/chatbot/>.
+- `mongodb-chatbot-server`
+- `mongodb-chatbot-ui` (still used, refer to [UI](https://mongodb.github.io/chatbot/ui))
+- `mongodb-rag-core`
+- `mongodb-rag-ingest`
 
-You can also check out the following articles and videos about the framework:
+The MongoDB Chatbot Framework in now deprecated. We will no longer be maintaining it.
 
-- [[Video] MongoDB Chatbot Framework Learning Byte](https://learn.mongodb.com/courses/mongodb-chatbot-framework)
-- [[Article] Build a Production-Ready, Intelligent Chatbot With the MongoDB Chatbot Framework](https://dev.to/mongodb/build-a-production-ready-intelligent-chatbot-with-the-mongodb-chatbot-framework-4dd)
-- [[Article] Taking RAG to Production with the MongoDB Documentation AI Chatbot](https://www.mongodb.com/developer/products/atlas/taking-rag-to-production-documentation-ai-chatbot/)
+To learn more about the framework, refer to the the blog post [Build a Production-Ready, Intelligent Chatbot With the MongoDB Chatbot Framework](https://dev.to/mongodb/build-a-production-ready-intelligent-chatbot-with-the-mongodb-chatbot-framework-4dd).
 
-## MongoDB Docs AI Chatbot Implementation
+### Why Deprecate the MongoDB Chatbot Framework?
 
-This repo also contains the implementation of the MongoDB Docs Chatbot,
-which uses the MongoDB Chatbot Framework.
+Since we first launched the framework a year and a half ago, there's been a lot of progress in the TypeScript ecosystem for AI frameworks. We have decided that these frameworks remove the need for the MongoDB Chatbot Framework. Additionally, supporting the framework in addition to the Knowledge Service has been a maintenance burden on our small team.
 
-The MongoDB Docs Chatbot uses the MongoDB [documentation](https://www.mongodb.com/docs/) and [Developer Center](https://www.mongodb.com/developer/) as its sources of truth.
+In particular, we've been very impressed by the [Vercel AI SDK](https://ai-sdk.dev/docs/introduction). It has a great developer experience, is well maintained, and a robust feature set. We've moved most of our LLM call logic to the AI SDK. You can refer to our [mongodb/chatbot repository](https://github.com/mongodb/chatbot) to see how we're using it. For a tutorial on building with MongoDB Atlas and the AI SDK, refer to the blog post [Building a Chat Application That Doesn't Forget!](https://dev.to/mongodb/building-a-chat-application-with-mongodb-memory-provider-for-vercel-ai-sdk-56ap) by MongoDB's own Jesse Hall.
 
-The chatbot builds on the following technologies:
+For building more agentic applications in TypeScript, [Mastra](https://mastra.ai/en/docs) (itself built on the AI SDK), [LangGraph.js](https://langchain-ai.github.io/langgraphjs/), and the [OpenAI Agents SDK](https://openai.github.io/openai-agents-js/) all seem to be solid options. 
 
-- Atlas Vector Search: Indexes and queries content for use in project.
-- MongoDB Atlas: Persists conversations and content.
-- ChatGPT API: LLM to pre-process user queries and summarize responses to user queries.
-- OpenAI Embeddings API: Create vector embeddings for user queries and content. Used by Atlas Vector Search.
-
-To learn more about how we built the chatbot, check out the MongoDB Developer Center blog post
-[Taking RAG to Production with the MongoDB Documentation AI Chatbot](https://www.mongodb.com/developer/products/atlas/taking-rag-to-production-documentation-ai-chatbot/).
 
 ## Contributing
 

docs/docs/contact.md

@@ -0,0 +1,5 @@
+# Contact
+
+
+- MongoDB employees can reach out to the Education AI team on Slack at `#ask-education-ai`
+- External users can create an issue on the [mongodb/chatbot GitHub repo](https://github.com/mongodb/chatbot/issues/new)

docs/docs/data-sources.md

@@ -0,0 +1,26 @@
+# Data Sources
+
+The MongoDB Knowledge Service uses [retrieval augmented generation](https://www.mongodb.com/resources/basics/artificial-intelligence/retrieval-augmented-generation) to answer user queries.
+
+All the data sources are public on the web.
+
+
+## Sources
+
+The Knowledge Service ingests the following data sources:
+
+- MongoDB Technical Documentation (https://mongodb.com/docs)
+- MongoDB Developer Center blog (https://mongodb.com/developer)
+- MongoDB University transcripts and landing pages (https://learn.mongodb.com)
+- Select marketing and sales pages from https://mongodb.com
+- Select external data sources:
+  - Mongoose.js docs (https://mongoosejs.com)
+  - Prisma MongoDB connector docs (https://www.prisma.io/docs/orm/overview/databases/mongodb)
+  - Terraform MongoDB Provider docs (https://registry.terraform.io/providers/mongodb/mongodbatlas/latest)
+ - WiredTiger docs (https://source.wiredtiger.com/)
+ - Practical MongoDB Aggregations book (https://www.practical-mongodb-aggregations.com/)
+
+
+## Source Code
+
+You can see the source code for the data source ingestion here: https://github.com/mongodb/chatbot/tree/main/packages/ingest-mongodb-public

docs/docs/datasets.md

@@ -0,0 +1,19 @@
+# Datasets
+
+The Education AI team maintains various datasets for use with AI systems. All datasets can be found in the [MongoDB Education AI HuggingFace](https://huggingface.co/mongodb-eai).
+
+## Content
+
+Content datasets can be useful for building RAG systems and training models.
+
+| Name | Type | Description  | Visibility | Use Cases | Links |
+| :---- | :---- | :---- | :---- | :---- | :---- |
+| Public documentation | Long-form content  | Markdown version of docs and developer center content. | Public | RAG, model training | https://huggingface.co/datasets/mongodb-eai/docs  |
+| Code example dataset | Prompt-completion | Code examples extracted from the MongoDB docs and developer center with prompts that could be used to generate the code.  | Public  | Model fine-tuning | https://huggingface.co/datasets/mongodb-eai/code-example-prompts  |
+
+## Benchmarks
+
+| Name | Type | Description  | Visibility |  |
+| :---- | :---- | :---- | :---- | :---- |
+| Natural language-to-Node.js Mongosh | Code generation | Assess how well LLMs generate `mongosh` code given a natural language prompt and information about a database. | External | https://huggingface.co/datasets/mongodb-eai/natural-language-to-mongosh |
+

docs/docs/framework.md

@@ -0,0 +1,20 @@
+# MongoDB Chatbot Framework (deprecated)
+
+The team building the MongoDB Knowledge Service previous developed the MongoDB Chatbot Framework. This consisted of the npm packages:
+
+- `mongodb-chatbot-server`
+- `mongodb-chatbot-ui` (still used, refer to [UI](./ui.md))
+- `mongodb-rag-core`
+- `mongodb-rag-ingest`
+
+The MongoDB Chatbot Framework in now deprecated. We will no longer be maintaining it.
+
+To learn more about the framework, refer to the the blog post [Build a Production-Ready, Intelligent Chatbot With the MongoDB Chatbot Framework](https://dev.to/mongodb/build-a-production-ready-intelligent-chatbot-with-the-mongodb-chatbot-framework-4dd).
+
+## Why Deprecate the MongoDB Chatbot Framework?
+
+Since we first launched the framework a year and a half ago, there's been a lot of progress in the TypeScript ecosystem for AI frameworks. We have decided that these frameworks remove the need for the MongoDB Chatbot Framework. Additionally, supporting the framework in addition to the Knowledge Service has been a maintenance burden on our small team.
+
+In particular, we've been very impressed by the [Vercel AI SDK](https://ai-sdk.dev/docs/introduction). It has a great developer experience, is well maintained, and a robust feature set. We've moved most of our LLM call logic to the AI SDK. You can refer to our [mongodb/chatbot repository](https://github.com/mongodb/chatbot) to see how we're using it. For a tutorial on building with MongoDB Atlas and the AI SDK, refer to the blog post [Building a Chat Application That Doesn't Forget!](https://dev.to/mongodb/building-a-chat-application-with-mongodb-memory-provider-for-vercel-ai-sdk-56ap) by MongoDB's own Jesse Hall.
+
+For building more agentic applications in TypeScript, [Mastra](https://mastra.ai/en/docs) (itself built on the AI SDK), [LangGraph.js](https://langchain-ai.github.io/langgraphjs/), and the [OpenAI Agents SDK](https://openai.github.io/openai-agents-js/) all seem to be solid options. 

docs/docs/index.md

@@ -1,77 +1,16 @@
 ---
 title: Home
-description: Build full-stack intelligent chatbot applications using MongoDB and Atlas Vector Search.
+description: MongoDB Knowledge Service
 ---
 
-# MongoDB Chatbot Framework
+# MongoDB Knowledge Service
 
-:::warning[👷‍♂️ Work In Progress 👷‍♂️]
+The MongoDB Knowledge Service lets you learn about MongoDB using generative AI.
 
-The MongoDB Chatbot Framework is under active development
-and may undergo breaking changes.
+## Server
 
-We aim to keep the documentation up to date with the latest version.
+To call the MongoDB Knowledge Service API, refer to the [OpenAPI specification](/server/openapi/).
 
-:::
+## UI Library
 
-Build full-stack intelligent chatbot applications using MongoDB
-and [Atlas Vector Search](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/).
-
-The MongoDB Chatbot Framework is a set of libraries that you can use to build a
-production-ready chatbot application. The framework provides first-class support
-for retrieval augmented generation (RAG), and is extensible to support other
-patterns for building intelligent chatbots.
-
-The framework can take your chatbot application from prototype to production.
-
-You can quickly get an AI chatbot enhanced with your data up and running using
-the framework's built-in data ingest process, chatbot server, and web UI. As you
-refine your application and scale to more users, you can modify the chatbot's
-behavior to meet your needs.
-
-The framework is flexible and customizable. It supports multiple AI models and
-complex prompting strategies.
-
-## How It Works
-
-The MongoDB Chatbot Framework has the following core components:
-
-- [MongoDB Atlas](./mongodb.md): Database for the application that stores content and conversation.
-  Indexes content using Atlas Vector Search.
-- [Ingest CLI](./ingest/configure.md): Configurable CLI application that you can use to ingest content into a MongoDB collection for use with Atlas Vector Search.
-- [Chatbot Server](./server/configure.md): Express.js server routes that you can use to build a chatbot application.
-- [Chatbot UI](./ui.md): React.js UI components that you can use to build a chatbot application.
-
-## Quick Start
-
-To get started using the MongoDB Chatbot Framework, refer to the [Quick Start](./quick-start.md) guide.
-
-## Design Principles
-
-The MongoDB Chatbot Framework is designed around the following principles:
-
-- Composability: You can use components of the chatbot framework independently of each other.
-  For example, we have some users who are using only our ingestion CLI to ingest content into MongoDB Atlas, but use other tools to build their chatbot and UI.
-- Pluggability: You can plug in your own implementations of components.
-  For example, you can plug in your own implementations of the `DataSource` interface
-  to ingest content from different data sources.
-- Inversion of Control: The framework makes decisions about boilerplate aspects
-  of intelligent chatbot systems so that you can focus on building logic unique to your application.
-
-## MongoDB Docs Chatbot
-
-This framework is used to build the MongoDB Docs Chatbot, a RAG chatbot that answers questions about the MongoDB documentation. You can try it out on [mongodb.com/docs](https://www.mongodb.com/docs/).
-
-Here's a reference architecture for how the MongoDB Chatbot Framework system works for the MongoDB Docs Chatbot.
-
-Data ingestion:
-
-![Data Ingestion Architecture](/img/ingest-diagram.webp)
-
-Chat Server:
-
-![Chat Server Architecture](/img/server-diagram.webp)
-
-### How We Built It
-
-- To learn more about how we built the chatbot, check out the MongoDB Developer Center blog post [Taking RAG to Production with the MongoDB Documentation AI Chatbot](https://www.mongodb.com/developer/products/atlas/taking-rag-to-production-documentation-ai-chatbot/).
+You can call the MongoDB Knowledge Service API from a web application using the `mongodb-chatbot-ui` npm package. The UI library provides a React component that you can use to build a chat interface. Learn more in the [UI documentation](./ui.md).