Skip to content

Add Alibaba Cloud OpenSearch as a new vector store #6154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
rlancemartin merged 4 commits into from
Jun 20, 2023
Merged

Add Alibaba Cloud OpenSearch as a new vector store #6154

rlancemartin merged 4 commits into from
Jun 20, 2023

Conversation

@zhaoshengbo
Copy link
Contributor

@zhaoshengbo zhaoshengbo commented Jun 14, 2023 •
edited
Loading

Hello Folks,

Thanks for creating and maintaining this great project. I'm excited to submit this PR to add Alibaba Cloud OpenSearch as a new vector store.

OpenSearch is a one-stop platform to develop intelligent search services. OpenSearch was built based on the large-scale distributed search engine developed by Alibaba. OpenSearch serves more than 500 business cases in Alibaba Group and thousands of Alibaba Cloud customers. OpenSearch helps develop search services in different search scenarios, including e-commerce, O2O, multimedia, the content industry, communities and forums, and big data query in enterprises.

OpenSearch provides the vector search feature. In specific scenarios, especially test question search and image search scenarios, you can use the vector search feature together with the multimodal search feature to improve the accuracy of search results.

This PR includes:

A AlibabaCloudOpenSearch class that can connect to the Alibaba Cloud OpenSearch instance.
add embedings and metadata into a opensearch datasource.
querying by squared euclidean and metadata.
integration tests.
ipython notebook and docs.

I have read your contributing guidelines. And I have passed the tests below

  • make format
  • make lint
  • make coverage
  • make test

@zhaoshengbo zhaoshengbo changed the title Add alibabacloud opensearch vectorstore Add Alibaba Cloud OpenSearch vector store Jun 14, 2023
@zhaoshengbo zhaoshengbo changed the title Add Alibaba Cloud OpenSearch vector store Add Alibaba Cloud OpenSearch as a new vector store Jun 14, 2023
@vercel
Copy link

vercel bot commented Jun 18, 2023

@zhaoshengbo is attempting to deploy a commit to the LangChain Team on Vercel.

A member of the Team first needs to authorize it.

hwchase17
hwchase17 reviewed Jun 18, 2023
View reviewed changes
langchain/vectorstores/alibabacloud_opensearch.py Outdated
from hashlib import sha1
from typing import Any, Dict, Iterable, List, Optional, Tuple

from alibabacloud_ha3engine import client, models
Copy link
Contributor

@hwchase17 hwchase17 Jun 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these will need to be optional imports

Copy link
Contributor Author

@zhaoshengbo zhaoshengbo Jun 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for reviewing, I have fixed it.

pyproject.toml Outdated
nebula3-python = {version = "^3.4.0", optional = true}
langchainplus-sdk = ">=0.0.9"
awadb = {version = "^0.3.3", optional = true}
alibabacloud-ha3engine = "^1.3.4"
Copy link
Contributor

@hwchase17 hwchase17 Jun 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will need to be optional

Copy link
Contributor Author

@zhaoshengbo zhaoshengbo Jun 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for reviewing, I have fixed it.

@rlancemartin
Copy link
Contributor

rlancemartin commented Jun 19, 2023

Thanks for the contribution! I will review and help get this merged tomorrow.

@rlancemartin
Copy link
Contributor

rlancemartin commented Jun 20, 2023 •
edited
Loading

@zhaoshengbo can you please rebase on current master?

Your branch has some merge conflicts, as you can see in the tests.

I checked out your branched and attempted to rebase, confirmed these:

CONFLICT (file location): docs/integrations/alibabacloud_opensearch.md added in f963630f (add alibaba cloud openseearch vector store) inside a directory that was renamed in HEAD, suggesting it should perhaps be moved to docs/extras/ecosystem/integrations/alibabacloud_opensearch.md.
CONFLICT (file location): docs/modules/indexes/vectorstores/examples/alibabacloud_opensearch.ipynb added in f963630f (add alibaba cloud openseearch vector store) inside a directory that was renamed in HEAD, suggesting it should perhaps be moved to docs/extras/modules/data_connection/vectorstores/integrations/alibabacloud_opensearch.ipynb.
Auto-merging langchain/vectorstores/__init__.py
CONFLICT (content): Merge conflict in langchain/vectorstores/__init__.py
Auto-merging poetry.lock
CONFLICT (content): Merge conflict in poetry.lock
Auto-merging pyproject.toml
CONFLICT (content): Merge conflict in pyproject.toml

These are quick-fixes.

@rlancemartin rlancemartin force-pushed the add-alibabacloud-opensearch-vectorstore branch from d2b7aa8 to 9aab2fe Compare June 20, 2023 16:55
@rlancemartin
Copy link
Contributor

rlancemartin commented Jun 20, 2023 •
edited
Loading

Just rebased for you, and kicked off testing. Hopefully this clears us to merge.

rlancemartin
rlancemartin approved these changes Jun 20, 2023
View reviewed changes
Copy link
Contributor

@rlancemartin rlancemartin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@rlancemartin rlancemartin merged commit ab44c24 into langchain-ai:master Jun 20, 2023
@zhaoshengbo
Copy link
Contributor Author

zhaoshengbo commented Jun 20, 2023

Nice work!

thanks for your help

@zhaoshengbo
Copy link
Contributor Author

zhaoshengbo commented Jun 20, 2023

Just rebased for you, and kicked off testing. Hopefully this clears us to merge.

thanks for your help

bdonkey added a commit to bdonkey/langchain that referenced this pull request Jun 20, 2023
@bdonkey
Merge branch 'master' into ssdev
2feb704 
* master: (158 commits)
  Fix link (langchain-ai#6501)
  docs/fix links (langchain-ai#6498)
  Update notebook for MD header splitter and create new cookbook (langchain-ai#6399)
  Vector store support for Cassandra (langchain-ai#6426)
  improve documentation on base chain (langchain-ai#6468)
  fix: change ddg to DDGS (langchain-ai#6480)
  release 207 (langchain-ai#6488)
  Add Alibaba Cloud OpenSearch as a new vector store (langchain-ai#6154)
  fix openai qa chain (langchain-ai#6487)
  add FunctionMessage support to `_convert_dict_to_message()` in OpenAI chat model (langchain-ai#6382)
  bump version to 206 (langchain-ai#6465)
  fix neo4j schema query (langchain-ai#6381)
  Update serpapi.py Support baidu list type answer_box (langchain-ai#6386)
  fix: llm caching for replicate (langchain-ai#6396)
  feat: use latest duckduckgo_search API to call (langchain-ai#6409)
  Harrison/unstructured page number (langchain-ai#6464)
  Improve error message (langchain-ai#6275)
  Fix the issue where ANTHROPIC_API_URL set in environment is not takin… (langchain-ai#6400)
  Fix broken links in autonomous agents docs (langchain-ai#6398)
  Update SinglStoreDB vectorstore (langchain-ai#6423)
  ...
This was referenced Jun 25, 2023
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hwchase17 hwchase17 hwchase17 left review comments

@rlancemartin rlancemartin rlancemartin approved these changes

Assignees

@rlancemartin rlancemartin

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zhaoshengbo @rlancemartin @hwchase17