Changes related to Meilisearch v1.13.0

[curquiza]

This issue gathers the changes related to the v1.13.0 of Meilisearch that will impact the integrations scope.

📅 Release date: 17th February

Timelines & steps

Pre-release

• With the help of the changelog & this CI, define which integrations should be updated and how (New feature? Update README? Update tests?)
  • Fill in the "What to implement?" section below in this issue 👇. Minial implementation: PHP, JS, meilisearch-js-plugins.
  • Open implementation issues in repositories that need implementation if necessary for developers who will work on the tasks
  • Discuss with the Product team if needed. At least share this issue to let them know about the decisions.
• Create a branch by running Octopus script: only open branches for the integrations we choose to update (defined in the previous step) + Kubernetes repository + Cloud provider repository (changing the version)
• Update integrations according to the decisions (cf "What to implement?" section below in this issue 👇)
  ⚠️ If possible, this step is done before pre-release, once the feature is ready thanks to the prototype released by the engine team
  • JS
  • PHP
  • Dart
  • Go
  • Java
  • Python
  • Ruby
  • Rust
• Add code samples for the chosen up-to-date integrations with the new version of Meilisearch
• Update the library version of the related integrations and prepare the changelogs

Release day

• Release the integrations or only merge the related PRs (sometimes a release is not needed)
  • JS v0.49
  • PHP v1.13
  • meilisearch-js-plugins (at least update meilisearch-js version)
  • Dart v0.17
  • Go v0.31
  • Python v0.34
  • Rust v0.28
• Merge the related PR in K8s repository
• Publish DevOps tools:
  • create the git tag
  • publish images (steps are in CONTRIBUTING.md)
• Remove useless AWS images (by using our internal script)
• Open issues in the repositories that are not up-to-date with the latest version of Meilisearch (including code samples)

Post-release

• Open issues post-release for stabilization of AI

Pre-release: What to implement?

AI stabilization

Related issue in the engine: meilisearch/meilisearch#4733

Usage: https://meilisearch.notion.site/v1-13-AI-search-changes-17a4b06b651f80538b65d31724545def

• removal of the vectorStore experimental feature
• new error codes

TODO:

• JS: remove vectorStore exp feature + fix tests
• PHP: remove vectorStore exp feature + fix tests
• Dart: Remove vector store experimental feature · Issue #409 · meilisearch/meilisearch-dart Why are tests not breaking?
• Go: Remove vector store experimental feature · Issue #605 · meilisearch/meilisearch-go
• Java: Remove vector store experimental feature · Issue #812 · meilisearch/meilisearch-java
• Python: Remove vector store experimental feature · Issue #1077 · meilisearch/meilisearch-python
• Ruby: Remove vector store experimental feature · Issue #605 · meilisearch/meilisearch-ruby
• Rust: Remove vector store experimental feature · Issue #644 · meilisearch/meilisearch-rust

Postponed to v1.14.0
### Granular filterable attributes

Related issue in the engine: meilisearch/meilisearch#5163

Usage: https://meilisearch.notion.site/API-usage-Settings-to-opt-out-indexing-features-filterableAttributes-1764b06b651f80aba8bdf359b2df3ca8?pvs=74

• accept the new formal of filterable attributes (the old one is still allowed)

TODO:

• JS: adapt code base + fix tests
• [ ] PHP: adapt code base + fix tests

Post release day - implement missing feature for vector store

Spec

• embedders setting. Methods getEmbedders, updateEmbedders, resetEmbedders. Also, the method updateSettings should be able to accept the new embedders field. Here is the list of the acceptable sub fields:
  • source sub field is available and accepts: ollama, rest, openAI, huggingFace and userProvided
  • apiKey sub field is available (string) - optional because not compatible with all sources. Only for openAi, ollama, rest.
  • model sub field is available (string) - optional because not compatible with all sources. Only for ollama, openAI, huggingFace
  • documentTemplate sub field is available (string) - optional
  • dimensions - optional because not compatible with all sources. Only for openAi, huggingFace, ollama, and rest
  • distribution - optional
  • request - mandatory only if using rest embedder
  • response - mandatory only if using rest embedder
  • documentTemplateMaxBytes - optional
  • revision - optional, only for huggingFace
  • headers - optional, only for rest
  • binaryQuantized - optional
• Search
  • hybrid search parameter, with sub fields semanticRatio and embedder. embedder is mandatory if hybrid is set.
  • vector parameter is available
  • retrieveVectors parameter available
  • semanticHitCount in search response
  • Accept _semanticScore in the search response (optional)
  • vector should be returned in the search response, but optional (because depends on search parameters)
  • _vectors should NOT be present in the search response
• Similar. Implement searchSimilarDocuments associated with the POST /indexes/:uid/similar. Do NOT implement with GET.

TODO

• Java: [v1.13] Stabilize AI-powered search meilisearch-java#817
• Python: [v1.13] Stabilize AI-powered search meilisearch-python#1081
• Ruby: [v1.13] Stabilize AI-powered search meilisearch-ruby#608

Post-release: Remote federated search requests (exp)

Related issue in the engine: meilisearch/meilisearch#4980

Usage: https://meilisearch.notion.site/API-usage-Remote-search-request-f64fae093abf409e9434c9b9c8fab6f3

• New remote route
• accept remote search parameter in /multi-search route

TODO:

• JS: [v1.13] Remote federated search requests (experimental) meilisearch-js#1872
• PHP: [v1.13] Remote federated search requests (experimental) meilisearch-php#725

Post-release: New usedDatabaseSize in /stats

Usage: meilisearch/meilisearch#5318

• Accept the usedDatabaseSize in the response of the getStats method

TODO (Updated: removed from scope)

• [ ] JS
• PHP: no need to change 🎉

Issue to open after the release: .NET, dart, go, Java, Rust, Swift (not Ruby, or Python)

Post-release: Add embbedings database metrics to stats route

Usage: meilisearch/meilisearch#5321

• Accept indexes.INDEXNAME.numberOfEmbeddedDocuments in the response of the getStats method
• Accept indexes.INDEXNAME.numberOfEmbeddings in the response of the getStats method

TODO (Updated: removed from scope)

• [ ] JS
• PHP: no need to change 🎉

Post-release (v1.13.1): Add documents database metrics to stats route

meilisearch/meilisearch#5319

• Accept indexes.INDEXNAME.rawDocumentDbSize in the response of the getStats method
• [ ] Accept indexes.INDEXNAME.maxDocumentSize in the response of the getStats method
• Accept indexes.INDEXNAME.avgDocumentSize in the response of the getStats method

TODO (Updated: removed from scope)

• [ ] JS
• PHP: no need to change 🎉