[Question] New demo type/use case: semantic search (SemanticFinder)

[do-me]

Hi @xenova,
first of all thanks for the amazing library - it's awesome to be able to play around with the models without a backend!

I just created SemanticFinder, a semantic search engine in the browser with the help of transformers.js and sentence-transformers/all-MiniLM-L6-v2.

You can find some technical details in the blog post.

I was wondering whether you'd be interested in showcasing semantic search as new demo type. Technically, it's not a new model but it's a new use case with an existing model so I don't know whether it's out of scope.

Anyway, just wanted to let you know that you're work is very much appreciated!

[xenova]

This is so cool! I plan to completely rewrite the demo application which, as you can tell, is extremely simple... so this definitely sounds like something I can add!

PS: Do you have a Twitter post I can retweet? I'd love to share it! Edit: Found it!

[xenova]

@do-me Just a heads up that I updated the feature-extraction API to support other models (not just sentence-transformers). To use the updated API, you just need to add { pooling: 'mean', normalize: true } to the pipeline call. Your demo site seems unaffected (as it is still using the previous version), but if you'd like to add support for other models, you can make the following changes:

For example:

Before:

let extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
let result = await extractor('This is a simple test.');
console.log(result);
// Tensor {
//     type: 'float32',
//     data: Float32Array [0.09094982594251633, -0.014774246141314507, ...],
//     dims: [1, 384]
// }

After:

let extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
let result = await extractor('This is a simple test.', { pooling: 'mean', normalize: true });
console.log(result);
// Tensor {
//     type: 'float32',
//     data: Float32Array [0.09094982594251633, -0.014774246141314507, ...],
//     dims: [1, 384]
// }

---

And if you don't want to do pooling/normalization, you can leave it out. You will then get the embeddings for each token in the sequence.

[xenova]

Also - we're planning on releasing a semantic search demo next week 🥳 (so, watch this space!)

[do-me]

This is awesome, thanks for pinging me!

I'm very interested in this feature, mainly for speed improvements. Do you have some benchmarks at hand how the new pooling approach compares to sequential processing?

Also, I'd be curious to know if there's a sweet spot somewhere how many elements could/should be passed to the model at once.

And one more detail but it's probably also model dependent: can you track the progress of a batch/pool that has been passed to the model? E.g. if I pass 1000 elements at once, is there any theoretic way to return the progress so I can update the progress bar in the frontend meanwhile?

[do-me]

fyi
SemanticFinder just had a great contribution from @VarunNSrivastava improving the UI significantly with new features. Also updated the transformers.js version: New Demo

[lizozom]

Hey, joining the semantic search on the FE party 🥳 .

I'm wondering if we can leverage the power of threads in this scenario by setting env.backends.onnx.wasm.numThreads = 4.
I don't see any errors throw, but also no drastic performance improvements.

[xenova]

@lizozom Hi there! 👋

So, the most likely reason for this is that SharedArrayBuffer is not available because COOP/COEP headers are not set for the hosted files. You can check your network tab when running the model and you should see ort-wasm-simd.wasm loaded instead of ort-wasm-simd-threaded.wasm. For more information, check out this related open issue: #161

To fix this, it depends where you are hosting the website, as these headers must be set by the server. At the moment, GitHub pages does not offer this (https://github.com/orgs/community/discussions/13309), but there are some workarounds (cc @josephrocca). On the other hand, we are actively working to support this feature in Hugging Face spaces (huggingface/huggingface_hub#1525), which should hopefully be ready soon!

[do-me]

Seems like netlify offers a little more flexibility. I'm a very happy user of netlify (hosting my blog there since 2019 without any trouble) and it's pretty easy to link a GitHub repo to it. @lizozom if needed, we might consider switching from GitHub pages to netlify.

[lizozom]

Cool!
I'll check and let you know.

[josephrocca]

Current workaround is to put this file beside your HTML file, and then import it with a script tag in your document <head>. The Github Pages engineering lead said a few days ago that they are working on custom headers but there's no ETA.

I personally wouldn't go with Netlify, since their pricing is a bit to aggressive for my use cases, but depends on what you're doing. Netlify's free 100GB could be used up very quickly if you have a few assets like ML models or videos or whatever (even just a few thousand visitors - e.g. due to being shared on Twitter or HN). Cloudflare Pages is much better imo (unlimited bandwidth and requests for free), but again it depends on your use case - Netlify may suffice.

[do-me]

Thanks for the hint! Does Cloudflare Pages offer custom headers? 
Unlimited bandwidth sounds indeed great! Will check it out.
Luckily we don't need to host the models but only the static page with the framework (currently everything bundled is ~2Mb) so it's not that bad but still something to keep in mind.

[josephrocca]

I haven't actually had to do that with Cloudflare Pages yet, but here are their docs for custom headers: https://developers.cloudflare.com/pages/platform/headers/

[lizozom]

I tested this out on a local webpack project, serving files with these headers:

  devServer: {
    headers: {
      'Cross-Origin-Opener-Policy': 'same-origin', 
      'Cross-Origin-Embedder-Policy': 'require-corp',
    },
  },

And indeed, this causes the threaded version (ort-wasm-simd-threaded.wasm) to be loaded.
I'm not seeing much of a performance difference right away, but I'll tinker with it some more.

@xenova In your opinion, should I expect to see performance improvements if I'm running a large batch of embeddings pipelines single vs. multi threaded?