Some pages on jqueryui.com appear unindexed (or dominated by changelog matches)

[Krinkle]

Reported by @mgol in #jquery_dev:gitter.im on Matrix:

when searching for "transfer" on jqueryui.com, no API pages are reported.
Searching for "effect" does uncover https://api.jqueryui.com/effect/.
However, the page https://api.jqueryui.com/transfer/ exists and
is not even new. Why is it not showing up?

The Typesense config for jqueryui.com was added in jquery/jqueryui.com@6cf42a4, ref #33.

[Krinkle]

The first issue I see is that changelog pages may be dominating the results.

When I take the ajax request for the "transfer" search, and increase to result limit from 5 to 15, I do see the effect page show up as result number 8. Typesense Link

Ranking is mostly automatic based on relevant to the overall document and relevance/importance to a given page (e.g. 1 match in the page title outranks many matches in a footnote).

Ranking can be finetuned via the config file at: https://github.com/jquery/jqueryui.com/blob/main/docsearch.config.json

Both jquery.com and jqueryui.com are configured to rank the handful of base domain pages (e.g. "download", "contact", "support" and such) lower than any API pages. This is something we kept as-is when we moved from Algolia. I don't know for sure, but I believe this was because they're already prominently linked in the site menu, and tend to contain mentions of APIs but are not the main landing page for those of course.

Changelog pages are especially noisey in this regard, and should be ranked even lower, if at all.
In Algolia we used to de-rank these completely

https://github.com/algolia/docsearch-configs/blob/b0630cbffa079a670a98d7c2f8e235ec7b8c50d7/configs/jqueryui.json

(This file is linked from https://github.com/jquery/infrastructure-puppet/blob/HEAD/doc/search.md#algolia-docsearch).

When we migrated jQuery UI I mainly copied what we did for jQuery Core instead of checking back with the Algolia config, so we lost that tuning. Let me fix that.

[Krinkle]

There appears to be a second issue, though, because indeed some pages seem to be impossible to match, even with a higher limit and/or more narrow query.

Here is query for an exact sentence on the transfer() page on api.jqueryu.com, and there is only 1 result, and it isn't this page. This means it isn't just about ranking weight, the page appears just unindexed?

I'll look at that first actually, since I'd rather add and see where it ranks, than to first mess with the ranking and then add more content.

[Krinkle]

Looking at a recent scrape, it doesn't even visit this page:

https://github.com/jquery/jqueryui.com/actions/runs/11598167321/job/32293407235

I guess that's because there's no path to discover this page, when starting from the home page. There's no link from content in another page, to this one. With the exception of category pages like https://api.jqueryui.com/category/effects-core/, but we exclude these since we don't want to index those.

This works today via stop_urls in jqueryui.com's docsearch.config.json. This controls which links it follows. When it matches it will not even visit that page. This means category pages are not crawled, not followed, and not indexed. Documentation via https://typesense.org/docs/guide/docsearch.html.

This isn't an issue for jQuery Core at api.jquery.com, because that site has a home page listing all API pages.

There's broadly three ways to solve this:

1. Remove /category/ from stop_urls, so that they start being crawled and followed. Then, to re-create the (desirable) behaviour we have today, of of not indexing them for their text content, we can add something like body.category to selectors_exclude so that they are treated as empty pages when extracting text records for the actual search index. This might still create an entry for the page titled "Category: Effects" but with no content. Not great.
2. Remove /category/ from stop_urls, so that they start being crawled and followed. Then add something like <div class="category-archive-meta"> to stop_content so that the page is skipped entirely for content indexing, thus not even leaving a "Category: Effects" stub.
3. Add sitemap_urls, and point it to https://api.jqueryui.com/wp-sitemap.xml and https://jqueryui.com/wp-sitemap.xml. Leave stop_urls unchanged.

I'll try the last one. Sitemaps best represent want we want: Follow and noindex.

[Krinkle]

The first issue I see is that changelog pages may be dominating the results.

This is now fixed: