Slow peer recovery of vector data

[weizijun]

Elasticsearch Version

main

Installed Plugins

No response

Java Version

bundled

OS Version

null

Problem Description

we find that the vector data input is slow in MemorySegmentIndexInput case, this result slow io in flat type vector search and peer recovery.
In peer recovery case, even if set indices.recovery.max_bytes_per_sec to a large value like “4000mb”, the rate of vec file copy is slow.
The size of every read size is 8kb, it can’t use the readahead mechanism.
Do you know this case, is some feature about MemorySegmentIndexInput, or if we don’t call MemorySegmentIndexInput.prefetch, it can’t use the readahead?
this is the iostat record:

I added a line of code in RecoverySourceHandler:

currentInput.updateReadAdvice(ReadAdvice.SEQUENTIAL);

The rate of vec file copy is fast. The size of every read size become bigger.

Steps to Reproduce

peer recovery of vector data

Logs (if relevant)

No response

[weizijun]

hi, @benwtrent ,can you help to see the issue?

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[DaveCTurner]

The proposal here is AIUI:

diff --git a/server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java b/server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java
index 622e56f596e1..2aec98a215ef 100644
--- a/server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java
+++ b/server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java
@@ -1315,6 +1315,7 @@ public class RecoverySourceHandler {
                         currentInput = null;
                     } else {
                         currentInput = store.directory().openInput(md.name(), IOContext.DEFAULT);
+                        currentInput.updateReadAdvice(ReadAdvice.SEQUENTIAL);
                     }
                 }

Makes sense to me. We should probably do something similar for snapshots too:

diff --git a/server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java b/server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java
index dea02908d215..58944348b3de 100644
--- a/server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java
+++ b/server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java
@@ -4125,6 +4125,7 @@ public abstract class BlobStoreRepository extends AbstractLifecycleComponent imp
             Releasable ignored = context.withCommitRef();
             IndexInput indexInput = store.openVerifyingInput(file, IOContext.DEFAULT, fileInfo.metadata())
         ) {
+            indexInput.updateReadAdvice(ReadAdvice.SEQUENTIAL);
             for (int i = 0; i < fileInfo.numberOfParts(); i++) {
                 final long partBytes = fileInfo.partBytes(i);

[weizijun]

This feature affects not only the recovery rate, but also the io rate of the flat vector.
Even if the flat vector query is sequential read, the size of every read size is 8kb.

[DaveCTurner]

This feature affects not only the recovery rate, but also the io rate of the flat vector.

That's a separate issue - at least, it's a different team that owns this question, and we can't route this one issue to both teams. Let's keep this one focussed on the peer recovery (and snapshot) aspect.