Skip to content

Fix behavior for _index LIKE for ESQL #130849

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 53 commits into from
Jul 9, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
ada1a42
Get basic case to work
julian-elastic Jun 25, 2025
1cc6167
Add ExpressionQueryBuilder with serialization
julian-elastic Jun 25, 2025
a96672f
[CI] Auto commit changes from spotless
Jun 26, 2025
c556367
Get _index like working newToNew
Jun 26, 2025
2b2d3f9
[CI] Auto commit changes from spotless
Jun 26, 2025
993f61e
Move functionality from AutomatonTranslatable to TranslationAware
julian-elastic Jun 26, 2025
9d43d0e
Remove AutomatonQueryBuilder and EsqlAutomatonQuery
julian-elastic Jun 26, 2025
f393916
Clean up
julian-elastic Jun 26, 2025
afa28f3
Clean up and comments
julian-elastic Jun 26, 2025
e7310fc
working on getting the tests to pass in bwc mode
julian-elastic Jun 26, 2025
887cafa
[CI] Auto commit changes from spotless
Jun 26, 2025
f7d162b
Revert "Remove AutomatonQueryBuilder and EsqlAutomatonQuery"
julian-elastic Jun 27, 2025
bf0b19f
Remove skipped check
julian-elastic Jun 27, 2025
7761b62
Fix unit tests failing
julian-elastic Jun 27, 2025
b3717a3
[CI] Auto commit changes from spotless
Jun 27, 2025
2b3971b
Fix transport protocol issues with the unit tests
julian-elastic Jun 27, 2025
6da11e7
Clean up classes no longer needed
julian-elastic Jun 27, 2025
ccf9561
[CI] Auto commit changes from spotless
Jun 27, 2025
20c11dc
bigfix
julian-elastic Jun 27, 2025
f0a7ca6
Fix more failing UTs
julian-elastic Jun 27, 2025
40299d1
Fix more failing UTs
julian-elastic Jun 27, 2025
b6a1497
[CI] Auto commit changes from spotless
Jun 27, 2025
a2911c4
Revert change in SearchShardsRequest
julian-elastic Jun 27, 2025
4fe6eb8
Revert change in SearchShardsRequest
julian-elastic Jun 27, 2025
4b814be
Address code review feedback part 1
julian-elastic Jun 30, 2025
a63f24f
Update docs/changelog/130019.yaml
julian-elastic Jul 1, 2025
8eb4a47
Address code review feedback part 2 (still needs UT with the flag set…
julian-elastic Jul 1, 2025
9f33f83
[CI] Auto commit changes from spotless
Jul 1, 2025
fba667c
UT fixes
julian-elastic Jul 1, 2025
128d190
Fix merge errors
julian-elastic Jul 1, 2025
5f6356e
Add Supplier for Automaton and CharacterRunAutomaton
julian-elastic Jul 1, 2025
a3f2a3c
Fix UT errors
julian-elastic Jul 1, 2025
bbfb65d
Fix UT errors
julian-elastic Jul 1, 2025
82577ae
Change behavior for LIKE LIST and _index
julian-elastic Jul 1, 2025
4fc0dfb
Plan serialization?
nik9000 Jul 1, 2025
1536e59
fixup
nik9000 Jul 1, 2025
a9e1dad
Address no commits
julian-elastic Jul 2, 2025
7b9c27c
Fix failing UTs
julian-elastic Jul 2, 2025
fe7b1a8
[CI] Auto commit changes from spotless
Jul 2, 2025
619b0df
Remove some debugging print statements
julian-elastic Jul 3, 2025
79f2e46
[CI] Auto commit changes from spotless
Jul 3, 2025
323ae64
Move flag to EsqlFlags
nik9000 Jul 3, 2025
90ec86f
Clean up and fix UTs
julian-elastic Jul 3, 2025
516ed37
[CI] Auto commit changes from spotless
Jul 3, 2025
a049858
Fix UT fail
julian-elastic Jul 3, 2025
fe3165d
Address code review comments
julian-elastic Jul 7, 2025
d1610e6
Fix merge errors
julian-elastic Jul 8, 2025
5e6b23b
Merge branch 'main' into indexLike_final
julian-elastic Jul 8, 2025
0a56a0d
Merge branch 'main' into indexLike_final
julian-elastic Jul 9, 2025
6aad43d
Update docs/changelog/130849.yaml
julian-elastic Jul 9, 2025
b43ce3f
Address code review comments
julian-elastic Jul 9, 2025
081b473
Merge remote-tracking branch 'origin/indexLike_final' into indexLike_…
julian-elastic Jul 9, 2025
ee49073
Merge branch 'main' into indexLike_final
julian-elastic Jul 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,11 @@ public class QueryPlanningBenchmark {
private EsqlParser defaultParser;
private Analyzer manyFieldsAnalyzer;
private LogicalPlanOptimizer defaultOptimizer;
private Configuration config;

@Setup
public void setup() {

var config = new Configuration(
this.config = new Configuration(
DateUtils.UTC,
Locale.US,
null,
Expand Down Expand Up @@ -116,7 +116,7 @@ public void setup() {
}

private LogicalPlan plan(EsqlParser parser, Analyzer analyzer, LogicalPlanOptimizer optimizer, String query) {
var parsed = parser.createStatement(query, new QueryParams(), telemetry);
var parsed = parser.createStatement(query, new QueryParams(), telemetry, config);
var analyzed = analyzer.analyze(parsed);
var optimized = optimizer.optimize(analyzed);
return optimized;
Expand Down
6 changes: 6 additions & 0 deletions docs/changelog/130849.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
pr: 130849
summary: Fix behavior for `_index` LIKE for ESQL
area: ES|QL
type: bug
issues:
- 129511
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,7 @@ static TransportVersion def(int id) {
public static final TransportVersion ESQL_SPLIT_ON_BIG_VALUES = def(9_116_0_00);
public static final TransportVersion ESQL_LOCAL_RELATION_WITH_NEW_BLOCKS = def(9_117_0_00);
public static final TransportVersion ML_INFERENCE_CUSTOM_SERVICE_EMBEDDING_TYPE = def(9_118_0_00);
public static final TransportVersion ESQL_FIXED_INDEX_LIKE = def(9_119_0_00);

/*
* STOP! READ THIS FIRST! No, really,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
import org.apache.lucene.search.MultiTermQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.automaton.Automaton;
import org.apache.lucene.util.automaton.CharacterRunAutomaton;
import org.elasticsearch.common.lucene.search.Queries;
import org.elasticsearch.common.regex.Regex;
import org.elasticsearch.core.Nullable;
Expand All @@ -23,6 +25,7 @@

import java.util.Collection;
import java.util.Map;
import java.util.function.Supplier;

/**
* A {@link MappedFieldType} that has the same value for all documents.
Expand Down Expand Up @@ -135,9 +138,47 @@ public final Query wildcardQuery(String value, boolean caseInsensitive, QueryRew
}
}

/**
* Returns a query that matches all documents or no documents
* It usually calls {@link #wildcardQuery(String, boolean, QueryRewriteContext)}
* except for IndexFieldType which overrides this method to use its own matching logic.
*/
public Query wildcardLikeQuery(String value, boolean caseInsensitive, QueryRewriteContext context) {
return wildcardQuery(value, caseInsensitive, context);
}

@Override
public final boolean fieldHasValue(FieldInfos fieldInfos) {
// We consider constant field types to always have value.
return true;
}

/**
* Returns the constant value of this field as a string.
* Based on the field type, we need to get it in a different way.
*/
public abstract String getConstantFieldValue(SearchExecutionContext context);

/**
* Returns a query that matches all documents or no documents
* depending on whether the constant value of this field matches or not
*/
@Override
public Query automatonQuery(
Supplier<Automaton> automatonSupplier,
Supplier<CharacterRunAutomaton> characterRunAutomatonSupplier,
@Nullable MultiTermQuery.RewriteMethod method,
SearchExecutionContext context,
String description
) {
CharacterRunAutomaton compiled = characterRunAutomatonSupplier.get();
boolean matches = compiled.run(getConstantFieldValue(context));
if (matches) {
return new MatchAllDocsQuery();
} else {
return new MatchNoDocsQuery(
"The \"" + context.getFullyQualifiedIndex().getName() + "\" query was rewritten to a \"match_none\" query."
);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,13 @@
package org.elasticsearch.index.mapper;

import org.apache.lucene.search.MatchAllDocsQuery;
import org.apache.lucene.search.MatchNoDocsQuery;
import org.apache.lucene.search.MultiTermQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.util.BytesRef;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.regex.Regex;
import org.elasticsearch.core.Nullable;
import org.elasticsearch.index.fielddata.FieldData;
import org.elasticsearch.index.fielddata.FieldDataContext;
import org.elasticsearch.index.fielddata.IndexFieldData;
Expand All @@ -27,6 +31,7 @@

import java.util.Collections;
import java.util.List;
import java.util.Locale;

public class IndexFieldMapper extends MetadataFieldMapper {

Expand Down Expand Up @@ -102,6 +107,38 @@ public StoredFieldsSpec storedFieldsSpec() {
};
}

@Override
public Query wildcardLikeQuery(
String value,
@Nullable MultiTermQuery.RewriteMethod method,
boolean caseInsensitve,
SearchExecutionContext context
) {
String indexName = context.getFullyQualifiedIndex().getName();
return getWildcardLikeQuery(value, caseInsensitve, indexName);
}

@Override
public Query wildcardLikeQuery(String value, boolean caseInsensitive, QueryRewriteContext context) {
String indexName = context.getFullyQualifiedIndex().getName();
return getWildcardLikeQuery(value, caseInsensitive, indexName);
}

private static Query getWildcardLikeQuery(String value, boolean caseInsensitve, String indexName) {
if (caseInsensitve) {
value = value.toLowerCase(Locale.ROOT);
indexName = indexName.toLowerCase(Locale.ROOT);
}
if (Regex.simpleMatch(value, indexName)) {
return new MatchAllDocsQuery();
}
return new MatchNoDocsQuery("The \"" + indexName + "\" query was rewritten to a \"match_none\" query.");
}

@Override
public String getConstantFieldValue(SearchExecutionContext context) {
return context.getFullyQualifiedIndex().getName();
}
}

public IndexFieldMapper() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ protected boolean matches(String pattern, boolean caseInsensitive, QueryRewriteC
return Regex.simpleMatch(pattern, indexMode, caseInsensitive);
}

@Override
public String getConstantFieldValue(SearchExecutionContext context) {
return context.getIndexSettings().getMode().getName();
}

@Override
public Query existsQuery(SearchExecutionContext context) {
return new MatchAllDocsQuery();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,15 @@
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.MultiTerms;
import org.apache.lucene.index.Term;
import org.apache.lucene.index.Terms;
import org.apache.lucene.index.TermsEnum;
import org.apache.lucene.search.MultiTermQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.automaton.Automata;
import org.apache.lucene.util.automaton.Automaton;
import org.apache.lucene.util.automaton.CharacterRunAutomaton;
import org.apache.lucene.util.automaton.CompiledAutomaton;
import org.apache.lucene.util.automaton.CompiledAutomaton.AUTOMATON_TYPE;
import org.apache.lucene.util.automaton.Operations;
Expand All @@ -51,6 +53,7 @@
import org.elasticsearch.index.fielddata.SourceValueFetcherSortedBinaryIndexFieldData;
import org.elasticsearch.index.fielddata.StoredFieldSortedBinaryIndexFieldData;
import org.elasticsearch.index.fielddata.plain.SortedSetOrdinalsIndexFieldData;
import org.elasticsearch.index.query.AutomatonQueryWithDescription;
import org.elasticsearch.index.query.SearchExecutionContext;
import org.elasticsearch.index.similarity.SimilarityProvider;
import org.elasticsearch.script.Script;
Expand Down Expand Up @@ -82,6 +85,7 @@
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.function.Supplier;

import static org.apache.lucene.index.IndexWriter.MAX_TERM_LENGTH;
import static org.elasticsearch.core.Strings.format;
Expand Down Expand Up @@ -1042,6 +1046,17 @@ public IndexSortConfig getIndexSortConfig() {
public boolean hasDocValuesSkipper() {
return hasDocValuesSkipper;
}

@Override
public Query automatonQuery(
Supplier<Automaton> automatonSupplier,
Supplier<CharacterRunAutomaton> characterRunAutomatonSupplier,
@Nullable MultiTermQuery.RewriteMethod method,
SearchExecutionContext context,
String description
) {
return new AutomatonQueryWithDescription(new Term(name()), automatonSupplier.get(), description);
}
}

private final boolean indexed;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@
import org.apache.lucene.search.Query;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.automaton.Automaton;
import org.apache.lucene.util.automaton.CharacterRunAutomaton;
import org.elasticsearch.ElasticsearchException;
import org.elasticsearch.ElasticsearchParseException;
import org.elasticsearch.cluster.metadata.IndexMetadata;
Expand Down Expand Up @@ -54,6 +56,7 @@
import java.util.Objects;
import java.util.Set;
import java.util.function.Function;
import java.util.function.Supplier;

import static org.elasticsearch.search.SearchService.ALLOW_EXPENSIVE_QUERIES;

Expand Down Expand Up @@ -329,6 +332,19 @@ public final Query wildcardQuery(String value, @Nullable MultiTermQuery.RewriteM
return wildcardQuery(value, method, false, context);
}

/**
* Similar to wildcardQuery, except that we change the behavior for ESQL
* to behave like a string LIKE query, where the value is matched as a string
*/
public Query wildcardLikeQuery(
String value,
@Nullable MultiTermQuery.RewriteMethod method,
boolean caseInsensitve,
SearchExecutionContext context
) {
return wildcardQuery(value, method, caseInsensitve, context);
}

public Query wildcardQuery(
String value,
@Nullable MultiTermQuery.RewriteMethod method,
Expand Down Expand Up @@ -370,6 +386,23 @@ public Query regexpQuery(
);
}

/**
* Returns a Lucine pushable Query for the current field
* For now can only be AutomatonQuery or MatchAllDocsQuery() or MatchNoDocsQuery()
*/
public Query automatonQuery(
Supplier<Automaton> automatonSupplier,
Supplier<CharacterRunAutomaton> characterRunAutomatonSupplier,
@Nullable MultiTermQuery.RewriteMethod method,
SearchExecutionContext context,
String description
) {
throw new QueryShardException(
context,
"Can only use automaton queries on keyword fields - not on [" + name + "] which is of type [" + typeName() + "]"
);
}

public Query existsQuery(SearchExecutionContext context) {
if (hasDocValues() || (isIndexed() && getTextSearchInfo().hasNorms())) {
return new FieldExistsQuery(name());
Expand Down

This file was deleted.

Loading
Loading