Skip to content

Commit 8398263

Browse files
abdonpijpelinkabdonpijpelink
authored
[DOCS] ES|QL limitations (#101673) (#101776)
* Supported types * More limitations * Document the behavior of text fields * Review feedback * Review feedback
1 parent 1f4e3d1 commit 8398263

File tree

3 files changed

+142
-25
lines changed

3 files changed

+142
-25
lines changed

‎docs/reference/esql/esql-enrich-data.asciidoc

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,3 +137,9 @@ include::{es-repo-dir}/ingest/apis/enrich/execute-enrich-policy.asciidoc[tag=upd
137137
==== Update an enrich policy
138138

139139
include::../ingest/enrich.asciidoc[tag=update-enrich-policy]
140+
141+
==== Limitations
142+
// tag::limitations[]
143+
The {esql} `ENRICH` command only supports enrich policies of type `match`.
144+
Furthermore, `ENRICH` only supports enriching on a column of type `keyword`.
145+
// end::limitations[]

‎docs/reference/esql/esql-kibana.asciidoc

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -250,18 +250,20 @@ image::images/esql/esql-kibana-create-rule.png[align="center",width=50%]
250250
[[esql-kibana-limitations]]
251251
=== Limitations
252252

253+
// tag::limitations[]
253254
* The user interface to filter data is not enabled when Discover is in {esql}
254255
mode. To filter data, write a query that uses the <<esql-where>> command
255256
instead.
256257
* In {esql} mode, clicking a field in the field list in Discover does not show
257258
quick statistics for that field.
258259
* Discover shows no more than 10,000 rows. This limit only applies to the number
259-
of rows that are retrieved by the query and displayed in Discover. Any query or
260-
aggregation runs on the full data set.
260+
of rows that are retrieved by the query and displayed in Discover. Queries and
261+
aggregations run on the full data set.
261262
* Discover shows no more than 50 columns. If a query returns
262263
more than 50 columns, Discover only shows the first 50.
263264
* Querying many many indices at once without any filters can cause an error in
264265
kibana which looks like `[esql] > Unexpected error from Elasticsearch: The
265-
content length (536885793) is bigger than the maximum allowed string (536870888)`.
266-
The response from {esql} is too long. Use <<esql-drop>> or <<esql-keep>> to limit the number
267-
of fields returned.
266+
content length (536885793) is bigger than the maximum allowed string
267+
(536870888)`. The response from {esql} is too long. Use <<esql-drop>> or
268+
<<esql-keep>> to limit the number of fields returned.
269+
// end::limitations[]

‎docs/reference/esql/esql-limitations.asciidoc

Lines changed: 129 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -5,36 +5,145 @@
55
<titleabbrev>Limitations</titleabbrev>
66
++++
77

8-
This is work in progress.
8+
[discrete]
9+
[[esql-max-rows]]
10+
=== Result set size limit
11+
12+
By default, an {esql} query returns up to 500 rows. You can increase the number
13+
of rows up to 10,000 using the <<esql-limit>> command. Queries do not return
14+
more than 10,000 rows, regardless of the `LIMIT` command's value.
15+
16+
This limit only applies to the number of rows that are retrieved by the query
17+
and displayed in Discover. Queries and aggregations run on the full data set.
18+
19+
To overcome this limitation:
20+
21+
* Reduce the result set size by modifying the query to only return relevant
22+
data. Use <<esql-where>> to select a smaller subset of the data.
23+
* Shift any post-query processing to the query itself. You can use the {esql}
24+
<<esql-stats-by>> command to aggregate data in the query.
25+
* Increase the limit with the `esql.query.result_truncation_max_size` static
26+
cluster setting.
927

1028
[discrete]
1129
[[esql-supported-types]]
12-
=== Supported types
30+
=== Field types
31+
32+
[discrete]
33+
==== Supported types
1334

1435
{esql} currently supports the following <<mapping-types,field types>>:
1536

16-
** `alias`
17-
** `boolean`
18-
** `date`
19-
** `double` (`float`, `half_float`, `scaled_float` are represented as `double`)
20-
** `ip`
21-
** `keyword` family including `keyword`, `constant_keyword`, and `wildcard`
22-
** `int` (`short` and `byte` are represented as `int`)
23-
** `long`
24-
** `null`
25-
** `text`
26-
** `unsigned_long`
27-
** `version`
37+
* `alias`
38+
* `boolean`
39+
* `date`
40+
* `double` (`float`, `half_float`, `scaled_float` are represented as `double`)
41+
* `ip`
42+
* `keyword` family including `keyword`, `constant_keyword`, and `wildcard`
43+
* `int` (`short` and `byte` are represented as `int`)
44+
* `long`
45+
* `null`
46+
* `text`
47+
* `unsigned_long`
48+
* `version`
49+
50+
[discrete]
51+
==== Unsupported types
52+
53+
{esql} does not yet support the following field types:
54+
55+
* TSDB metrics
56+
** `counter`
57+
** `position`
58+
** `aggregate_metric_double`
59+
* Geo/spatial
60+
** `geo_point`
61+
** `geo_shape`
62+
** `point`
63+
** `shape`
64+
* Date/time
65+
** `date_nanos`
66+
** `date_range`
67+
* Other types
68+
** `binary`
69+
** `completion`
70+
** `dense_vector`
71+
** `double_range`
72+
** `float_range`
73+
** `histogram`
74+
** `integer_range`
75+
** `ip_range`
76+
** `long_range`
77+
** `nested`
78+
** `rank_feature`
79+
** `rank_features`
80+
** `search_as_you_type`
81+
82+
Querying a column with an unsupported type returns an error. If a column with an
83+
unsupported type is not explicitly used in a query, it is returned with `null`
84+
values, with the exception of nested fields. Nested fields are not returned at
85+
all.
86+
87+
[discrete]
88+
[[esql-limitations-text-fields]]
89+
=== `text` fields behave like `keyword` fields
90+
91+
While {esql} supports <<text,`text`>> fields, {esql} does not treat these fields
92+
like the Search API does. {esql} queries do not query or aggregate the
93+
<<analysis,analyzed string>>. Instead, an {esql} query will try to get a `text`
94+
field's subfield of the <<keyword,keyword family type>> and query/aggregate
95+
that. If it's not possible to retrieve a `keyword` subfield, {esql} will get the
96+
string from a document's `_source`. If the `_source` cannot be retrieved, for
97+
example when using synthetic source, `null` is returned.
98+
99+
Note that {esql}'s retrieval of `keyword` subfields may have unexpected
100+
consequences. An {esql} query on a `text` field is case-sensitive. Furthermore,
101+
a subfield may have been mapped with a <<normalizer,normalizer>>, which can
102+
transform the original string. Or it may have been mapped with <<ignore-above>>,
103+
which can truncate the string. None of these mapping operations are applied to
104+
an {esql} query, which may lead to false positives or negatives.
105+
106+
To avoid these issues, a best practice is to be explicit about the field that
107+
you query, and query `keyword` sub-fields instead of `text` fields.
28108

29109
[discrete]
30110
[[esql-tsdb]]
31-
=== {esql} and time series data
111+
=== Time series data streams are not supported
32112

33-
{esql} does not support time series data (TSDS).
113+
{esql} does not support querying time series data streams (TSDS).
34114

35115
[discrete]
36-
[[esql-max-rows]]
37-
=== 10,000 row maximum
116+
[[esql-limitations-date-math]]
117+
=== Date math limitations
118+
119+
Date math expressions work well when the leftmost expression is a datetime, for
120+
example:
121+
[source,txt]
122+
----
123+
now() + 1 year - 2hour + ...
124+
----
125+
126+
But using parentheses or putting the datetime to the right is not always supported yet. For example, the following expressions fail:
127+
[source,txt]
128+
----
129+
1year + 2hour + now()
130+
now() + (1year + 2hour)
131+
----
132+
133+
Date math does not allow subtracting two datetimes, for example:
134+
[source,txt]
135+
----
136+
now() - 2023-10-26
137+
----
138+
139+
[discrete]
140+
[[esql-limitations-enrich]]
141+
=== Enrich limitations
142+
143+
include::esql-enrich-data.asciidoc[tag=limitations]
144+
145+
[discrete]
146+
[[esql-limitations-kibana]]
147+
=== Kibana limitations
38148

39-
A single query will not return more than 10,000 rows, regardless of the
40-
`LIMIT` command's value.
149+
include::esql-kibana.asciidoc[tag=limitations]

0 commit comments

Comments
 (0)