Refactor parsing of queries/filters, aggs, suggester APIs

[clintongormley]

Copied from #9901:

Today we have a massive infrastructure to parse all our requests. We have client side builders and server side parsers but no real representation of the query, filter, aggregation etc until it's executed. What is produced from a XContent binary is a Lucene query directly which causes huge parse methods in separate classes etc. that hare hard to test and don't allow decoupled modifications or actions on the query itself between parsing and executing.

This refactoring splits the parsing and the creation of the lucene query, this has a couple of advantages

• XContent parsing creation are in one file and can be tested more easily
• the class allows a typed in-memory representation of the query that can be modified before a lucene query is build
• the query can be normalized and serialized via Streamable to be used as a normalized cache key (not depending on the order of the keys in the XContent)
• the query can be parsed on the coordinating node to allow document prefetching etc. forwarding to the executing nodes would work via Streamable binary representation --> Should we parse search requests on the coordinating node? #8150
• for the query cache a query tree can be "walked" to rewrite range queries into match all queries with MIN/MAX terms to get cache hits for sliding windows --> Kibana 4 unable to utilize query cache #9526
• code wise two classes are merged into one which is nice

Queries (Completed)

• AndQueryBuilder (Query refactoring: AndQueryBuilder and Parser #11628)
• BoolQueryBuilder (Query refactoring: BoolQueryBuilder and Parser #11121)
• BoostingQueryBuilder (Query refactoring: BoostingQueryBuilder and Parser #11621)
• CommonTermsQueryBuilder (Refactors CommonTermsQuery #11345)
• ConstantScoreQueryBuilder (Query refactoring: ConstantScoreQueryBuilder and Parser #11629)
• DisMaxQueryBuilder (Query Refactoring: DisMaxQueryBuilder and Parser #11703)
• ExistsQueryBuilder (Query Refactoring: ExistsQueryBuilder and Parser #11427)
• FieldMaskingSpanQueryBuilder (Query refactoring: FieldMaskingSpanQueryBuilder and Parser #11717)
• FilteredQueryBuilder (Refactors FilteredQueryBuilder #11885)
• FQueryFilterBuilder (Query refactoring: QueryFilterBuilder and Parser #11729)
• FunctionScoreQueryBuilder (Query refactoring: function_score #13653)
• FuzzyQueryBuilder (Refactoring of FuzzyQuery #11865)
• GeoBoundingBoxQueryBuilder (Refactoring of GeoBoundingBoxQueryBuilder and -Parser #11969)
• GeoDistanceQueryBuilder(Refactors GeoDistanceQueryBuilder/-Parser #13496)
• GeoDistanceRangeQueryBuilder (Refactors GeoDistanceQueryBuilder/-Parser #13496)
• GeohashCellQuery (Refactor of GeohashCellQuery #13393)
• GeoPolygonQueryBuilder (Refactor of GeoPolygonQuery #13426)
• GeoShapeQueryBuilder (Refactor of GeoShapeQuery #13466)
• HasChildQueryBuilder (Split HasChildQueryParser into toQuery and formXContent #13333)
• HasParentQueryBuilder (Split HasParentQueryParser into toQuery and formXContent #13408)
• IdsQueryBuilder (Query refactoring: IdsQuery #10670)
• IndicesQueryBuilder (Refactoring of Indices Query #12031)
• LimitQueryBuilder (Query refactoring: refactored LimitQueryBuilder and Parser and added test #11551)
• MatchAllQueryBuilder (Query refactoring: MatchAllQueryBuilder #10668)
• MatchQueryBuilder (Query refactoring: MatchQueryBuilder #13402)
• MissingQueryBuilder (Refactoring of MissingQuery #12030)
• MoreLikeThisQueryBuilder (Refactors MoreLikeThisQueryBuilder and Parser #13486)
• MultiMatchQueryBuilder (Refactors MultiMatchQueryBuilder and Parser #13405)
• NestedQueryBuilder (Split NestedQueryParser into toQuery and formXContent #13424)
• NotQueryBuilder (Query refactoring: NotQueryBuilder and Parser #11823)
• OrQueryBuilder (Query refactoring: OrQueryBuilder and Parser #11817)
• PrefixQueryBuilder (Refactors PrefixQuery #12032)
• QueryFilterBuilder (Query refactoring: QueryFilterBuilder and Parser #11729)
• QueryStringQueryBuilder (Query refactoring: query_string query #13284)
• QueryWrappingQueryBuilder (will be removed once all queries refactored)
• RangeQueryBuilder (Query refactoring: RangeQueryBuilder and Parser #11108)
• RegexpQueryBuilder (Refactoring of RegexpQuery #11896)
• ScriptQueryBuilder (Refactors ScriptQueryBuilder and Parser #12115)
• SimpleQueryStringBuilder (Refactors SimpleQueryStringBuilder/Parser #11274)
• SpanContainingQueryBuilder (Query refactoring: SpanContainingQueryBuilder and Parser #12057)
• SpanFirstQueryBuilder (Query refactoring: SpanFirstQueryBuilder and Parser #12073)
• SpanMultiTermQueryBuilder (Query refactoring: SpanMultiTermQueryBuilder and Parser #12182)
• SpanNearQueryBuilder (Query refactoring: SpanNearQueryBuilder and Parser #12156)
• SpanNotQueryBuilder (Query refactoring: SpanNotQueryBuilder and Parser #12365)
• SpanOrQueryBuilder (Query refactoring: SpanOrQueryBuilder and Parser #12342)
• SpanTermQueryBuilder (Refactors SpanTermQueryBuilder. #11005)
• SpanWithinQueryBuilder (Query refactoring: SpanWithinQueryBuilder and Parser #12396)
• TemplateQueryBuilder (Refactors TemplateQueryBuilder and Parser #13253)
• TermQueryBuilder (Query refactoring: TermQueryBuilder refactoring and test #10669)
• TermsLookupQueryBuilder (Refactors TermsQueryBuilder and Parser #12042)
• TermsQueryBuilder (Refactors TermsQueryBuilder and Parser #12042)
• TypeQueryBuilder (Refactors TypeQuery #12035)
• WildcardQueryBuilder (Refactors WildcardQueryBuilder and Parser #12110)
• WrapperQueryBuilder (Refactors WrapperQueryBuilder and Parser #12037)

Total of 54 Queries
54 done

Former filters were mostly merged or converted to queries and are included in this list.

Aggregations (Completed - #14136)

• Min Aggregation (Aggregations Refactor: Refactor Min Aggregation #12832)
• Max Aggregation (Aggregations Refactor: Refactor Max, Avg, and Sum Aggregations #14687)
• Sum Aggregation (Aggregations Refactor: Refactor Max, Avg, and Sum Aggregations #14687)
• Avg Aggregation (Aggregations Refactor: Refactor Max, Avg, and Sum Aggregations #14687)
• Stats Aggregation (Aggregations Refactor: Refactor Stats and Extended_Stats Aggregations #14688)
• Extended Stats Aggregation (Aggregations Refactor: Refactor Stats and Extended_Stats Aggregations #14688)
• Value Count Aggregation (Aggregations Refactor: Refactor Value Count Aggregation #14689)
• Percentiles Aggregation (Aggregations Refactor: Refactor Percentiles and Percentile Ranks Aggregation #14836)
• Percentile Ranks Aggregation (Aggregations Refactor: Refactor Percentiles and Percentile Ranks Aggregation #14836)
• Cardinality Aggregation (Aggregations Refactor: Refactor Cardinality Aggregation #14893)
• Geo Bounds Aggregation (Aggregations Refactor: Refactor Geobounds Aggregation #14934)
• Top hits Aggregation (Aggregations Refactor: Refactor Top Hits Aggregation #15566)
• Scripted Metric Aggregation (Aggregations Refactor: Refactor Scripted Metric Aggregation #14966)
• Global Aggregation (Refactor Global Aggregation #14139)
• Filter Aggregation (Aggregations Refactor: Refactor Filter Aggregation #14974)
• Filters Aggregation (Aggregations Refactor: Refactor Filters Aggregation #15437)
• Missing Aggregation (Aggregations Refactor: Refactor Missing Aggregation #14975)
• Nested Aggregation (Aggregations Refactor: Refactor Nested and Reverse Nested Aggregations #15006)
• Reverse nested Aggregation (Aggregations Refactor: Refactor Nested and Reverse Nested Aggregations #15006)
• Children Aggregation (Aggregations Refactor: Refactor Children Aggregation #15008)
• Terms Aggregation (Aggregations Refactor: Refactor Terms Aggregation #15386)
• Significant Terms Aggregation (Aggregations Refactor: Refactor Terms Aggregation #15386)
• Sampler Aggregation (Aggregations Refactor: Refactor Sampler Aggregation #15418)
• Range Aggregation (Aggregations Refactor: Refactor Range Aggregations #15399)
• Date Range Aggregation (Aggregations Refactor: Refactor Range Aggregations #15399)
• IPv4 Range Aggregation (Aggregations Refactor: Refactor Range Aggregations #15399)
• Histogram Aggregation (Refactor Histogram and Date Histogram Aggregation #14140)
• Date Histogram Aggregation (Refactor Histogram and Date Histogram Aggregation #14140)
• Geo Distance Aggregation (Aggregations Refactor: Refactor Range Aggregations #15399)
• GeoHash grid Aggregation (Refactor Geohash Grid Aggregation #14138)
• Geo Centroid Aggregation (Aggregations Refactor: Refactor Geo Centroid Aggregation #15002)
• Min Bucket Aggregation (Aggregations Refactor: Refactor Avg Bucket, Min Bucket, Max Bucket, Sum Bucket, Percentiles Bucket, Stats Bucket and Extended Stats Bucket Aggregations #15009)
• Max Bucket Aggregation (Aggregations Refactor: Refactor Avg Bucket, Min Bucket, Max Bucket, Sum Bucket, Percentiles Bucket, Stats Bucket and Extended Stats Bucket Aggregations #15009)
• Avg Bucket Aggregation (Aggregations Refactor: Refactor Avg Bucket, Min Bucket, Max Bucket, Sum Bucket, Percentiles Bucket, Stats Bucket and Extended Stats Bucket Aggregations #15009)
• Sum Bucket Aggregation (Aggregations Refactor: Refactor Avg Bucket, Min Bucket, Max Bucket, Sum Bucket, Percentiles Bucket, Stats Bucket and Extended Stats Bucket Aggregations #15009)
• Stats Bucket Aggregation (Aggregations Refactor: Refactor Avg Bucket, Min Bucket, Max Bucket, Sum Bucket, Percentiles Bucket, Stats Bucket and Extended Stats Bucket Aggregations #15009)
• Extended Stats Bucket Aggregation (Aggregations Refactor: Refactor Avg Bucket, Min Bucket, Max Bucket, Sum Bucket, Percentiles Bucket, Stats Bucket and Extended Stats Bucket Aggregations #15009)
• Percentiles Bucket Aggregation (Aggregations Refactor: Refactor Avg Bucket, Min Bucket, Max Bucket, Sum Bucket, Percentiles Bucket, Stats Bucket and Extended Stats Bucket Aggregations #15009)
• Bucket Script Aggregation (Aggregations Refactor: Refactor Bucket Script Aggregation #15014)
• Cumulative Sum Aggregation (Aggregations Refactor: Refactor Cumulative Sum Aggregation #15015)
• Derivative Aggregation (Refactor Derivative Aggregation #14137)
• Bucket Selector Aggregation (Aggregations Refactor: Refactor Bucket Selector Aggregation #15147)
• Moving Average Aggregation (Aggregations Refactor: Refactor Moving Average Aggregation #15180)
• Serial Differencing Aggregation (Aggregations Refactor: Refactor Serial Differencing Aggregation #15058)

Total of 44 Aggregations
44 done

Suggesters

• Term suggester
• Phrase Suggester
• Completion Suggester
• Context Suggester

Total of 4 Suggesters
4 done, 0 in open PRs

Highlighters

• plain
• fvh
• postings

Total of 3 Highlighters
3 done

Others

• from (Refactor SearchRequest to be parsed on the coordinating node #13752)
• size (Refactor SearchRequest to be parsed on the coordinating node #13752)
• min_score (Refactor SearchRequest to be parsed on the coordinating node #13752)
• fields (Refactor SearchRequest to be parsed on the coordinating node #13752)
• script_fields (Refactor SearchRequest to be parsed on the coordinating node #13752)
• fielddata_fields (Refactor SearchRequest to be parsed on the coordinating node #13752)
• _source (Refactor SearchRequest to be parsed on the coordinating node #13752)
• sort (Switch to using refactored SortBuilder instead of using BytesReference in serialization #17205)
• timeout (Refactor SearchRequest to be parsed on the coordinating node #13752)
• explain (Refactor SearchRequest to be parsed on the coordinating node #13752)
• terminate_after (Refactor SearchRequest to be parsed on the coordinating node #13752)
• version (Refactor SearchRequest to be parsed on the coordinating node #13752)
• stats (Refactor SearchRequest to be parsed on the coordinating node #13752)
• indices_boost (Refactor SearchRequest to be parsed on the coordinating node #13752)
• track_scores (Refactor SearchRequest to be parsed on the coordinating node #13752)
• inner_hits (Refactored inner hits parsing and intoduced InnerHitBuilder #17291)
• query_binary: (removed, queries should only be specified via type-safe builders in the Java API, see Remove query_binary, filter_binary & aggs_binary  #14308)
• filter_binary: (removed, filters should only be specified via type-safe builders in the Java API, see Remove query_binary, filter_binary & aggs_binary  #14308)
• aggregations_binary: (to be removed, aggregations should only be specified via type-safe builders in the Java API, see Remove query_binary, filter_binary & aggs_binary  #14308 and Refactoring of Aggregations #14136)
• rescore (Make RescoreBuilder implement Writable & fromXContent #15559)
• post_filter (Refactor SearchRequest to be parsed on the coordinating node #13752)

APIs to be adapted/revised besides _search

• search exists api (removed, see Remove search exists api #13911)
• explain api: (Explain api: move query parsing to the coordinating node #14270)
• validate query api: (Validate query api: move query parsing to the coordinating node #14384)
• suggest api (Remove suggest transport action #17198)
• percolator (Planned to be removed, see Replace percolate APIs with a percolator query #16349)
• index warmers (removed, see Remove query warmers and the warmer API. #15614)
• alias filters

The above apis don't necessarily have to change to parse queries in our intermediate format, for instance the percolator will still need to parse to lucene query straight-away, but we should still have a look at each of those and double check if anything needs to be adjusted after all the infra changes we have made.

[mattweber]

+1, In #3278 I perform a terms lookup after parsing but this happened on all shards and resulted in multiple lookup requests for a single query. This would allow the expensive lookup to be performed once on the coordinating node which would be very beneficial!

[javanna]

Here is our rough plan:

First step (everything still happens on the nodes that hold the relevant shards):

• split existing QueryParser#parse() method into 1) fromXContent() that parses the query but allows to have an intermediate format for it and 2) toQuery() that allows to create the lucene query ouf the intermediate query. Keep the exisiting Query parse() method around temporarily, which will call both fromXContent and toQuery.
• Move toQuery to QueryBuilders so that every query (intermediate format) can be transformed into the corresponding lucene query on each data node.
• make all the QueryBuilders implement Streamable so that the intermediate query format can be sent over the wire between the nodes (will be unused at first)

Second step (query parsing moved to the coordinating node):

• Refactor the whole SearchRequest so that instead of holding the source bytes array in json format, it holds different elements of a search request, all in Streamable format
• leverage existing request validation mechanism so that SearchRequest#validate validates the query too (during TransportSearchAction#execute): this is very convenient as it gets called on the coordinating node, no matter where the request comes from (could come as json through rest layer, or as java objects through java api, either transport client or node client).
• call fromXContent on the coordinating node and make use of Streamable methods to send queries over the wire
• delete parse method from all QueryBuilders, not needed anymore.

Things we are not too happy about and might need improvement, will be tackled later on:

• rename QueryBuilders by removing the Builders suffix, as they will not be really just builders anymore
• java api side of things is heavier on users, since a single class exposes a lot of internal aspects, we might be able to make methods package private

[dakrone]

@clintongormley et al,

Because we are touching every single query in this change, it also gives us the ability to remove support for camel-casing in queries where it exists. How do we feel about removing the camel casing in these PRs as well?

[clintongormley]

@dakrone ++ - should we be doing this by using parseField and then later using #8963 to warn about the deprecations?