All params should be optional for ngram tokenizer and edge ngram tokenizer

[39charactersisnotenoughforagoodusername]

Java API client version

8.14.3

Java version

17

Elasticsearch Version

8.14.3

Problem description

Hello! I'm running into a MissingRequiredPropertyException when trying to create indices that use ngram/edge ngram tokenizers with the Java client.

Minimal code repro - settings are the same as docs but with token_chars omitted:

String settingsJson =
        "{\"settings\": {\"analysis\": {\"analyzer\": {\"my_analyzer\": {\"tokenizer\": \"my_tokenizer\"}},\"tokenizer\": {\"my_tokenizer\": {\"type\": \"ngram\",\"min_gram\": 3,\"max_gram\": 3}}}}}";
IndexSettings settings = IndexSettings.of(i -> i.withJson(new StringReader(settingsJson)));

throws

Exception in thread "main" co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch._types.analysis.TokenizerDefinition: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'NGramTokenizer.tokenChars' (JSON path: settings.analysis.tokenizer.my_tokenizer) (line no=1, column no=162, offset=161)
	at co.elastic.clients.json.JsonpMappingException.from0(JsonpMappingException.java:134)
	at co.elastic.clients.json.JsonpMappingException.from(JsonpMappingException.java:121)
...
	at co.elastic.clients.elasticsearch.indices.IndexSettings.of(IndexSettings.java:308)
	at Scratch.main(Scratch.java:11)
Caused by: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'NGramTokenizer.tokenChars'
	at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
	at co.elastic.clients.util.ApiTypeHelper.unmodifiableRequired(ApiTypeHelper.java:141)
	at co.elastic.clients.elasticsearch._types.analysis.NGramTokenizer.<init>(NGramTokenizer.java:79)

Similar example throwing for no maxGram if min_gram/max_gram/token_chars are all omitted

String settingsJson =
        "{\"settings\": {\"analysis\": {\"analyzer\": {\"my_analyzer\": {\"tokenizer\": \"my_tokenizer\"}},\"tokenizer\": {\"my_tokenizer\": {\"type\": \"ngram\"}}}}}";
IndexSettings settings = IndexSettings.of(i -> i.withJson(new StringReader(settingsJson)));

Exception in thread "main" co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch._types.analysis.TokenizerDefinition: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'NGramTokenizer.maxGram' (JSON path: settings.analysis.tokenizer.my_tokenizer) (line no=1, column no=134, offset=133)
	at co.elastic.clients.json.JsonpMappingException.from0(JsonpMappingException.java:134)
	at co.elastic.clients.json.JsonpMappingException.from(JsonpMappingException.java:121)
...
	at co.elastic.clients.elasticsearch.indices.IndexSettings.of(IndexSettings.java:308)
	at Scratch.main(Scratch.java:11)
Caused by: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'NGramTokenizer.maxGram'
	at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
	at co.elastic.clients.elasticsearch._types.analysis.NGramTokenizer.<init>(NGramTokenizer.java:77)

This seems to be because the spec that's used to generate the Java client requires min_gram, max_gram, and token_chars for ngram/edge ngram tokenizers, even though they have defaults in docs (also supported by server code and Lucene defaults).

I can also confirm that creating an index via curl without specifying min_gram / max_gram / token_chars works.

kubectl exec es8-data-0 -- curl -XPUT "https://localhost:9200/test-index" -H "Content-Type: application/json" -d '{"settings": {"analysis": {"analyzer": {"my_analyzer": {"tokenizer": "my_tokenizer"}},"tokenizer": {"my_tokenizer": {"type": "ngram"}}}}}'

returns

{"acknowledged":true,"shards_acknowledged":true,"index":"test-index"}

The same is true for "type": "edge_ngram" as well.

[l-trotta]

Hello, thank you for the detailed report! We'll fix the specification soon and regenerate the java client to fix this issue.

[kved96]

Hi @l-trotta, it looks like the patch was only applied for max_gram and min_gram. token_chars is also an optional parameter that has a default value of []. Looking at the code here, there is no optional value for token_chars, even though the docs say there should be. This is causing errors when trying to get indices created that have an ngram tokenizer with an empty token_chars array supplied since the server will not return the token_chars key when getting settings.

[l-trotta]

Hey @kved96, thanks for reporting this. There must be an issue with the client generator, in the specification token_chars now is represented as:

  /**
   * @server_default []
   */
  token_chars?: TokenChar[]

which should be enough to set it as optional in the client. Checking the client generator ASAP.

[l-trotta]

Sorry for the confusion, this is correct, it's how the java client handles nullable collections: it's not marked as @Nullable because if it gets deserialized as null, then it will be set as an empty list. This is the ApiTypeHelper.unmodifiable method:

    /**
     * Returns an unmodifiable view of a list. If {@code list} is {@code null}, an {@link #undefinedList()} is returned.
     */
    public static <T> List<T> unmodifiable(@Nullable List<T> list) {
        if (list == null) {
            return undefinedList();
        }
        if (list == UNDEFINED_LIST) {
            return list;
        }
        return Collections.unmodifiableList(list);
    }

so if token_chars is set either as null or as an empty list, it will result in an empty list, which the server should accept.

To test this, I created an index with an NgramTokenizer with empty token_chars using the Kibana devtools like this:

PUT ngram-test
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "my_tokenizer"
        }
      },
      "tokenizer": {
        "my_tokenizer": {
          "type": "ngram",
          "min_gram": 3,
          "max_gram": 3
        }
      }
    }
  }
}

which I then retrieved using the java client:

esClient.indices().getSettings(s -> s.index("ngram-test"));

The client can correctly deserialize it:

If you're getting errors then maybe there could be some other issue happening? Let me know.

[kved96]

Hi @l-trotta, I bumped the client version to 8.16.1 and I'm specifically running into an issue after creating an index with an n_gram tokenizer with the token_chars field when using the below method to get all the indices in the cluster. The specific settings I'm using at create time are:

{
"index.analysis.analyzer.test_ngram_analyzer.tokenizer": "ngram_tokenizer",
"index.analysis.tokenizer.ngram_tokenizer.type": "ngram",
}

Then using the get indices endpoint like below:

esClient.indices()
                .get(r -> r.index("_all")
                        .allowNoIndices(true)
                        .expandWildcards(ExpandWildcard.Open, ExpandWildcard.Closed))
                .result()

This throws with:

status: 200, [es/indices.get] Failed to decode response","throwable2_message":"Error deserializing co.elastic.clients.elasticsearch._types.analysis.TokenizerDefinition: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'NGramTokenizer.tokenChars' 

I would expect this to be patched by this fix, but it doesn't look like it is. The ES server version I'm using is 8.14.3

[kved96]

Filed #918 for tracking