Metadata key contains illegal characters

[kirillgroshkov]

We've started noticing a new error in production (and in staging env also) that happens ~50 times a day for us (for many thousands or maybe even millions of calls per day, don't have exact number). It looks like this in a stacktrace:

unhandledRejection:  { 
Error: Metadata key "a" contains illegal characters     
at Http2CallStream.call.on (/ncbackend3/node_modules/@grpc/grpc-js/build/src/client.js:102:45)     
at Http2CallStream.emit (events.js:194:15)     
at Http2CallStream.EventEmitter.emit (domain.js:459:23)     
at Http2CallStream.endCall (/ncbackend3/node_modules/@grpc/grpc-js/build/src/call-stream.js:74:18)     
at ClientHttp2Stream.stream.on (/ncbackend3/node_modules/@grpc/grpc-js/build/src/call-stream.js:205:30)     
at ClientHttp2Stream.emit (events.js:189:13)     
at ClientHttp2Stream.EventEmitter.emit (domain.js:459:23)     at emit (internal/http2/core.js:236:8)     
at process._tickCallback (internal/process/next_tick.js:63:19)  
 code: 2,   
details:    'Metadata key "a\u0000\u0000\u0000" contains illegal characters',   
metadata: Metadata { internalRepr: Map {} },   
note:    'Exception occurred in retry method that was not classified as transient' }

I've even seen this error myself once while calling a certain endpoint that was doing 1 query and 2-3 getById calls to Datastore. So, no saves was made, only querying.

This error is not reproducible all the time, as I mentioned, rather "rare", but still noticeable (~50 hard errors every day, that end up in http 500 errors for our api users).

I don't know how to dig further and provide other helpful info. Important to note that it only happened after updating from ^3.0.0 (latest version in 3.x.x) to 4.0.0. I'm assuming the error could somewhere downstream (in grpc-js itself) that was updated as a sub-dependency. We're on the latest version of everything as of today (yarn upgrade was done and we have no deps pinned).

ps: forgot to add that the "key" in the metadata has different values in each other error (we see it in Sentry), examples (err.message here):

Metadata key "a" contains illegal characters
Metadata key " �û·" contains illegal characters
Metadata key " �`ñ" contains illegal characters
Metadata key " r�ð" contains illegal characters
Metadata key " òe×" contains illegal characters
Metadata key " �wâ" contains illegal characters
...

pps: we were using .runQuery() method with 2 simple filters by number fields and .get(key) (that's what I called getById). Same code in 3.x.x version worked just fine.

Noticed that a occured more often than others.

Environment details

• OS: node:10-alpine official docker image
• Node.js version: 10.15.3
• npm version: not applicable
• @google-cloud/datastore version: 4.0.0

[kmontag]

Just confirming I'm seeing this frequently as well, but also not reproducibly. Retrying the request in question almost always succeeds. It happens regardless of whether there are simultaneous parallel requests.

Our project contains get, runQuery, and upsert calls, and I've seen it happen with all of them.

[kirillgroshkov]

I think we'll be forced to downgrade to 3.x due to this soon. Too many errors that affect users...

[bcoe]

@alexander-fenster I wonder if this could relate to our upgrade to grpc-js?

[kirillgroshkov]

@alexander-fenster I wonder if this could relate to our upgrade to grpc-js?

It's probably not, cause we did this sequence of things:

1. Downgraded to @google-cloud/datastore@^3.0.0. Error was gone.
2. Upgraded to @grpc/grpc-js@0.4.1 (the one that was unpublished later) and it was working fine for 2 days (no "metadata errors" or new errors).

ps: After I've just written this, I realized I'm not 100% sure. Tried to trace changes via yarn.lock commit history, but the dependency tree between google libs is more complicated to grasp it in few seconds. If it's important - I can try to trace more carefully. But I see multiple of versions of grpc-js in my lock due to different google libs installed.

[alexander-fenster]

@kirillgroshkov @kmontag The main difference between v3.x and v4.x under the hood is the replacement of the original grpc (which uses C++ code compiled into Node.js binary) with the new @grpc/grpc-js. The error you folks see comes from grpc-js so it might be totally possible that the upgrade is causing this. Similar metadata check exists in grpc as well but it has completely different logic.

The good news is that you can keep using the C++ grpc for the time being by passing it to the client constructor:

npm install grpc

const grpc = require('grpc'); // not '@grpc/grpc-js'
const {Datastore} = require('@google-cloud/datastore');
const projectId = 'YOUR_PROJECT_ID';
const datastore = new Datastore({
  projectId,
  grpc   // <-- passing it down to the client
});

I would really appreciate it if you folks try that and update this issue with the results (e.g. if the errors disappear or not). This will help us report a bug to gRPC developers.

Thanks!

[alexander-fenster]

@murgatroid99 @nicolasnoble This is FYI for now but will likely be upgraded to a real @grpc/grpc-js issue :)

[alexander-fenster]

A quick clarification: please upgrade back to v4.0.0 before substituting grpc.

[murgatroid99]

At the moment, the best possible thing to help figure this out would be a tcpdump of a stream that has this error, preferably not all encryped with TLS. If anyone could manage to get one of those and share it here, that would be very helpful.