Error writing out of order points to Stackdriver output

[Legogris]

Relevant telegraf.conf:

[agent]
    interval = "30s"
    round_interval = true
    flush_interval = "60s"
    flush_jitter = "0s"
    debug = false
    hostname = ""

System info:

Telegraf 1.9.0 (git: HEAD 2bf21c0)
OS: Amazon Linux (4.14.77-81.59.amzn2.x86_64 #1 SMP Mon Nov 12 21:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux)

Stackdriver requires points to be written in order - it seems that this is not guaranteed by the stackdriver output plugin:

2019-01-29T00:30:00Z E! [output.stackdriver] unable to write to Stackdriver: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T00:30:00Z E! [agent] Error writing to output [stackdriver]: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T00:31:00Z E! [output.stackdriver] unable to write to Stackdriver: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T00:31:00Z E! [agent] Error writing to output [stackdriver]: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T00:32:00Z E! [output.stackdriver] unable to write to Stackdriver: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T00:32:00Z E! [agent] Error writing to output [stackdriver]: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T00:33:00Z E! [output.stackdriver] unable to write to Stackdriver: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T00:33:00Z E! [agent] Error writing to output [stackdriver]: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T00:34:00Z E! [output.stackdriver] unable to write to Stackdriver: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T00:34:00Z E! [agent] Error writing to output [stackdriver]: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0

For hosts with higher number of metrics, we also see this quite a bit:

2019-01-29T08:57:00Z W! [agent] output "stackdriver" did not complete within its flush interval
2019-01-29T08:59:00Z W! [agent] output "stackdriver" did not complete within its flush interval
2019-01-29T09:10:00Z W! [agent] output "stackdriver" did not complete within its flush interval
2019-01-29T09:12:00Z W! [agent] output "stackdriver" did not complete within its flush interval
2019-01-29T09:15:00Z W! [agent] output "stackdriver" did not complete within its flush interval
2019-01-29T09:17:00Z W! [agent] output "stackdriver" did not complete within its flush interval
2019-01-29T09:25:59Z E! [output.stackdriver] unable to write to Stackdriver: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T09:25:59Z E! [agent] Error writing to output [stackdriver]: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Points must be written in order. One or more of the points specified had an older end time than the most recent point.: timeSeries[0]
2019-01-29T09:44:00Z W! [agent] output "stackdriver" did not complete within its flush interval

Don't know if this is related to #2099 or not.

[danielnelson]

This is quite problematic, as Telegraf normally sends the newest points first and works backwards. This will be even more pronounced in the latest version as we are now even sending the batch with the newest metrics first. Even if we were to send to the stackdriver output in the reverse order, at any time an old point could be received.

In the meantime we can send new batches in order and treat this as a warning, dropping any old points.

[danielnelson]

A more complete solution would be a metric buffer variation that operates in time ascending order, this could be useful for stackdriver, file, and the queue outputs.

[danielnelson]

I'm going to close this issue with the sort fix in #5385 with additional improvements in #5407 , and we will consider adding the metric buffer variation described above as a future change, which would primarily be a performance improvement and implementation simplification.