Support StatisticValues in cloudwatch output plugin (#4318) by david7482 · Pull Request #4364 · influxdata/telegraf

david7482 · 2018-06-30T09:39:06Z

Required for all PRs:

Signed CLA.
Associated README.md updated.
Has appropriate unit tests.

Add enable_statistic_values flag. If true, this plugin would try to parse required statistic fields and send StatisticSet to Cloudwatch.

danielnelson · 2018-07-11T00:18:55Z

plugins/outputs/cloudwatch/cloudwatch.go

+	}
+
+	// Try to parse statisticSet first, then build statistic/value datum accordingly.
+	set, ok := getStatisticSet(point)


What about if the telegraf.Metric contains multiple aggregated field sets like this (leaving out some required aggregations):

cpu foo_min=1,foo_max=2,bar_min=1,bar_max=2

If getStatisticSet() could not get all required fields, it just keeps the original behavior and sends them as independent metrics. It would generate 4 metrics to cloudwatch: cpu_foo_min, cpu_foo_max, cpu_bar_min and cpu_bar_max.

Sorry, my example was somewhat lazy. What if there are two full sets of statistics in a single telegraf.Metric, does it handle this case?

cpu foo_min=1,foo_max=2,foo_sum=4,foo_count=2,bar_min=1,bar_max=2,bar_sum=4,bar_count=2

Good point. Will support this multiple sets case.

danielnelson · 2018-07-11T01:52:49Z

plugins/outputs/cloudwatch/cloudwatch.go

+  ## calculate required statistic fields (count, min, max, and sum) and 
+  ## enable this flag. This plugin would try to parse those fields and 
+  ## send statistic values to Cloudwatch.
+  # enable_statistic_values = false


Let's call this option write_statistics, this is little shorter, but also communicates that it is something that affects the output data. So long as you are in here, can you mention in the comment that if a field does not have the required statistic fields then it will still be sent as a custom/raw metric.

okay, will update the field name and comments

david7482 · 2018-07-18T11:27:40Z

@danielnelson any update for this PR?
We are looking forward to using this on the nightly build.

david7482 · 2018-07-29T16:10:57Z

Already supported multiple statistic sets.

danielnelson · 2018-07-30T19:12:29Z

plugins/outputs/cloudwatch/cloudwatch.go

+	switch {
+	case strings.HasSuffix(name, "_max"):
+		sType = statisticTypeMax
+		fieldName = name[:len(name)-len("_max")]


You can use string.TrimSuffix(name, "_max") here

danielnelson · 2018-07-30T19:21:47Z

plugins/outputs/cloudwatch/cloudwatch.go

+
+func (f *valueField) buildDatum() []*cloudwatch.MetricDatum {
+
+	// Do CloudWatch boundary checking


Don't we still need to do the boundary checking on the statistic datums? Maybe we can do this where we call convert?

All field values would apply to the same boundary checking.

david7482 · 2018-07-31T17:45:56Z

Requested changes are fixed.

glinton · 2018-07-31T19:16:02Z

plugins/outputs/cloudwatch/README.md

+
+### write_statistics
+
+If you have a large amount of metrics, you should consider to send statistic 


Some minor grammar:

s/consider to send/consider sending/

s/ which could/. This should/

s/If enable this flag/If this flag is enabled/

glinton · 2018-07-31T19:16:56Z

plugins/outputs/cloudwatch/cloudwatch.go

+
+	if f.hasAllFields() {
+		// If we have all required fields, we build datum with StatisticValues
+		min, _ := f.values[statisticTypeMin]


You don't need to change this, but just using min := f.values[statisticTypeMin] is sufficient, since you are ignoring the bool value anyways.

glinton · 2018-07-31T19:19:34Z

plugins/outputs/cloudwatch/cloudwatch.go

  ## Namespace for the CloudWatch MetricDatums
  namespace = "InfluxData/Telegraf"
+
+  ## If you have a large amount of metrics, you should consider to send statistic 


Same grammar suggestions as above.

glinton · 2018-07-31T19:23:40Z

plugins/outputs/cloudwatch/cloudwatch.go

+// Make a MetricDatum from telegraf.Metric. It would check if all required fields of
+// cloudwatch.StatisticSet are available. If so, it would build MetricDatum from statistic values.
+// Otherwise, fields would still been built independently.
+func BuildMetricDatum(buildStatistic bool, point telegraf.Metric) []*cloudwatch.MetricDatum {


In general, changing an exported function's signature is advised against. Since nothing outside of this package uses it, maybe add a note that BuildMetricDatum(telegraf.Metric) is deprecated and create a new (unexported) buildMetricDatum(bool, Metric) function.

This isn't something we are concerned about right now, maybe someday but not now. The only API we provide stability for is the TOML file. I keep meaning to document this, but haven't gotten to it.

If this is not an issue, I could help to unexport all exported functions to avoid confusion.
How do you think?

glinton · 2018-07-31T19:26:00Z

plugins/outputs/cloudwatch/cloudwatch.go

+		value = t
+	case bool:
+		if t {
+			value = 1


I see this is how it was before, but should this be float64(1) and float64(0) in order to preserve the type when sending to an output? cc @danielnelson

This looks okay to me, value is a float64 and we are assigning a const to it.

danielnelson added feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin area/aws AWS plugins including cloudwatch, ecs, kinesis labels Jul 7, 2018

danielnelson added this to the 1.8.0 milestone Jul 7, 2018

danielnelson reviewed Jul 11, 2018

View reviewed changes

david7482 added 2 commits July 24, 2018 02:59

Support StatisticValues in cloudwatch output plugin (influxdata#4318)

93ac576

Update comments and flag name

08a11aa

david7482 force-pushed the fix-GH4318 branch from 4b6bb14 to 08a11aa Compare July 23, 2018 19:05

Support multiple statistic sets

c3d212f

danielnelson suggested changes Jul 30, 2018

View reviewed changes

david7482 added 2 commits August 1, 2018 01:11

Use strings.TrimSuffix() for better code readability.

4fcd076

Move boundary check to convert().

9757472

All field values would apply to the same boundary checking.

glinton suggested changes Jul 31, 2018

View reviewed changes

glinton approved these changes Aug 1, 2018

View reviewed changes

danielnelson approved these changes Aug 1, 2018

View reviewed changes

glinton merged commit 199841a into influxdata:master Aug 1, 2018

david7482 deleted the fix-GH4318 branch August 2, 2018 03:28

rgitzel pushed a commit to rgitzel/telegraf that referenced this pull request Oct 17, 2018

Support StatisticValues in cloudwatch output plugin (influxdata#4364)

889b009

otherpirate pushed a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019

Support StatisticValues in cloudwatch output plugin (influxdata#4364)

033cb5f


		func (f valueField) buildDatum() []cloudwatch.MetricDatum {

		// Do CloudWatch boundary checking


		### write_statistics

		If you have a large amount of metrics, you should consider to send statistic

Conversation

david7482 commented Jun 30, 2018

Required for all PRs:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david7482 commented Jul 18, 2018

Uh oh!

david7482 commented Jul 29, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david7482 commented Jul 31, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants