Fix bug: too many cloudwatch metrics

[leon-barrett]

Cloudwatch metrics were being added incorrectly. The most obvious
symptom of this was that too many metrics were being added. A simple
check against the name of the metric proved to be a sufficient fix. In
order to test the fix, a metric selection function was factored out.

[leon-barrett]

Hi, maintainers. Do you have any feedback about this pull request? We have a workaround for ourselves, but it would be nice to get this fix merged.

[sparrc]

from what I can tell there appears to be overlap between #1885 and #1903. I would appreciate if one of you could open an issue with a full writeup on what exactly is wrong the cloudwatch plugin.

As this plugin is particularly difficult to reproduce problems on, please be sure to include examples of requests and responses, and links to documentation where necessary.

Please also provide concrete examples that illustrate what is wrong with the current behavior of the plugin.

If I am wrong and there isn't overlap, then feel free to open two separate issues. In either case I think they need to be seen and discussed before I can accept a PR changing the behavior of this plugin.

[johnrengelman]

@sparrc I looked at this change and it looks correct. I verified using the provided spec. What is happening currently is that when using wildcard dimensions, the metric is selected multiple times because it has an inner loop that ranges over all available metrics from the Cloudwatch API but it doesn't compare that it's looking at the same metric that's configured.

So if I have this:

selectedMetrics = ["Connections"]

and Cloudwatch is returning:

availableMetrics = ["Connections", "Disk", "CPU"]

then the code is currently doing:

for _, sm := range selectedMetrics {
  for _, am := range availableMetrics {
    //if the available metric has the same possible dimensions as the metric being looked at, 
    //then it's appended.
    if (isSelected(am, dimension)) { 
      append(sm)
    }
  }
}

So the change here is correct in that it first checks that we are comparing our selected metric against only those metrics in cloudwatch of the same name.

[johnrengelman]

The name logic check should just be folded into the isSelected method.
So, isSelected(name, metric, m.Dimensions)

[leon-barrett]

Done

[sparrc]

OK, fair enough, thanks for the review @johnrengelman

[leon-barrett]

Thanks for taking a close look at my fix. I'll try to be more complete in explanations in future pull requests.

[sparrc]

Looks good, just update the changelog and I'll get it merged

[sparrc]

move this into "features" under the 1.2 section

[leon-barrett]

done