Multiple SIGHUPs in short succession causes a panic

[tommoa]

Relevant telegraf.conf:

Default configuration from ./telegraf config --output-filter file as well as

# Telegraf Configuration
[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = false

[[outputs.file]]
   files = ["stdout"]
   data_format = "influx"
[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.diskio]]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]

System info:

Linux kernel: 5.0.11-arch1-1-ARCH
Telegraf version: Telegraf unknown (git: master e6dd8536)

Steps to reproduce:

1. Start telegraf
2. Run while [ true ]; do kill -s SIGHUP $(pidof telegraf); done

Expected behavior:

Configuration reloads on a loop.

Actual behavior:

E! [telegraf] Error running agent: context canceled

Additional info:

This bug actually doesn't occur if the config being used is small enough; I can reproduce this using the default generated config however if I remove all of the commented lines from that config (as per the configuration above), then the error never occurs.

[sgtsquiggs]

Not too sure about the unintended side effects, but this can be mitigated in cmd/telegraf/telegraf.go:

diff --git a/cmd/telegraf/telegraf.go b/cmd/telegraf/telegraf.go
index a3fae740..7fead672 100644
--- a/cmd/telegraf/telegraf.go
+++ b/cmd/telegraf/telegraf.go
@@ -100,7 +100,9 @@ func reloadLoop(
                }()
 
                err := runAgent(ctx, inputFilters, outputFilters)
-               if err != nil {
+               if err == context.Canceled {
+                       log.Printf("E! [telegraf] Error running agent: %v", err)
+               } else if err != nil {
                        log.Fatalf("E! [telegraf] Error running agent: %v", err)
                }
        }

Since the context is only cancelled via SIGHUP or stop, I believe we can safely ignore contex.Canceled error from runAgent.

[danielnelson]

Yeah that should basically work, though I think we just want to ignore context.Canceled. 3d7a718