Telegraf Syslog plugin.

[bogski87]

Relevant telegraf.conf:

[[inputs.syslog]]
#   ## Specify an ip or hostname with port - eg., tcp://localhost:6514, tcp://10.0.0.1:6514
#   ## Protocol, address and port to host the syslog receiver.
#   ## If no host is specified, then localhost is used.
#   ## If no port is specified, 6514 is used (RFC5425#section-4.1).
   server = "tcp://localhost:6514"
#
#   ## TLS Config
#   # tls_allowed_cacerts = ["/etc/telegraf/ca.pem"]
#   # tls_cert = "/etc/telegraf/cert.pem"
#   # tls_key = "/etc/telegraf/key.pem"
#
#   ## Period between keep alive probes.
#   ## 0 disables keep alive probes.
#   ## Defaults to the OS configuration.
#   ## Only applies to stream sockets (e.g. TCP).
#    keep_alive_period = "5m"
#
#   ## Maximum number of concurrent connections (default = 0).
#   ## 0 means unlimited.
#   ## Only applies to stream sockets (e.g. TCP).
#   # max_connections = 1024
#
#   ## Read timeout (default = 500ms).
#   ## 0 means unlimited.
#   # read_timeout = 500ms
#
#   ## Whether to parse in best effort mode or not (default = false).
#   ## By default best effort parsing is off.
#   # best_effort = false
#
#   ## Character to prepend to SD-PARAMs (default = "_").
#   ## A syslog message can contain multiple parameters and multiple identifiers within structured data section.
#   ## Eg., [id1 name1="val1" name2="val2"][id2 name1="val1" nameA="valA"]
#   ## For each combination a field is created.
#   ## Its name is created concatenating identifier, sdparam_separator, and parameter name.
#   # sdparam_separator = "_"

System info:

Ubuntu 16.04.3 x64 Telegraf 1.7

Steps to reproduce:

1. Install the latest Telegraf build

2. Followed this topic and read me

3. Restart Telegraf after editing the config and wait for the data, tail the telegraf logs

Expected behavior:

Collect mertics from Syslog and send them to InfluxDB

Actual behavior:

Telegraf does send the data to InfluxDB but this error appears in the telegaf log:

2018-06-22T11:19:49Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN 2018-06-22T11:19:59Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN 2018-06-22T11:20:09Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN 2018-06-22T11:20:19Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN 2018-06-22T11:20:29Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN 2018-06-22T11:20:39Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN 2018-06-22T11:20:49Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN 2018-06-22T11:20:59Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN 2018-06-22T11:21:02Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN 2018-06-22T11:21:09Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN

Additional info:

I think my config is valid, Telegraf starts with no issues and sends the metrics to InfluxDB. I've also tested this with a separate Telegraf instance sending data to a remote InfluxDB.

If i can provide anything else let me know.

Regards,

PhilB

[russorat]

@bogski87 thanks for opening this issue. could you add some sample log lines and the relevant syslog (or rsyslog) conf files as well?

[bogski87]

Hi @russorat , sure thing.

I don't have anywhere to host the configs so it is a cut and paste job unfortunately.

contents of /etc/rsyslog.conf

#  /etc/rsyslog.conf	Configuration file for rsyslog.
#
#			For more information see
#			/usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
#  Default logging rules can be found in /etc/rsyslog.d/50-default.conf


#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
module(load="imklog")   # provides kernel logging support
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

# Enable non-kernel facility klog messages
$KLogPermitNonKernelFacility on

###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

Contents of /etc/rsyslog.d/50-default.conf

#  Default rules for rsyslog.
#
#			For more information see rsyslog.conf(5) and /etc/rsyslog.conf

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*			/var/log/auth.log
*.*;auth,authpriv.none		-/var/log/syslog
#cron.*				/var/log/cron.log
#daemon.*			-/var/log/daemon.log
kern.*				-/var/log/kern.log
#lpr.*				-/var/log/lpr.log
mail.*				-/var/log/mail.log
#user.*				-/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info			-/var/log/mail.info
#mail.warn			-/var/log/mail.warn
mail.err			/var/log/mail.err

#
# Logging for INN news system.
#
news.crit			/var/log/news/news.crit
news.err			/var/log/news/news.err
news.notice			-/var/log/news/news.notice

#
# Some "catch-all" log files.
#
#*.=debug;\
#	auth,authpriv.none;\
#	news.none;mail.none	-/var/log/debug
#*.=info;*.=notice;*.=warn;\
#	auth,authpriv.none;\
#	cron,daemon.none;\
#	mail,news.none		-/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg                                :omusrmsg:*

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#	news.=crit;news.=err;news.=notice;\
#	*.=debug;*.=info;\
#	*.=notice;*.=warn	/dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
# you must invoke `xconsole' with the `-file' option:
# 
#    $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
#      busy site..
#
daemon.*;mail.*;\
	news.err;\
	*.=debug;*.=info;\
	*.=notice;*.=warn	|/dev/xconsole



$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down

# forward over tcp with octet framing according to RFC 5425
*.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

Output from tail -f /var/log/syslog

Jun 26 15:58:42 tick-box influxd[1075]: [httpd] ::1 - philbdb [26/Jun/2018:15:58:42 +0100] "POST /query?db=&q=SHOW+SUBSCRIPTIONS HTTP/1.1" 200 243 "-" "KapacitorInfluxDBClient" 6aaab24c-7951-11e8-808f-000000000000 1172
Jun 26 15:58:50 tick-box influxd[1075]: [httpd] 192.168.1.43 - philbdb [26/Jun/2018:15:58:50 +0100] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "telegraf" 6f6549a7-7951-11e8-8090-000000000000 4959
Jun 26 15:59:00 tick-box influxd[1075]: [httpd] 192.168.1.43 - philbdb [26/Jun/2018:15:59:00 +0100] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "telegraf" 755b2131-7951-11e8-8091-000000000000 10235
Jun 26 15:59:10 tick-box influxd[1075]: [httpd] 192.168.1.43 - philbdb [26/Jun/2018:15:59:10 +0100] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "telegraf" 7b510fe7-7951-11e8-8092-000000000000 7535
Jun 26 15:59:20 tick-box influxd[1075]: [httpd] 192.168.1.43 - philbdb [26/Jun/2018:15:59:20 +0100] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "telegraf" 8146f80a-7951-11e8-8093-000000000000 12046
Jun 26 15:59:30 tick-box influxd[1075]: [httpd] 192.168.1.43 - philbdb [26/Jun/2018:15:59:30 +0100] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "telegraf" 873cbede-7951-11e8-8094-000000000000 6961

Output from /var/log/telegraf/telegraf.log

2018-06-26T14:59:00Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN
2018-06-26T14:59:10Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN
2018-06-26T14:59:20Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN
2018-06-26T14:59:30Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN
2018-06-26T14:59:38Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN
2018-06-26T14:59:40Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN
2018-06-26T14:59:50Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN
2018-06-26T15:00:00Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN
2018-06-26T15:00:10Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN
2018-06-26T15:00:20Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN
2018-06-26T15:00:30Z E! Error in plugin [inputs.syslog]: found EOF, expecting a MSGLEN

No other messages are in the telegraf log other than the initial start up messages.

Data is being written, so i know the plugin works in that respect. Although the server time is out by an hour.

If there's anything else i can provide please let me know.

Apologies for the lack of availability to upload the conf files and link to them.

Phil

[glinton]

I've finally been able to reproduce this. I noticed no network traffic occurring at the time the log gets generated and after closer inspection of your telegraf.logs, it seems to be telegraf parsing related (the timestamp in your logs are 10s apart)

[mthota15]

is this issue fixed? I am seeing same errors in latest version of telegraf(1.7.2). I am trying to send my docker daemon logs via syslog drivers.

018/07/27 16:09:10 I! Using config file: /etc/telegraf/telegraf.conf
2018-07-27T16:09:10Z I! Starting Telegraf v1.7.2
2018-07-27T16:09:10Z I! Loaded inputs: inputs.syslog
2018-07-27T16:09:10Z I! Loaded aggregators:
2018-07-27T16:09:10Z I! Loaded processors:
2018-07-27T16:09:10Z I! Loaded outputs: file
2018-07-27T16:09:10Z I! Tags enabled:
2018-07-27T16:09:10Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"", Flush Interval:10s
2018-07-27T16:09:11Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
2018-07-27T16:09:16Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN
2018-07-27T16:09:21Z E! Error in plugin [inputs.syslog]: found ILLEGAL(<), expecting a MSGLEN

Here is my docker daemon json config:

{
"log-opts" : {
"syslog-address" : "tcp://localhost:6514"
},
"debug" : true,
"experimental" : false,
"log-driver" : "syslog"
}

[turbo]

This issue is not fixed. I experience the exact same error as @mthota15

[glinton]

@turbo, what version of telegraf are you using, what does your config look like, and have you tried the suggestions outlined in #4482? (specifically having telegraf syslog address use udp as it doesn't require octet framing)

[glinton]

@turbo if you don't mind continuing this in 4482, that better describes your issue, thanks.