Skip to content

Make output format of logger compatible with fluentd #163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 1, 2018
Merged

Make output format of logger compatible with fluentd #163

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f60a160
Make output format of logger compatible with fluentd
smacker Feb 23, 2018
02d5823
Use original logrus JSONFormatter
smacker Feb 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 60 additions & 4 deletions server/service/logger.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,78 @@
package service

import (
"encoding/json"
"fmt"
"time"

"github.com/sirupsen/logrus"
)

// NewLogger returns a logrus Logger
func NewLogger(env string) *logrus.Logger {
logger := logrus.New()
logger.Formatter = &logrus.TextFormatter{
FullTimestamp: true,
TimestampFormat: "2006-01-02 15:04:05",
}

if env == "dev" {
logger.Formatter = &logrus.TextFormatter{
FullTimestamp: true,
TimestampFormat: "2006-01-02 15:04:05",
}
logger.SetLevel(logrus.DebugLevel)
} else {
logger.Formatter = &FluentdFormatter{}
logger.SetLevel(logrus.WarnLevel)
}

return logger
}

// FluentdFormatter is similar to logrus.JSONFormatter but with log level that are recognized
// by kubernetes fluentd.
type FluentdFormatter struct {
Copy link
Contributor

@dpordomingo dpordomingo Feb 27, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused...
I think we could avoid copy-pasting this bunch of code if we could use FluentdFormatter from joonix/log, as can be found at https://github.com/joonix/log/blob/master/fluentd.go

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't have any policy about it.

@bzz what the party says?

This particular code can be found in 2 repositories:
https://github.com/joonix/log/blob/master/fluentd.go
https://github.com/elafarge/gin-http-logger/blob/master/logrus-formatters/fluentd-formatter.go
And I saw more which are absolutely the same but have different naming. For example:
https://github.com/everflow-io/logrus-gcloud-format/blob/master/formatter.go

and it's copy past of original logrus formatter + 9 lines of simple mapping code (can be written in 3):
https://github.com/sirupsen/logrus/blob/master/json_formatter.go

P.S. When I write Go I try to avoid 1 liners dependency hell as it is in npm, but it's up to project/company.

Copy link
Contributor Author

@smacker smacker Feb 27, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops. My bad. Not 9 additional line. Just changes of 3 lines.

I got confused at first. Actually now it looks like it's possible to archive the same effect using only logrus. But I still need clarification about dependencies for the future @bzz

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

idk... joonix/FluentdFormatter is what sirupsen/logrus suggests.
(or the fluentd hook that idk if it would suit our necessities)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need fluentd hook if you don't use kubernetes.

Those only 3 lines formatter changes:

	data["time"] = entry.Time.Format(timestampFormat)
	data["message"] = entry.Message
	data["severity"] = entry.Level.String()

So we can just do:

&JSONFormatter{
  	FieldMap: FieldMap{
		 FieldKeyTime: "time",
		 FieldKeyLevel: "severity",
		 FieldKeyMsg: "message",
   },
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only requirements, in case of copy-pasting a code with some modifications, is to keep attribution (copyright) and make sure licenses are compatible.

@smacker was there any particular place where this code comes from?

Using logrus JSONFormatter with 3 lines of configuration sounds even better to me, as will avoid extra hassle with licenses/copyrights in future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe I took it from some gist, then found out its actually fluend format not GCloud format. So I took name from joonix. And there is no license in files in logrus.

TimestampFormat string
}

// Format the log entry. Implements logrus.Formatter.
func (f *FluentdFormatter) Format(entry *logrus.Entry) ([]byte, error) {
data := make(logrus.Fields, len(entry.Data)+3)
for k, v := range entry.Data {
switch v := v.(type) {
case error:
// Otherwise errors are ignored by `encoding/json`
// https://github.com/Sirupsen/logrus/issues/137
data[k] = v.Error()
default:
data[k] = v
}
}
prefixFieldClashes(data)

timestampFormat := f.TimestampFormat
if timestampFormat == "" {
timestampFormat = time.RFC3339
}

data["time"] = entry.Time.Format(timestampFormat)
data["message"] = entry.Message
data["severity"] = entry.Level.String()

serialized, err := json.Marshal(data)
if err != nil {
return nil, fmt.Errorf("Failed to marshal fields to JSON, %v", err)
}
return append(serialized, '\n'), nil
}

func prefixFieldClashes(data logrus.Fields) {
if t, ok := data["time"]; ok {
data["fields.time"] = t
}

if m, ok := data["msg"]; ok {
data["fields.msg"] = m
}

if l, ok := data["level"]; ok {
data["fields.level"] = l
}
}