You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fluentd and Loki are part of a flexible chain of service-logging apps. When combined with Grafana and Prometheus, they create a full stack for log presentation and querying. The Fluentd/Loki/Prometheus/Grafana stack provides decision support using time series-based log data and streams from various log formats. This stack can scale with instance- or pod-deployment configuration changes.
12
+
Fluentd and Loki are part of a flexible chain of service-logging apps. When combined with Prometheus and Grafana, they create a full stack for log presentation and querying. The Fluentd/Loki/Prometheus/Grafana stack provides decision support using time series-based log data and streams from various log formats. This stack can scale when an instance or poddeployment configuration changes.
13
13
14
14
This log aggregation and console framework is built from established system log sources, or adapted as needed. This approach helps avoid vendor lock-in due to the popularity and independence of separate product developer communities. The stack's seamless integration make it a widely adopted tool set. Its modular architecture is well-suited for the log monitoring and trend analysis visualization needs of system administrators.
15
15
16
-
It also works with Akamai Kubernetes components for cloud-native stack control through the Akamai Cloud Marketplace Prometheus & Grafana deployment. It can also be integrated with other Grafana consoles assembled by different methods. For example, a simpler but less flexible stack is available using Loki, Prometheus-Promtail, and Grafana. However, the Fluentd/Loki stack's key utility stems from its extensive number of plugins, rapid query capabilities, and Grafana’s visualization tools.
16
+
It also works with Akamai Kubernetes components for cloud-native stack control through the Akamai Cloud Marketplace's Prometheus & Grafana app. It can also be integrated with other Grafana consoles assembled by different methods. For example, a simpler but less flexible stack is available using Loki, Prometheus-Promtail, and Grafana. However, the Fluentd/Loki stack's key utility stems from its extensive number of plugins, rapid query capabilities, and Grafana’s visualization tools.
17
17
18
18
This toolkit captures a wide variety of service logs from diverse instances in single- or multi-tenant environments. Adding labels and timestamps to logs allows for rapid correlation across tenants and their instances, which can help with troubleshooting and forensic examination. Grafana visually couples or decouples sources and tenants. Using appended labels, timestamps, and time series-based sorting, logs from different components are correlated to each other in the Grafana console. This correlation presents events as a time series-based graph visualization of underlying log streams.
19
19
@@ -49,7 +49,7 @@ In this example stack, Loki takes logs as input from log streams aggregated by F
49
49
50
50
Logs collected by Fluentd can be categorized based on their single- or multi-tenancy status. Data from output plugin streams or logs from Fluentd are relabeled and timestamped. Loki then stores this data and serves as the source repository for query and correlation by Grafana, the data presentation layer.
51
51
52
-
###Loki Deployment Modes
52
+
## Loki Deployment Modes
53
53
54
54
There are three deployment modes for Loki:
55
55
@@ -59,7 +59,7 @@ There are three deployment modes for Loki:
59
59
60
60
- Loki's third and final mode is *Microservices mode*, tailored for microservices with container fleets, especially those using Kubernetes control planes for pod scaling.
61
61
62
-
###Stack Installation Considerations
62
+
## Stack Installation Considerations
63
63
64
64
Accurate timestamps within log data sources and consistency in changes made through log aggregation processes are critical for ensuring visualization accuracy later in the stack. All instances, whether log sources or log processors, must be synchronized to the same time source. Use a common NTP server for all instances in the stack to ensure synchronization with this time source and maintain system integrity.
65
65
@@ -69,17 +69,25 @@ The gathered Fluentd logs are organized into JSON-formatted entries by Loki. Pro
69
69
70
70
Other configurations use Promtail, Loki, Prometheus, and Grafana either separately or in combination. For instance, Loki, Promtail, and Grafana work well in strictly Kubernetes-sourced log consoles, but have limited plugins for other data sources.
71
71
72
-
This example stack uses three groups of host instances:
72
+
## Before You Begin
73
73
74
-
-The first group consists of instances to monitor: discrete Linux instances in an Akamai Kubernetes pod.
74
+
The example stack in this article uses three groups of instances:
75
75
76
-
-The second group is the host where Fluentd gathers the logs and sends them to a Loki instance within the same host.
76
+
-**Group #1** consists of instances to monitor: discrete Linux instances in an Akamai Kubernetes pod.
77
77
78
-
-The third group consists of an instance running Grafana and Prometheus, deployed to a Nanode instance by the Akamai Cloud Marketplace app.
78
+
-**Group #2** is the instance where Fluentd gathers the logs and sends them to a Loki instance within the same host.
79
79
80
-
## Stack Installation Steps
80
+
-**Group #3** consists of an instance running Grafana and Prometheus, deployed to a Nanode instance by the Akamai Cloud Marketplace app.
81
81
82
-
### Grafana-Prometheus
82
+
1. If you do not already have a virtual machine to use, create a Compute Instance with at least 4 GB of memory. See our [Getting Started with Linode](/docs/products/platform/get-started/) and [Creating a Compute Instance](/docs/products/compute/compute-instances/guides/create/) guides.
83
+
84
+
1. Follow our [Setting Up and Securing a Compute Instance](/docs/products/compute/compute-instances/guides/set-up-and-secure/) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access.
85
+
86
+
{{< note >}}
87
+
This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/docs/guides/linux-users-and-groups/) guide.
88
+
{{< /note >}}
89
+
90
+
## Prometheus-Grafana Installation
83
91
84
92
The Akamai Cloud Marketplace Prometheus & Grafana app renders a standalone server instance. Follow the steps below to deploy it:
85
93
@@ -129,9 +137,9 @@ The Akamai Cloud Marketplace Prometheus & Grafana app renders a standalone serve
129
137
130
138

131
139
132
-
### Fluentd
140
+
## Fluentd Installation
133
141
134
-
Fluentd gathers log instances via Fluentd and plugins. This example uses a Ruby gem version of Fluentd. The commands below install the build tools, Ruby and its development libraries, and finally, Fluentd:
142
+
Fluentd gathers log instances via Fluentd and plugins. This example uses a Ruby gem version of Fluentd onto a Nanode. The commands below install the build tools, Ruby and its development libraries, and Fluentd:
135
143
136
144
1. Update and upgrade the Ubuntu system, then restart the Nanode:
137
145
@@ -177,9 +185,9 @@ Fluentd gathers log instances via Fluentd and plugins. This example uses a Ruby
177
185
fluentd -c ./fluent/fluent.conf -vv &
178
186
```
179
187
180
-
### Loki
188
+
## Loki Installation
181
189
182
-
The example log monitoring stack uses the same instance that hosts Fluentd. The plugin that links Loki as the accumulator of Fluentd logs is installed and attached after Fluentd and Loki.
190
+
The example log monitoring stack uses the same Nanode instance that hosts Fluentd. The plugin that links Loki as the accumulator of Fluentd logs is installed and attached after Fluentd and Loki.
183
191
184
192
1. First, download the Debian package from the Grafana Loki Releases page:
185
193
@@ -213,12 +221,14 @@ Authentication options for Grafana's Data Source range from none to TLS Certific
213
221
214
222
Grafana also offers a context-sensitive log query builder. This tool allows you to link common and disparate data sources into a time-series format. You can then use this data for various graphical visualizations, such as histograms, to correlate the query results.
215
223
216
-
## Final Results
224
+
## Generating Visualizations
217
225
218
-
The web browser interface allows you to selectspecifictime frames and fields to generate histograms of log data occurrences. In the screenshot below, a 24-hour time-series across hosts shows traffic sorted by critical messages. These messages are correlated from log sources originating from `/var/log/` information across the monitored sample instances and Kubernetes pod.
226
+
The web browser interface allows you to selectspecifictime frames and fields to generate histograms of log data occurrences. For example, the screenshot belowshows traffic sorted by critical messages within a 24-hour time-series across hosts:
219
227
220
228

221
229
230
+
These messages are correlated from log sources originating from `/var/log/` information across the monitored sample instances and Kubernetes pod.
231
+
222
232
## Conclusion
223
233
224
234
The Fluentd/Loki combination excels in handling diverse log source streams and efficiently archiving log data. The Prometheus/Grafana combination serves as the log store archive and central hub for visualizing time-series events across various log sources, whether discrete instances or Kubernetes pods.
0 commit comments