Skip to content

Commit 94df593

Browse files
committed
Tech Edit 2
1 parent 4b1163e commit 94df593

File tree

1 file changed

+26
-16
lines changed
  • docs/guides/uptime/logs/how-to-use-fluentd-and-loki-to-access-service-logs

1 file changed

+26
-16
lines changed

docs/guides/uptime/logs/how-to-use-fluentd-and-loki-to-access-service-logs/index.md

Lines changed: 26 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,11 @@ keywords: ['fluentd and loki','fluentd','loki','k8s','open source data logging',
99
license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)'
1010
---
1111

12-
Fluentd and Loki are part of a flexible chain of service-logging apps. When combined with Grafana and Prometheus, they create a full stack for log presentation and querying. The Fluentd/Loki/Prometheus/Grafana stack provides decision support using time series-based log data and streams from various log formats. This stack can scale with instance- or pod-deployment configuration changes.
12+
Fluentd and Loki are part of a flexible chain of service-logging apps. When combined with Prometheus and Grafana, they create a full stack for log presentation and querying. The Fluentd/Loki/Prometheus/Grafana stack provides decision support using time series-based log data and streams from various log formats. This stack can scale when an instance or pod deployment configuration changes.
1313

1414
This log aggregation and console framework is built from established system log sources, or adapted as needed. This approach helps avoid vendor lock-in due to the popularity and independence of separate product developer communities. The stack's seamless integration make it a widely adopted tool set. Its modular architecture is well-suited for the log monitoring and trend analysis visualization needs of system administrators.
1515

16-
It also works with Akamai Kubernetes components for cloud-native stack control through the Akamai Cloud Marketplace Prometheus & Grafana deployment. It can also be integrated with other Grafana consoles assembled by different methods. For example, a simpler but less flexible stack is available using Loki, Prometheus-Promtail, and Grafana. However, the Fluentd/Loki stack's key utility stems from its extensive number of plugins, rapid query capabilities, and Grafana’s visualization tools.
16+
It also works with Akamai Kubernetes components for cloud-native stack control through the Akamai Cloud Marketplace's Prometheus & Grafana app. It can also be integrated with other Grafana consoles assembled by different methods. For example, a simpler but less flexible stack is available using Loki, Prometheus-Promtail, and Grafana. However, the Fluentd/Loki stack's key utility stems from its extensive number of plugins, rapid query capabilities, and Grafana’s visualization tools.
1717

1818
This toolkit captures a wide variety of service logs from diverse instances in single- or multi-tenant environments. Adding labels and timestamps to logs allows for rapid correlation across tenants and their instances, which can help with troubleshooting and forensic examination. Grafana visually couples or decouples sources and tenants. Using appended labels, timestamps, and time series-based sorting, logs from different components are correlated to each other in the Grafana console. This correlation presents events as a time series-based graph visualization of underlying log streams.
1919

@@ -49,7 +49,7 @@ In this example stack, Loki takes logs as input from log streams aggregated by F
4949

5050
Logs collected by Fluentd can be categorized based on their single- or multi-tenancy status. Data from output plugin streams or logs from Fluentd are relabeled and timestamped. Loki then stores this data and serves as the source repository for query and correlation by Grafana, the data presentation layer.
5151

52-
### Loki Deployment Modes
52+
## Loki Deployment Modes
5353

5454
There are three deployment modes for Loki:
5555

@@ -59,7 +59,7 @@ There are three deployment modes for Loki:
5959

6060
- Loki's third and final mode is *Microservices mode*, tailored for microservices with container fleets, especially those using Kubernetes control planes for pod scaling.
6161

62-
### Stack Installation Considerations
62+
## Stack Installation Considerations
6363

6464
Accurate timestamps within log data sources and consistency in changes made through log aggregation processes are critical for ensuring visualization accuracy later in the stack. All instances, whether log sources or log processors, must be synchronized to the same time source. Use a common NTP server for all instances in the stack to ensure synchronization with this time source and maintain system integrity.
6565

@@ -69,17 +69,25 @@ The gathered Fluentd logs are organized into JSON-formatted entries by Loki. Pro
6969

7070
Other configurations use Promtail, Loki, Prometheus, and Grafana either separately or in combination. For instance, Loki, Promtail, and Grafana work well in strictly Kubernetes-sourced log consoles, but have limited plugins for other data sources.
7171

72-
This example stack uses three groups of host instances:
72+
## Before You Begin
7373

74-
- The first group consists of instances to monitor: discrete Linux instances in an Akamai Kubernetes pod.
74+
The example stack in this article uses three groups of instances:
7575

76-
- The second group is the host where Fluentd gathers the logs and sends them to a Loki instance within the same host.
76+
- **Group #1** consists of instances to monitor: discrete Linux instances in an Akamai Kubernetes pod.
7777

78-
- The third group consists of an instance running Grafana and Prometheus, deployed to a Nanode instance by the Akamai Cloud Marketplace app.
78+
- **Group #2** is the instance where Fluentd gathers the logs and sends them to a Loki instance within the same host.
7979

80-
## Stack Installation Steps
80+
- **Group #3** consists of an instance running Grafana and Prometheus, deployed to a Nanode instance by the Akamai Cloud Marketplace app.
8181

82-
### Grafana-Prometheus
82+
1. If you do not already have a virtual machine to use, create a Compute Instance with at least 4 GB of memory. See our [Getting Started with Linode](/docs/products/platform/get-started/) and [Creating a Compute Instance](/docs/products/compute/compute-instances/guides/create/) guides.
83+
84+
1. Follow our [Setting Up and Securing a Compute Instance](/docs/products/compute/compute-instances/guides/set-up-and-secure/) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access.
85+
86+
{{< note >}}
87+
This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/docs/guides/linux-users-and-groups/) guide.
88+
{{< /note >}}
89+
90+
## Prometheus-Grafana Installation
8391

8492
The Akamai Cloud Marketplace Prometheus & Grafana app renders a standalone server instance. Follow the steps below to deploy it:
8593

@@ -129,9 +137,9 @@ The Akamai Cloud Marketplace Prometheus & Grafana app renders a standalone serve
129137

130138
![The Add New Connections page in the Connections section of Grafana's left-side menu.](Grafana-Connections-Add-New-Connections-Menu.png)
131139
132-
### Fluentd
140+
## Fluentd Installation
133141
134-
Fluentd gathers log instances via Fluentd and plugins. This example uses a Ruby gem version of Fluentd. The commands below install the build tools, Ruby and its development libraries, and finally, Fluentd:
142+
Fluentd gathers log instances via Fluentd and plugins. This example uses a Ruby gem version of Fluentd onto a Nanode. The commands below install the build tools, Ruby and its development libraries, and Fluentd:
135143
136144
1. Update and upgrade the Ubuntu system, then restart the Nanode:
137145
@@ -177,9 +185,9 @@ Fluentd gathers log instances via Fluentd and plugins. This example uses a Ruby
177185
fluentd -c ./fluent/fluent.conf -vv &
178186
```
179187
180-
### Loki
188+
## Loki Installation
181189
182-
The example log monitoring stack uses the same instance that hosts Fluentd. The plugin that links Loki as the accumulator of Fluentd logs is installed and attached after Fluentd and Loki.
190+
The example log monitoring stack uses the same Nanode instance that hosts Fluentd. The plugin that links Loki as the accumulator of Fluentd logs is installed and attached after Fluentd and Loki.
183191
184192
1. First, download the Debian package from the Grafana Loki Releases page:
185193
@@ -213,12 +221,14 @@ Authentication options for Grafana's Data Source range from none to TLS Certific
213221

214222
Grafana also offers a context-sensitive log query builder. This tool allows you to link common and disparate data sources into a time-series format. You can then use this data for various graphical visualizations, such as histograms, to correlate the query results.
215223

216-
## Final Results
224+
## Generating Visualizations
217225

218-
The web browser interface allows you to select specific time frames and fields to generate histograms of log data occurrences. In the screenshot below, a 24-hour time-series across hosts shows traffic sorted by critical messages. These messages are correlated from log sources originating from `/var/log/` information across the monitored sample instances and Kubernetes pod.
226+
The web browser interface allows you to select specific time frames and fields to generate histograms of log data occurrences. For example, the screenshot below shows traffic sorted by critical messages within a 24-hour time-series across hosts:
219227

220228
![Grafana data stream with field selection criteria example.](fluentd-grafana-critical-message-dashboard.png)
221229

230+
These messages are correlated from log sources originating from `/var/log/` information across the monitored sample instances and Kubernetes pod.
231+
222232
## Conclusion
223233

224234
The Fluentd/Loki combination excels in handling diverse log source streams and efficiently archiving log data. The Prometheus/Grafana combination serves as the log store archive and central hub for visualizing time-series events across various log sources, whether discrete instances or Kubernetes pods.

0 commit comments

Comments
 (0)