You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fluentd and Loki are part of a flexible chain of service-logging apps. When combined with Prometheus and Grafana, they create a full stack for log presentation and querying. The Fluentd/Loki/Prometheus/Grafana stack provides decision support using time series-based log data and streams from various log formats. This stack can scale when an instance or pod deployment configuration changes.
12
+
[Fluentd](https://www.fluentd.org/) and [Loki](https://grafana.com/oss/loki/) are part of a flexible chain of service-logging apps. When combined with [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/), they create a full stack for log presentation and querying. The Fluentd/Loki/Prometheus/Grafana stack is widely adopted and provides decision support using time series-based log data and streams from various log formats. This stack can scale when an instance or pod deployment configuration changes, and also works with Kubernetes components for cloud-native stack control through our Marketplace [Prometheus & Grafana deployment](https://www.linode.com/marketplace/apps/linode/prometheus-grafana/).
13
13
14
-
This log aggregation and console framework is built from established system log sources, or adapted as needed. This approach helps avoid vendor lock-in due to the popularity and independence of separate product developer communities. The stack's seamless integration make it a widely adopted tool set. Its modular architecture is well-suited for the log monitoring and trend analysis visualization needs of system administrators.
14
+
This log aggregation and console framework is built from established system log sources, or adapted as needed. This approach helps avoid vendor lock-in due to the popularity and independence of separate product developer communities. Its modular architecture is well-suited for the log monitoring and trend analysis visualization needs of system administrators.
15
15
16
-
It also works with Akamai Kubernetes components for cloud-native stack control through the Akamai Cloud Marketplace's Prometheus & Grafana app. It can also be integrated with other Grafana consoles assembled by different methods. For example, a simpler but less flexible stack is available using Loki, Prometheus-Promtail, and Grafana. However, the Fluentd/Loki stack's key utility stems from its extensive number of plugins, rapid query capabilities, and Grafana’s visualization tools.
17
-
18
-
This toolkit captures a wide variety of service logs from diverse instances in single- or multi-tenant environments. Adding labels and timestamps to logs allows for rapid correlation across tenants and their instances, which can help with troubleshooting and forensic examination. Grafana visually couples or decouples sources and tenants. Using appended labels, timestamps, and time series-based sorting, logs from different components are correlated to each other in the Grafana console. This correlation presents events as a time series-based graph visualization of underlying log streams.
16
+
This toolkit captures a wide variety of service logs from diverse instances in single or multi-tenant environments. Adding labels and timestamps to logs allows for rapid correlation across tenants and their instances, which can help with troubleshooting and forensic examination. Grafana visually couples or decouples sources and tenants. Using appended labels, timestamps, and time series-based sorting, logs from different components are correlated to each other in the Grafana console. This correlation presents events as a time series-based graph visualization of underlying log streams.
19
17
20
18
This guide explains how to use Fluentd and Loki to aggregate logs and feeds them to a Prometheus store. Grafana then correlates and visualizes this log data.
21
19
22
20
## Stack Component Relationships
23
21
24
-
The Fluentd/Loki/Prometheus/Grafana stack provides administrators with a way to visually track error messages. Data sources within a time series can optionally be correlated to other event log message streams for rapid fault determination and forensic study via Grafana.
22
+
The Fluentd/Loki/Prometheus/Grafana stack provides administrators with a way to visually track error messages. Fluentd aggregates logs from diverse instance sources. Administrative actions and enabled output plugins send data to Loki. Loki then forwards the data to a Prometheus store, which is then visualized through queries made in Grafana. Data sources within a time series can optionally be correlated to other event log message streams for rapid fault determination and forensic study via Grafana.
25
23
26
-
Fluentd is chosen in this stack for its large number of community-supported source and destination data-handling plugins. These plugins allow for simple customization of source or destination data-handling to or from multiple scalable data streams.
24
+
-**Fluentd** is chosen in this stack for its large number of community-supported source and destination data-handling plugins. These plugins allow for simple customization of source or destination data-handling to or from multiple scalable data streams.
27
25
28
-
Loki is chosen for its aggregation and timestamping of log sources. Loki receives data from Fluentd, marks it, and pipes its output to Prometheus.
26
+
-**Loki** is chosen for its aggregation and timestamping of log sources. Loki receives data from Fluentd, marks it, and pipes its output to Prometheus.
29
27
30
-
Grafana is used for queries, correlating key values, and identifying trends through statistical or graphical data visualization. Prometheus and Loki share interchangeable characteristics in log storage, with Prometheus providing a data store that is queried through an open language.
28
+
-**Prometheus** and Loki share interchangeable characteristics in log storage, with Prometheus providing a data store that is queried through an open language.
31
29
32
-
The lightweight stack recommended in this guide provides options for auto-start with new Kubernetes instances and pod construction. This allows new deployments and instances to begin reporting log data to the Grafana console immediately. By adding other data sources, the time series-based Prometheus data store enables Grafana to correlate events across a wide instance map. This setup facilitates rapid visual trend analysis.
30
+
-**Grafana** is used for queries, correlating key values, and identifying trends through statistical or graphical data visualization.
33
31
34
-
Administrative uses of Grafana include trend analysis and forensic examination. This leads to better visibility for communications, general troubleshooting, load balancing, error details, and other trend-related needs.
32
+
When combined, this stack is lightweight and provides options for auto-start with new Kubernetes instances and pod construction. This allows new deployments and instances to begin reporting log data to the Grafana console immediately. By adding other data sources, the time series-based Prometheus data store enables Grafana to correlate events across a wide instance map. This setup facilitates rapid visual trend analysis.
35
33
36
-
## What Is Fluentd?
34
+
Administrative uses of Grafana include trend analysis and forensic examination. This leads to improved visibility for communications, general troubleshooting, load balancing, error details, and other trend-related needs.
37
35
38
-
Fluentd aggregates logs from diverse instance sources. Administrative actions and enabled output plugins send data to Loki. Loki then forwards the data to a Prometheus store, which is then visualized through queries made in Grafana.
36
+
### What Is Fluentd?
39
37
40
38
The [Cloud Native Computing Foundation](https://www.cncf.io)'s (CNCF) Fluentd is an open source log-aggregation framework with two deployment options: Fluentd and Fluent-bit. Both versions use input plugins to collect logs and output plugins to send the results. The Fluentd binary, written in Ruby, offers more input and output plugin options for differing sources and destinations. The Fluent-bit app, written in C, is a lightweight agent designed for use in virtual machines, containers, pods, and elemental compute sources.
41
39
42
40
From input plugins, Fluentd and Fluent-bit can optionally filter data fetches. After processing the input stream, they aggregate multiple streams and send them to a target destination. In this guide, the target destination of the multiple log sources is an instance of Loki, via a Ruby gem plugin.
43
41
44
-
## What Are Loki and Prometheus?
42
+
###What Are Loki and Prometheus?
45
43
46
44
Loki is a multi-tenant log aggregation system that serves as the source for data visualization by Grafana. It feeds time-series log data to Prometheus, where the aggregated and timestamped data is stored.
47
45
48
-
In this example stack, Loki takes logs as input from log streams aggregated by Fluentd. It can also natively handle Kubernetes logs directly. However, in this example, a wider variety of data sources enhances the options available for correlating and visualizing logs in Grafana.
46
+
In this example stack, Loki takes logs as input from log streams aggregated by Fluentd, but can also natively handle Kubernetes logs directly. A wider variety of data sources enhances the options available for correlating and visualizing logs in Grafana.
49
47
50
-
Logs collected by Fluentd can be categorized based on their single- or multi-tenancy status. Data from output plugin streams or logs from Fluentd are relabeled and timestamped. Loki then stores this data and serves as the source repository for query and correlation by Grafana, the data presentation layer.
48
+
Logs collected by Fluentd can be categorized based on their single or multi-tenancy status. Data from output plugin streams or logs from Fluentd are relabeled and timestamped. Loki then stores this data and serves as the source repository for query and correlation by Grafana, the data presentation layer.
51
49
52
50
## Loki Deployment Modes
53
51
54
52
There are three deployment modes for Loki:
55
53
56
-
-The example in this guide uses *Monolithic mode*. It is the simplest because it utilizes just a single instance and database stack. Multiple instances of monolithic mode can share a common database instance, permitting horizontal scaling, as queries to any instance accesses the same shared database.
54
+
-**Monolithic mode:** The simplest mode, and the example used in this guide. This mode utilizes a single instance and database stack. Multiple instances of monolithic mode can share a common database instance, permitting horizontal scaling, as queries to any instance accesses the same shared database.
57
55
58
-
-The *Scalable Deployment mode* is more complex. It separates reads and writes as data flow targets to prevent input host contention, jamming, and possible log loss. In this mode, Loki reads and writes from data sources and query tools take separate paths. This deployment requires the installation of a load balancer for Loki write nodes, while all other traffic is sent to read nodes. Designed for terabytes of traffic per day, Loki's scalable development mode is for large traffic volumes within a busy framework.
56
+
-**Scalable deployment mode:** A more complex mode that separates reads and writes as data flow targets to prevent input host contention, jamming, and possible log loss. In scalable deployment mode, Loki reads and writes from data sources and query tools take separate paths. This deployment requires the installation of a load balancer for Loki write nodes, while all other traffic is sent to read nodes. Designed for terabytes of traffic per day, scalable development mode is for large traffic volumes within a busy framework.
59
57
60
-
-Loki's third and final mode is*Microservices mode*, tailored for microservices with container fleets, especially those using Kubernetes control planes for pod scaling.
58
+
-**Microservices mode:** This mode is tailored for microservices with container fleets, especially those using Kubernetes control planes for pod scaling.
61
59
62
60
## Stack Installation Considerations
63
61
64
-
Accurate timestamps within log data sources and consistency in changes made through log aggregation processes are critical for ensuring visualization accuracy later in the stack. All instances, whether log sources or log processors, must be synchronized to the same time source. Use a common NTP server for all instances in the stack to ensure synchronization with this time source and maintain system integrity.
62
+
-**Time source:**Accurate timestamps within log data sources and consistency in changes made through log aggregation processes are critical for ensuring visualization accuracy later in the stack. All instances, whether log sources or log processors, must be synchronized to the same time source. Use a common NTP server for all instances in the stack to ensure synchronization with this time source and maintain system integrity.
65
63
66
-
Fluentd plays a crucial role in the logging stack by accumulating logs from various sources using plugins. In this example, log sources include the `/var/log` directories on separate Linux instances and a Kubernetes pod. The source of Fluentd logs is limited to the available source plugins provided by Fluentd or created by users. There are numerous input plugins available for various data sources.
64
+
-**Log and data sources:**Fluentd plays a crucial role in the logging stack by accumulating logs from various sources using plugins. In this example, log sources include the `/var/log` directories on separate Linux instances and a Kubernetes pod. The source of Fluentd logs is limited to the available source plugins provided by Fluentd or created by users. There are numerous input plugins available for various data sources.
67
65
68
-
The gathered Fluentd logs are organized into JSON-formatted entries by Loki. Prometheus stores these Loki logs, which are otherwise ephemeral. The Prometheus store acts as the data source for Grafana's visualization console. Grafana and Prometheus are typically deployed together. This example uses the Prometheus & Grafana installation from the Akamai Cloud Marketplace.
66
+
The gathered Fluentd logs are organized into JSON-formatted entries by Loki. Prometheus stores these Loki logs, which are otherwise ephemeral. The Prometheus store acts as the data source for Grafana's visualization console. Grafana and Prometheus are typically deployed together. This example uses our Marketplace [Prometheus & Grafana installation](https://www.linode.com/marketplace/apps/linode/prometheus-grafana/).
69
67
70
-
Other configurations use Promtail, Loki, Prometheus, and Grafana either separately or in combination. For instance, Loki, Promtail, and Grafana work well in strictly Kubernetes-sourced log consoles, but have limited plugins for other data sources.
68
+
-**Alternative software configurations:**Other configurations use Promtail, Loki, Prometheus, and Grafana either separately or in combination. For instance, Loki, Promtail, and Grafana work well in strictly Kubernetes-sourced log consoles, but have limited plugins for other data sources.
71
69
72
70
## Before You Begin
73
71
74
-
The example stack in this article uses three groups of instances:
72
+
The example stack in this guide uses three groups of instances:
75
73
76
-
-**Group #1**consists of instances to monitor: discrete Linux instances in an Akamai Kubernetes pod.
74
+
-**Group 1:**Consists of discrete Linux instances in a Kubernetes pod for monitoring.
77
75
78
-
-**Group #2**is the instance where Fluentd gathers the logs and sends them to a Loki instance within the same host.
76
+
-**Group 2:**The instance where Fluentd gathers the logs and sends them to a Loki instance within the same host.
79
77
80
-
-**Group #3**consists of an instance running Grafana and Prometheus, deployed to a Nanode instance by the Akamai Cloud Marketplace app.
78
+
-**Group 3:**Consists of an instance running Grafana and Prometheus, deployed to a Nanode using our Prometheus & Grafana Marketplace app.
81
79
82
-
1. If you do not already have a virtual machine to use, create a Compute Instance with at least 4 GB of memory. See our [Getting Started with Linode](/docs/products/platform/get-started/) and [Creating a Compute Instance](/docs/products/compute/compute-instances/guides/create/) guides.
80
+
1. If you do not already one deployed, create a Compute Instance with at least 4 GB of memory. See our [Getting Started with Linode](/docs/products/platform/get-started/) and [Creating a Compute Instance](/docs/products/compute/compute-instances/guides/create/) guides.
83
81
84
82
1. Follow our [Setting Up and Securing a Compute Instance](/docs/products/compute/compute-instances/guides/set-up-and-secure/) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access.
85
83
@@ -89,23 +87,23 @@ This guide is written for a non-root user. Commands that require elevated privil
89
87
90
88
## Prometheus-Grafana Installation
91
89
92
-
The Akamai Cloud Marketplace Prometheus & Grafana app renders a standalone server instance. Follow the steps below to deploy it:
90
+
The Prometheus & Grafana Marketplace deployment renders a standalone server instance. Follow the steps below to deploy it:
93
91
94
-
1. Choose the **Marketplace** option from the left menu in the Akamai Cloud dashboard.
92
+
1. Choose the **Marketplace** option from the left menu in the Cloud Manager.
95
93
96
-
1. Select **Prometheus & Grafana** from the Akamai Marketplace app menu. The Akamai Cloud [Get Started Guide](/docs/products/tools/marketplace/guides/grafana/)contains full deployment instructions for this app, but here are some options to choose:
94
+
1. Select **Prometheus & Grafana** from the Marketplace menu. See our [Deploy Prometheus and Grafana through the Linode Marketplace](/docs/products/tools/marketplace/guides/grafana/)for full deployment instructions. Below are some options to consider:
97
95
98
-
-Under **Select an Image**, there's an option for Ubuntu 22.04 LTS or Ubuntu 20.04 LTS. This example uses an Ubuntu 22.04 LTS instance.
96
+
-**Select an Image:** This example uses an Ubuntu 22.04 LTS instance.
99
97
100
-
-Under **Linode Plan**, click the **Shared CPU** tab. A minimal **Nanode 1 GB** is suitable for this example stack.
98
+
-**Linode Plan:** A Nanode 1 GB plan is suitable for this example stack.
1. Once the instance installs, it deploys a Let’s Encrypt TLS certificate, which allows you to access the instance via `HTTPS`in a web browser. When fully installed, the Grafana settings menu provides fields to connect to the Loki/Fluentd combination:
134
+
1. During deployment, a Let’s Encrypt TLS certificate is installed. This allows you to access the instance via `HTTPS`in a web browser. When fully installed, the Grafana settings menu provides fields to connect to the Loki/Fluentd combination:
137
135
138
136
0 commit comments