Skip to content

README.md

Latest commit

 

History

History
161 lines (105 loc) · 5.39 KB

README.md

File metadata and controls

161 lines (105 loc) · 5.39 KB
Raw

🚀 Microservices AIOps Platform

An end-to-end AIOps (Artificial Intelligence for IT Operations) platform built with a microservices architecture. This project demonstrates how to use Machine Learning to detect anomalies, predict incidents, and perform Root Cause Analysis (RCA) on a distributed system in real-time.

✨ Key Features & Functionalities

  1. Simulated Microservices Environment: High-concurrency Go services (API Gateway, Auth, Background Worker) generating real-time metrics and logs.
  2. Real-time Anomaly Detection: A Python-based ML service using Isolation Forest to detect unusual latency spikes or traffic patterns.
  3. Incident Prediction: An XGBoost model that forecasts potential system failures based on historical metric trends.
  4. Automated Root Cause Analysis (RCA): Correlates logs and metrics during an incident to pinpoint the failing service.
  5. Full Observability Stack: Integrated Prometheus (metrics), Loki (logs), and Grafana (dashboards) for complete system visibility.
  6. Chaos Engineering / Load Testing: A built-in script to generate synthetic traffic, simulate errors, and trigger the AI models.

🛠️ Tech Stack

  • Backend Services: Go (Golang)
  • AI / ML Services: Python (Scikit-learn, XGBoost, FastAPI)
  • Infrastructure: Docker & Docker Compose
  • Observability: Prometheus, Grafana, Promtail, Loki, Alertmanager

🚦 Prerequisites

Before you begin, ensure you have the following installed on your machine:

  • Docker Desktop (v24.0+ recommended)
  • Docker Compose (v2.0+)
  • Windows users: Ensure WSL2 is enabled.

🚀 Quick Start Guide

Follow these exact steps to start the platform, generate data, and see the AI in action.

1. Start the Platform

Clone the repository, navigate to the root directory, and start the Docker containers:

# Build and start all services in detached mode
docker compose up --build -d
(Note: The first build takes 10-15 minutes to download ML libraries. Subsequent starts take seconds).

Verify everything is running successfully:

Bash
docker compose ps
All services should show a status of Up.

2. Generate Traffic & Simulate Incidents
The system needs data for the AI to analyze. We will use the included load testing script to hit the endpoints, generate logs, and create simulated latency/error spikes.

Run the load test script and leave it running in your terminal:

Bash
chmod +x scripts/load_test.sh
./scripts/load_test.sh
3. Observe the AI Anomaly Detection
Open a new terminal window/tab to watch the Python machine learning service process the incoming data and flag anomalies in real-time:

Bash
docker compose logs -f anomaly_detector
4. View the Dashboards
With data flowing, open your web browser to explore the observability stack:

Grafana (Dashboards): http://localhost:3000

Login: admin

Password: admin (Skip the prompt to change password).

Go to Dashboards to view your CPU, Memory, Latency, and AI Anomaly Scores.

Prometheus (Raw Metrics): http://localhost:9090

API Gateway Health Check: http://localhost:8080/health

🧪 How to Reproduce Specific Scenarios
Scenario A: Triggering an Auth Service Failure
The load_test.sh script randomly generates invalid login attempts.

Watch the Promtail/Loki logs in Grafana.

The rca_service will catch the spike in 401 Unauthorized errors and correlate it to the Auth container.

Check the rca_service logs: docker compose logs -f rca_service

Scenario B: High Latency Anomaly
The load tester occasionally sends massive bursts of traffic.

The API Gateway will report increased latency to Prometheus.

The anomaly_detector (running Isolation Forest) scrapes Prometheus, detects the outlier, and flags the anomaly metric.

View the Anomaly Score spiking on your Grafana dashboard.

🧹 Cleanup & Useful Commands
To safely stop the platform and remove the containers:

Bash
docker compose down
To completely wipe all data volumes (resets Prometheus/Grafana data):

Bash
docker compose down -v


// run the command 


AIOps Incident Predictor: Operational Commands
1. Project Lifecycle
Use these commands to start or stop your entire observability and AI platform.

Start the platform (detached mode):
docker compose up -d

Stop and remove all containers:
docker compose down

Restart everything (after code changes):
docker compose up -d --build

2. Log Management
Use these to debug your Python AI services or monitor Go microservices in real-time.

View Anomaly Detector logs (real-time):
docker compose logs anomaly-detector -f

View Incident Predictor logs (real-time):
docker compose logs incident-predictor -f

View RCA Service logs (real-time):
docker compose logs rca-service -f

View all logs (concatenated):
docker compose logs -f

3. Load Simulation
Use these to trigger the metrics that your AI models are monitoring.

Start the background load generator:
./scripts/load_test.sh
(Ensure you run chmod +x scripts/load_test.sh first).

Stop the load generator:
Press Ctrl+C in the terminal where the script is running.

4. RCA Webhook Simulation
Use this to manually trigger an incident report to test your RCA service without waiting for an alert.

Manually trigger the RCA Webhook:

Bash
docker run --rm --network aiops-incident-predictor_default \
  curlimages/curl -X POST http://rca-service:8000/webhook \
  -H "Content-Type: application/json" \
  -d '{"alerts": [{"status": "firing", "labels": {"alertname": "HighIncident