An end-to-end AIOps (Artificial Intelligence for IT Operations) platform built with a microservices architecture. This project demonstrates how to use Machine Learning to detect anomalies, predict incidents, and perform Root Cause Analysis (RCA) on a distributed system in real-time.
- Simulated Microservices Environment: High-concurrency Go services (API Gateway, Auth, Background Worker) generating real-time metrics and logs.
- Real-time Anomaly Detection: A Python-based ML service using
Isolation Forestto detect unusual latency spikes or traffic patterns. - Incident Prediction: An
XGBoostmodel that forecasts potential system failures based on historical metric trends. - Automated Root Cause Analysis (RCA): Correlates logs and metrics during an incident to pinpoint the failing service.
- Full Observability Stack: Integrated Prometheus (metrics), Loki (logs), and Grafana (dashboards) for complete system visibility.
- Chaos Engineering / Load Testing: A built-in script to generate synthetic traffic, simulate errors, and trigger the AI models.
- Backend Services: Go (Golang)
- AI / ML Services: Python (Scikit-learn, XGBoost, FastAPI)
- Infrastructure: Docker & Docker Compose
- Observability: Prometheus, Grafana, Promtail, Loki, Alertmanager
Before you begin, ensure you have the following installed on your machine:
- Docker Desktop (v24.0+ recommended)
- Docker Compose (v2.0+)
- Windows users: Ensure WSL2 is enabled.
Follow these exact steps to start the platform, generate data, and see the AI in action.
Clone the repository, navigate to the root directory, and start the Docker containers:
# Build and start all services in detached mode
docker compose up --build -d
(Note: The first build takes 10-15 minutes to download ML libraries. Subsequent starts take seconds).
Verify everything is running successfully:
Bash
docker compose ps
All services should show a status of Up.
2. Generate Traffic & Simulate Incidents
The system needs data for the AI to analyze. We will use the included load testing script to hit the endpoints, generate logs, and create simulated latency/error spikes.
Run the load test script and leave it running in your terminal:
Bash
chmod +x scripts/load_test.sh
./scripts/load_test.sh
3. Observe the AI Anomaly Detection
Open a new terminal window/tab to watch the Python machine learning service process the incoming data and flag anomalies in real-time:
Bash
docker compose logs -f anomaly_detector
4. View the Dashboards
With data flowing, open your web browser to explore the observability stack:
Grafana (Dashboards): http://localhost:3000
Login: admin
Password: admin (Skip the prompt to change password).
Go to Dashboards to view your CPU, Memory, Latency, and AI Anomaly Scores.
Prometheus (Raw Metrics): http://localhost:9090
API Gateway Health Check: http://localhost:8080/health
🧪 How to Reproduce Specific Scenarios
Scenario A: Triggering an Auth Service Failure
The load_test.sh script randomly generates invalid login attempts.
Watch the Promtail/Loki logs in Grafana.
The rca_service will catch the spike in 401 Unauthorized errors and correlate it to the Auth container.
Check the rca_service logs: docker compose logs -f rca_service
Scenario B: High Latency Anomaly
The load tester occasionally sends massive bursts of traffic.
The API Gateway will report increased latency to Prometheus.
The anomaly_detector (running Isolation Forest) scrapes Prometheus, detects the outlier, and flags the anomaly metric.
View the Anomaly Score spiking on your Grafana dashboard.
🧹 Cleanup & Useful Commands
To safely stop the platform and remove the containers:
Bash
docker compose down
To completely wipe all data volumes (resets Prometheus/Grafana data):
Bash
docker compose down -v
// run the command
AIOps Incident Predictor: Operational Commands
1. Project Lifecycle
Use these commands to start or stop your entire observability and AI platform.
Start the platform (detached mode):
docker compose up -d
Stop and remove all containers:
docker compose down
Restart everything (after code changes):
docker compose up -d --build
2. Log Management
Use these to debug your Python AI services or monitor Go microservices in real-time.
View Anomaly Detector logs (real-time):
docker compose logs anomaly-detector -f
View Incident Predictor logs (real-time):
docker compose logs incident-predictor -f
View RCA Service logs (real-time):
docker compose logs rca-service -f
View all logs (concatenated):
docker compose logs -f
3. Load Simulation
Use these to trigger the metrics that your AI models are monitoring.
Start the background load generator:
./scripts/load_test.sh
(Ensure you run chmod +x scripts/load_test.sh first).
Stop the load generator:
Press Ctrl+C in the terminal where the script is running.
4. RCA Webhook Simulation
Use this to manually trigger an incident report to test your RCA service without waiting for an alert.
Manually trigger the RCA Webhook:
Bash
docker run --rm --network aiops-incident-predictor_default \
curlimages/curl -X POST http://rca-service:8000/webhook \
-H "Content-Type: application/json" \
-d '{"alerts": [{"status": "firing", "labels": {"alertname": "HighIncident