Monitors randomly stop working - monitors locked but never unlocked?

[Stitch10925]

Describe the bug
Monitors randomly break causing them to not alert anymore on incidents. I also cannot get them back to work without removing them and re-adding them. Then they work for a while again, but end up breaking again after a while.

I have read some other (closed) issues suggesting persistence issues, but this is not the case here:

• I only have Checkmate running on 1 dedicated machine
• The database is a bind-mount to the file system, so not a volume:
  - /docker_compose/checkmate/mongo/data:/data/db
• The server has not been down or turned off

To Reproduce
I cannot reproduce it at will.

Expected behavior
I expect the monitors to keep running and be fault-tolerant

Screenshots

---

---

---

---

---

Additional context
From the Job queue, it seems like the entries for these monitors are locked, but never unlocked. See screenshot.

[Stitch10925]

I just found the "Flush queue" button. Pushing it makes the monitors work again, so there is a work-around.

[KhurshedSayfiddinov]

How is your setup? Our Docker doesn't draw any graphs at all; it only finishes drawing when you refresh the browser page. Can you send me your Docker compose.yml?

[Stitch10925]

@KhurshedSayfiddinov
My compose is as follows:

version: '3.8'

services:
  server:
    image: ghcr.io/bluewave-labs/checkmate-backend-mono:v3.2.0
    restart: unless-stopped
    ports:
      - 9080:52345
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - UPTIME_APP_API_BASE_URL=http://hostOrIp:9080/api/v1
      - UPTIME_APP_CLIENT_HOST=http://hostOrIp
      - DB_CONNECTION_STRING=mongodb://mongodb:27017/uptime_db
      - CLIENT_HOST=http://hostOrIp
      - JWT_SECRET=secretToUseButMakeItLonger
    depends_on:
      - mongodb
  
  mongodb:
    image: ghcr.io/bluewave-labs/checkmate-mongo:v3.2.0
    restart: always
    command:
      - mongod
      - --quiet
      - --bind_ip_all
    volumes:
      - /docker_compose/checkmate/mongo/data:/data/db
    healthcheck:
      test:
        - CMD
        - mongosh
        - --eval
        - db.adminCommand('ping')
        - --quiet
      interval: 5s
      timeout: 30s
      start_period: 0s
      start_interval: 1s
      retries: 30

  capture:
    image: ghcr.io/bluewave-labs/capture:v1.3.2
    container_name: capture
    restart: unless-stopped
    depends_on:
      - server
    ports:
      - 9081:59232
    environment:
      - API_SECRET=secretToUseToCoupleInTheUI
    volumes:
      - /etc/os-release:/etc/os-release:ro

[KhurshedSayfiddinov]

@Stitch10925 Tnx! Do you have websocket configured? How do you update your graphs? I can't refresh the browser page, but the graphs aren't rendering.

[Stitch10925]

@KhurshedSayfiddinov
I didn't configure anything besides what's in the compose file. Graphs are updated after checks are done. Is your db corrupt maybe?

[KhurshedSayfiddinov]

version: '3.8'

services:
server:
image: ghcr.io/bluewave-labs/checkmate-backend-mono:v3.2.0
restart: unless-stopped
ports:
- 9080:52345
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- UPTIME_APP_API_BASE_URL=http://10.64.20.107:9080/api/v1
- UPTIME_APP_CLIENT_HOST=http://10.64.20.107
- DB_CONNECTION_STRING=mongodb://mongodb:27017/uptime_db
- CLIENT_HOST=http://10.64.20.107
- JWT_SECRET=a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6
depends_on:
- mongodb

mongodb:
image: ghcr.io/bluewave-labs/checkmate-mongo:v3.2.0
restart: always
command:
- mongod
- --quiet
- --bind_ip_all
volumes:
- /docker_compose/checkmate/mongo/data:/data/db
healthcheck:
test:
- CMD
- mongosh
- --eval
- db.adminCommand('ping')
- --quiet
interval: 5s
timeout: 30s
start_period: 0s
start_interval: 1s
retries: 30

capture:
image: ghcr.io/bluewave-labs/capture:v1.3.2
container_name: capture
restart: unless-stopped
depends_on:
- server
ports:
- 9081:59232
environment:
- API_SECRET=a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6
volumes:
- /etc/os-release:/etc/os-release:ro

@Stitch10925
I installed it using your file. First, I deleted everything, then deleted the root folder and reinstalled everything from scratch. I set up monitoring by pinging the server's IP address, but the graphs still don't display until I refresh the browser. I can't figure out what's going on.

[Stitch10925]

@KhurshedSayfiddinov
I don't know either, sorry

[Stitch10925]

@devs

It seems that when using the advanced JSON content checking, and the JSON does not return or does not return the expected the value the monitor gets stuck and doesn't unlock anymore.