Make it more obvious that `max_file` is exceeded

[alexbozhenko]

Proposed change

I saw a scenario where available disk space was decreased while the server was down(let's say new logs from other services were written to disk while nats was down).
It was not immediately obvious what is going on

1. Publish 500M of messages:

 nats bench  js  pub sync  test --replicas=3 --msgs=500 --storage=file --create --clients 100 --purge  --maxbytes="1GB" --size=1MB

2. Stop the cluster
3. Update the config to this:

jetstream {
  max_file: 490M
}

4. Start the cluster

# nats server report jetstream
╭────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                          JetStream Summary                                         │
├────────┬──────────────┬─────────┬───────────┬──────────┬───────┬────────┬──────┬─────────┬─────────┤
│ Server │ Cluster      │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File │ API Req │ Pending │
├────────┼──────────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ s1     │ proc_compose │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 5       │       0 │
│ s2     │ proc_compose │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 1       │       0 │
│ s3*    │ proc_compose │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 10      │       0 │
├────────┼──────────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│        │              │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 16      │       0 │
╰────────┴──────────────┴─────────┴───────────┴──────────┴───────┴────────┴──────┴─────────┴─────────╯

╭───────────────────────────────────────────────────────────────────────╮
│                      RAFT Meta Group Information                      │
├─────────────────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Connection Name │ ID       │ Leader │ Current │ Online │ Active │ Lag │
├─────────────────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ s1              │ k2My6qdB │        │ true    │ true   │ 962ms  │ 0   │
│ s2              │ noC8kOtg │        │ true    │ true   │ 962ms  │ 0   │
│ s3              │ 3ahZoO2Q │ yes    │ true    │ true   │ 0s     │ 0   │
╰─────────────────┴──────────┴────────┴─────────┴────────┴────────┴─────╯

The check that we use in Liveliness probe:
https://github.com/nats-io/k8s/blob/0329e987476428736dc4dc9c024f086bd35a6604/helm/charts/nats/files/stateful-set/nats-container.yaml#L38-L64
Will return "ok":

# curl http://localhost:8222/healthz?js-enabled-only=true
{"status":"ok"}

Full health check does show the issue though:

# nats server report health
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                           Health Report                                                          │
├────────┬──────────────┬────────┬────────────────┬────────┬───────────────────────────────────────────────────────────────────────┤
│ Server │ Cluster      │ Domain │ Status         │ Type   │ Error                                                                 │
├────────┼──────────────┼────────┼────────────────┼────────┼───────────────────────────────────────────────────────────────────────┤
│ s1     │ proc_compose │        │ error (503)    │        │                                                                       │
│        │              │        │                │ STREAM │ JetStream stream 'ACC > benchstream' is not current: stream not found │
│ s2     │ proc_compose │        │ error (503)    │        │                                                                       │
│        │              │        │                │ STREAM │ JetStream stream 'ACC > benchstream' is not current: stream not found │
│ s3     │ proc_compose │        │ error (503)    │        │                                                                       │
│        │              │        │                │ STREAM │ JetStream stream 'ACC > benchstream' is not current: stream not found │
├────────┼──────────────┼────────┼────────────────┼────────┼───────────────────────────────────────────────────────────────────────┤
│ 3      │ 1            │        │ ok: 0 / err: 3 │        │ 3                                                                     │
╰────────┴──────────────┴────────┴────────────────┴────────┴───────────────────────────────────────────────────────────────────────╯

Should nats server report jetstream report on such condition somehow?

Do we want to update probes in k8s to account for such condition?

Use case

Make troubleshooting easier.

Contribution

No response

[wallyqs]

when JS runs out of disk space it should auto disable, so curl http://localhost:8222/healthz?js-enabled-only=true should have returned false.

[alexbozhenko]

@wallyqs I just double-checked.
Sometimes(when it fails to write), it logs this and actually disables jetstream.

[1] 2025/12/16 21:21:26.548289 [ERR] JetStream failed to store a msg on stream 'ACC > benchstream': write /data/jetstream/ACC/streams/benchstream/msgs/77.blk: no space left on device
[1] 2025/12/16 21:21:26.548402 [WRN] Error applying entries to 'ACC > benchstream': write /data/jetstream/ACC/streams/benchstream/msgs/77.blk: no space left on device
[1] 2025/12/16 21:21:26.548409 [ERR] JetStream out of File resources, will be DISABLED
[1] 2025/12/16 21:21:26.549341 [INF] Initiating JetStream Shutdown...
[1] 2025/12/16 21:21:26.550048 [INF] JetStream Shutdown

Sometimes it does not disable jetstream, but resets the stream instead:
Steps to reproduce

1. create 3-node cluster with docker compose and jetstream data volume

    tmpfs:
      - /data:size=1500M

2. Observe in the logs auto-detected max storage:

[1] 2025/12/16 21:12:35.064881 [INF]   Max Storage:     1.10 GB

3. Write 900MB to the jetstream data volume

clush   -R exec -w test-storage-exhaustion-nats-[1-3]-1 'docker exec -t %host  dd if=/dev/zero of=/data/fill bs=1M count=900'

4. Publish until it fills the disk:

nats bench  js  pub sync  test --replicas=3 --msgs=1500 --storage=file --create  --purge  --maxbytes="1100Mb" --size=1000KB
13:23:19 Fatal error from client 0: publishing: publishing synchronously: nats: nats: API error: code=503 err_code=10077 description=expected sequence does not match store
nats: error: publishing: publishing synchronously: nats: nats: API error: code=503 err_code=10077 description=expected sequence does not match store

At this point, it still reports ok:

# curl -s "http://localhost:822{2,3,4}/healthz?js-enabled-only=true"
{"status":"ok"}{"status":"ok"}{"status":"ok"}

Stream is available:

# nats stream ls
╭─────────────────────────────────────────────────────────────────────────────────────╮
│                                       Streams                                       │
├─────────────┬─────────────┬─────────────────────┬──────────┬─────────┬──────────────┤
│ Name        │ Description │ Created             │ Messages │ Size    │ Last Message │
├─────────────┼─────────────┼─────────────────────┼──────────┼─────────┼──────────────┤
│ benchstream │             │ 2025-12-16 13:23:15 │ 613      │ 599 MiB │ 1m35s        │
╰─────────────┴─────────────┴─────────────────────┴──────────┴─────────┴──────────────╯

And data is present:

 # clush   -R exec -w test-storage-exhaustion-nats-[1-3]-1 'docker exec -t %host  du -hsc /data/jetstream/ACC/streams/benchstream'
test-storage-exhaustion-nats-1-1: 599M  /data/jetstream/ACC/streams/benchstream
test-storage-exhaustion-nats-1-1: 599M  total
test-storage-exhaustion-nats-2-1: 599M  /data/jetstream/ACC/streams/benchstream
test-storage-exhaustion-nats-2-1: 599M  total
test-storage-exhaustion-nats-3-1: 599M  /data/jetstream/ACC/streams/benchstream
test-storage-exhaustion-nats-3-1: 599M  total

5. Try to publish one more message:

#  head --bytes=1000 /dev/zero | nats pub -J 'test'
13:26:29 Reading payload from STDIN
13:26:29 Published 1000 bytes to "test"
nats: error: expected stream sequence does not match (10063)

[1] 2025/12/16 21:26:29.487357 [WRN] Error applying entries to 'ACC > benchstream': last sequence mismatch
[1] 2025/12/16 21:26:29.487948 [WRN] Resetting stream cluster state for 'ACC > benchstream'

Jetstream is not disabled at this point:

# curl -s "http://localhost:822{2,3,4}/healthz?js-enabled-only=true"
{"status":"ok"}{"status":"ok"}{"status":"ok"}#
#
# curl -s "http://localhost:822{2,3,4}/healthz"
{"status":"unavailable","error":"JetStream stream 'ACC \u003e benchstream' is not current: stream not found"}{"status":"unavailable","error":"JetStream stream 'ACC \u003e benchstream' is not current: stream not found"}{"status":"unavailable","error":"JetStream stream 'ACC \u003e benchstream' is not current: stream not found"}#