Elasticsearch "flood-stage watermark" and read-only indices: free the disk, then check the block

1. See the disks

curl -s 'localhost:9200/_cat/allocation?v&h=node,disk.percent,disk.used,disk.avail,disk.total,shards'

node    disk.percent disk.used disk.avail disk.total shards
node-1            96     480gb       20gb      500gb    312

Add -u elastic:password and use https:// if security is on, as it is by default since 8.0. Then see who’s using the space on that disk:

curl -s 'localhost:9200/_cat/indices?v&s=store.size:desc&h=index,docs.count,store.size,pri,rep' | head -20
sudo du -xh --max-depth=1 /var | sort -h | tail

If the disk’s filled by something other than Elasticsearch, such as logs, then no space left on device helps you find it.

2. Free some space

Deleting old indices is the quickest way, and it’s allowed under this block:

curl -X DELETE 'localhost:9200/logs-2026.09.*'

Check what a pattern matches before deleting, with _cat/indices/logs-2026.09.*. In 8.x, wildcard deletes may be refused by action.destructive_requires_name, in which case you name them one by one.

Other options:

• Take away replicas from old indices, if you can live with less safety for that data. It halves their size, as in {"index.number_of_replicas": 0}.
• Move shards away by adding a node or a bigger disk. Elasticsearch rebalances by itself.
• Grow the volume, which on the cloud is often the quickest.

Don’t delete files from /var/lib/elasticsearch by hand. Those are shards, and that corrupts the indices.

3. Check the block is gone

Since 7.4, it’s removed automatically when usage falls below the high watermark. So freeing up just a bit won’t do it, as you’ve got to go below 90%. Check:

curl -s 'localhost:9200/_all/_settings/index.blocks.read_only_allow_delete?pretty'

If it’s still there, on an older version or because you can’t wait, remove it:

curl -X PUT 'localhost:9200/_all/_settings' -H 'Content-Type: application/json' \
  -d '{"index.blocks.read_only_allow_delete": null}'

If the disk is still full, it comes back within a short time, since the check runs every 30 seconds by default.

4. Kibana stuck too

Kibana stores its own data in indices, so it may report errors while saving:

FORBIDDEN/12/index read-only / allow delete (api)

That’s the same block on .kibana*. Once there’s space, the same request as in section 3 clears it, if that doesn’t happen by itself. Restart Kibana if it’s still unhappy.

5. Make sure it doesn’t return

Delete old data automatically. Index Lifecycle Management deletes or moves data by age, so you don’t have to remember to:

curl -X PUT 'localhost:9200/_ilm/policy/logs-30d' -H 'Content-Type: application/json' -d '{
  "policy": { "phases": {
    "hot":    { "actions": { "rollover": { "max_age": "1d", "max_primary_shard_size": "50gb" } } },
    "delete": { "min_age": "30d", "actions": { "delete": {} } }
  } }
}'

Then attach it to your index template. On OpenSearch, the same job goes by Index State Management.

On large disks, use absolute values, as percentages leave too much unused:

curl -X PUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "100gb",
    "cluster.routing.allocation.disk.watermark.high": "50gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "20gb"
  }
}'

Absolute values mean free space that must remain, and they have to be used for all three. It’s fine to raise limits a bit for a rare situation, but it only buys time.

Set an alert on disk usage at 80%, so you know before it reaches the low watermark.

6. Docker and single-node

A single node with discovery.type=single-node still follows these limits, and on a laptop a full disk is easy to reach. So check df -h and docker system df.

For a local test only, you can turn the check off:

environment:
  - cluster.routing.allocation.disk.threshold_enabled=false

Never do that in production, where a truly full disk can corrupt data.