41 lines
1.1 KiB
Markdown
41 lines
1.1 KiB
Markdown
# Home Lab Incident Reports
|
|
|
|
---
|
|
|
|
## 2026-04-03 — Immich public outage via VPS OOM
|
|
|
|
### What happened
|
|
- Immich still worked internally in the homelab
|
|
- Public access failed first with HTTP 500, later with 502
|
|
- Pangolin had been OOM-killed on the VPS
|
|
- After that, Traefik could no longer resolve pangolin via Docker internal DNS, so it could not fetch dynamic config
|
|
|
|
### Evidence
|
|
```bash
|
|
# Check for OOM events
|
|
sudo dmesg -T | grep -i -E 'oom|out of memory|killed process'
|
|
|
|
# Pangolin log showed SIGKILL
|
|
docker logs pangolin
|
|
|
|
# Traefik log showed:
|
|
# Get "http://pangolin:3001/api/v1/traefik-config"
|
|
# lookup pangolin on 127.0.0.11:53
|
|
# read: connection refused
|
|
```
|
|
|
|
### Root cause
|
|
- **Primary cause:** VPS memory exhaustion
|
|
- **Secondary cause:** broken Docker service discovery / network state after the OOM event
|
|
|
|
### Fix
|
|
```bash
|
|
docker compose down
|
|
docker compose up -d
|
|
```
|
|
|
|
### Follow-up actions
|
|
- [ ] Add swap to VPS to prevent OOM cascade
|
|
- [ ] Add memory monitoring/alerting on VPS
|
|
- [ ] Consider adding Traefik health-check/config-refresh cron job as a resilience measure
|