From 232b4b9f32339043a9b08d2c1881817d85f2e5d2 Mon Sep 17 00:00:00 2001 From: Obsidian-MBPM4 Date: Tue, 17 Mar 2026 15:03:57 +0100 Subject: [PATCH] vault backup: 2026-03-17 15:03:57 Affected files: .obsidian/workspace.json 2 Personal/Home Lab/Baerhalten/Home Assistant -> InfluxDB -> Grafana setup and debugging notes.md 2 Personal/Home Lab/Baerhalten/Home Assistant.md --- .obsidian/workspace.json | 4 +- ...DB -> Grafana setup and debugging notes.md | 400 ++++++++++++++++++ .../Home Lab/Baerhalten/Home Assistant.md | 3 + 3 files changed, 405 insertions(+), 2 deletions(-) create mode 100644 2 Personal/Home Lab/Baerhalten/Home Assistant -> InfluxDB -> Grafana setup and debugging notes.md diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index 83516e6..39f4f0e 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -491,8 +491,9 @@ }, "active": "b948e87b03ca1ad1", "lastOpenFiles": [ - "2 Personal/Lists/Business Ideas.md", + "2 Personal/Home Lab/Baerhalten/Home Assistant -> InfluxDB -> Grafana setup and debugging notes.md", "2 Personal/Home Lab/Baerhalten/Home Assistant.md", + "2 Personal/Lists/Business Ideas.md", "2 Personal/Home Lab/NextiShareBot.md", "0 Journal/0 Daily/2026-01-29.md", "Temporary/Untitled 4.md", @@ -519,7 +520,6 @@ "2 Personal/Lists/Media/Bücher.md", "Attachments/ESPSomfyRTS 2026-01-18T16_26_16.backup", "Attachments/Pasted image 20260118150817.png", - "Temporary/Untitled 3.md", "Attachments/Pasted image 20251202214228.png", "2 Personal/1 Skills/AI", "2 Personal/Home Lab/Baerhalten", diff --git a/2 Personal/Home Lab/Baerhalten/Home Assistant -> InfluxDB -> Grafana setup and debugging notes.md b/2 Personal/Home Lab/Baerhalten/Home Assistant -> InfluxDB -> Grafana setup and debugging notes.md new file mode 100644 index 0000000..e52acff --- /dev/null +++ b/2 Personal/Home Lab/Baerhalten/Home Assistant -> InfluxDB -> Grafana setup and debugging notes.md @@ -0,0 +1,400 @@ +--- +title: Home Assistant -> InfluxDB -> Grafana setup and debugging notes +created_date: 2026-03-17 +updated_date: 2026-03-17 +aliases: +tags: +--- +# Home Assistant -> InfluxDB -> Grafana setup and debugging notes +## Architecture + +This setup has **three layers**: + +1. **Home Assistant** generates state changes and sensor values. +2. **InfluxDB 3 Core** stores that time-series data. +3. **Grafana** reads from InfluxDB and visualizes it. + +### Network layout + +- **Home Assistant** runs outside the Docker Compose network. +- **InfluxDB** and **Grafana** run together in the same Docker Compose stack. +- Therefore: + - **Home Assistant -> InfluxDB** must use the **Proxmox container IP** and exposed port. + - **Grafana -> InfluxDB** should use the **Docker service name**. + +## Actual endpoints + +### Home Assistant -> InfluxDB +Home Assistant should connect to InfluxDB using: + +- Protocol: `http` +- Host / URL: `192.168.194.120` +- Port: `8181` +- SSL verification: off + +Reason: Home Assistant is outside the Docker network, so it cannot resolve `influxdb3`. + +### Grafana -> InfluxDB +Grafana should connect to InfluxDB using: + +- URL: `http://influxdb3:8181` + +Reason: Grafana and InfluxDB are on the same Docker Compose network, so Docker DNS resolves `influxdb3`. + +## Docker Compose setup + +The relevant Compose structure is: + +```yaml +services: + influxdb3: + image: influxdb:3-core + ports: + - "8181:8181" + + grafana: + image: grafana/grafana:latest + depends_on: + - influxdb3 + ports: + - "3000:3000" +``` + +This means: + +- InfluxDB is reachable from outside Docker at `http://:8181` +- InfluxDB is reachable from Grafana internally at `http://influxdb3:8181` + +## Home Assistant InfluxDB configuration + +### Important migration detail + +Home Assistant is removing YAML-based InfluxDB **connection settings**. + +So these connection keys should **not** stay in YAML: + +- `api_version` +- `host` +- `port` +- `ssl` +- `verify_ssl` +- `ssl_ca_cert` +- `username` +- `password` +- `database` +- `token` +- `organization` +- `bucket` +- `path` + +Those should be configured through **Devices & Services** in the Home Assistant UI. + +### What can still stay in YAML + +Behavior/settings like these can still stay in YAML: + +- `measurement_attr` +- `default_measurement` +- `override_measurement` +- `include` +- `exclude` +- `tags` +- `tags_attributes` + +## Final Home Assistant behavior setting + +To get **one table per entity/sensor**, the key YAML setting is: + +```yaml +influxdb: + measurement_attr: entity_id +``` + +Reason: + +- Default behavior grouped data into tables like `W`, `Wh`, `V`, `%` +- That made many sensors appear "missing", because they were grouped by unit +- `measurement_attr: entity_id` changes that back to one measurement/table per entity, which is easier to use in Grafana + +## InfluxDB concepts in this setup + +### Databases +We used databases such as: + +- `home` +- `homeassistant` +- `ha_fresh` + +These are databases, not tables. + +### Tables / measurements +Inside a database, tables are created automatically when data is written. + +Examples after using `measurement_attr: entity_id`: + +- `sensor.solarnet_power_photovoltaics` +- `sensor.solarnet_power_grid_export` +- `sensor.solarnet_power_load_consumed` + +Examples from the old grouped schema: + +- `W` +- `Wh` +- `V` +- `A` + +## Tokens and authentication + +### What is used where + +- **Home Assistant** uses an InfluxDB token to write data +- **Grafana** uses an InfluxDB token to query data +- **Admin token** is used for CLI/database management + +### Important note +Tokens and secrets that were exposed should be rotated. + +## How we debugged the setup + +## 1. Verified InfluxDB is running + +We checked Docker: + +```bash +docker ps +docker logs --since 0.5m influxdb3 +``` + +This confirmed: +- Grafana container is running +- InfluxDB container is running +- InfluxDB is actively flushing writes + +## 2. Verified databases exist + +```bash +docker exec -it influxdb3 influxdb3 show databases --token "$INFLUX_ADMIN_TOKEN" +``` + +This showed databases like: + +- `_internal` +- `home` +- `homeassistant` +- later `ha_fresh` + +## 3. Verified Home Assistant can reach InfluxDB over the network + +From Home Assistant shell: + +```bash +curl http://192.168.194.120:8181/health +``` + +Response: + +```text +{"error": "the request was not authenticated"} +``` + +This was **good** because it proved: +- DNS/IP was correct +- network path worked +- InfluxDB was reachable +- only authentication remained + +## 4. Reconfigured Home Assistant in the UI + +Home Assistant InfluxDB integration was configured in **Devices & Services** using: + +- HTTP +- host/IP of Proxmox container +- port `8181` +- no SSL verification +- organization `dummy` +- bucket/database `ha_fresh` or similar +- token + +## 5. Enabled debug logs in Home Assistant + +We confirmed HA was writing by seeing log lines like: + +```text +Wrote 8 events. +Wrote 10 events. +``` + +That proved: + +- Home Assistant -> InfluxDB write path works + +## 6. Verified writes actually landed in InfluxDB + +We queried InfluxDB directly: + +```bash +docker exec -it influxdb3 influxdb3 query --database ha_fresh --token "$INFLUX_ADMIN_TOKEN" 'SELECT * FROM "W" ORDER BY time DESC LIMIT 20' +``` + +This showed fresh rows from Home Assistant sensors, including: + +- `pv_power` +- `solarnet_power_photovoltaics` +- `solarnet_power_load_consumed` +- `solarnet_power_grid_export` + +That proved: + +- Home Assistant -> InfluxDB works fully +- the issue was not ingestion +- the issue was schema / Grafana config + +## 7. Verified Grafana container can reach InfluxDB + +From inside the Grafana container, we tested: + +```bash +curl -i http://influxdb3:8181/api/v3/query_sql -H "Authorization: Token YOUR_GRAFANA_TOKEN" -H "Content-Type: application/json" --data '{"db":"home","q":"SHOW TABLES"}' +``` + +This working proved: + +- Docker networking is fine +- `influxdb3` resolves correctly +- Grafana-side token auth works +- the remaining problem was purely Grafana datasource configuration + +## 8. Fixed Grafana datasource setup + +### Correct Grafana datasource basics + +- Query language: **SQL** or the desired mode +- URL: `http://influxdb3:8181` +- Database: the actual InfluxDB database in use +- Token: Grafana token + +### Important TLS issue + +Grafana originally failed with an error like: + +```text +tls: first record does not look like a TLS handshake +``` + +That happened because Grafana tried TLS/FlightSQL behavior against a plain HTTP endpoint. + +Fix: +- use the correct datasource mode +- use the right endpoint +- keep the connection consistent with the actual InfluxDB setup + +## 9. Realized the schema had changed + +Old schema assumption: +- one table per sensor + +New schema that appeared initially: +- one table per **unit**, like `W`, `Wh`, `V` + +That is why old queries stopped working. + +Example old query: + +```sql +SELECT mean("value") +FROM "sensor.solarnet_power_photovoltaics" +WHERE $timeFilter +GROUP BY time(5m) fill(none) +``` + +Equivalent query in the grouped-by-unit schema: + +```sql +SELECT mean("value") +FROM "W" +WHERE + "entity_id" = 'solarnet_power_photovoltaics' + AND $timeFilter +GROUP BY time(5m) fill(none) +``` + +Then we changed HA back to per-entity measurements using: + +```yaml +influxdb: + measurement_attr: entity_id +``` + +## 10. Reset the fresh database to keep it clean + +After confirming the new schema worked, the goal was to keep only actually-used tables in the fresh DB. + +General approach: +- delete or recreate `ha_fresh` +- reconnect HA to it +- let HA repopulate it with only actively written entities + +That leaves a clean database without old junk. + +## Final recommended setup + +## Home Assistant +- Configure InfluxDB connection in the UI +- Keep only behavior options in YAML +- Use: + +```yaml +influxdb: + measurement_attr: entity_id +``` + +if per-entity tables are desired + +## InfluxDB +- Keep one clean database for HA data +- Use separate tokens for: + - admin + - Home Assistant + - Grafana + +## Grafana +- Use datasource URL: + +```text +http://influxdb3:8181 +``` + +- Point it to the correct InfluxDB database +- Use the Grafana token +- Rebuild old queries if schema changed + +## Quick troubleshooting checklist for the future + +### If HA is not writing +1. Check HA logs for InfluxDB errors +2. Test connectivity from HA: + ```bash + curl http://:8181/health + ``` +3. Verify token / database / organization / bucket in HA UI +4. Query InfluxDB directly to see whether data is arriving + +### If Grafana shows nothing +1. Verify datasource URL is `http://influxdb3:8181` +2. Test token from inside Grafana container +3. Confirm the right database is selected +4. Check whether schema is per-entity or grouped-by-unit +5. Rewrite queries accordingly + +### If sensors seem missing +1. Check `SHOW TABLES` +2. Query likely shared tables like `W`, `Wh`, `V` +3. Check whether HA is grouping by unit instead of entity +4. Set: + +```yaml +influxdb: + measurement_attr: entity_id +``` + +if one table per sensor is preferred diff --git a/2 Personal/Home Lab/Baerhalten/Home Assistant.md b/2 Personal/Home Lab/Baerhalten/Home Assistant.md index 2077da6..701338d 100644 --- a/2 Personal/Home Lab/Baerhalten/Home Assistant.md +++ b/2 Personal/Home Lab/Baerhalten/Home Assistant.md @@ -61,3 +61,6 @@ I put the tokens into chat gpt so they might be compromised. all Influx tokens ( ## Backup Ich hab ein Back-up User kreiert und einen geteilten Ordner auf der Nase bei meinen Eltern zu Hause. Home Assistant macht jeden Tag ein automatisches Back-up und behält dabei fünf Back-ups in Rotation. Die Verschlüsselung Schlüssel sind in meinem Lastpass. + + +