{"slug":"self-hosted-app-monitoring-prometheus-vps","url":"https://tako.sh/blog/self-hosted-app-monitoring-prometheus-vps/","canonical":"https://tako.sh/blog/self-hosted-app-monitoring-prometheus-vps/","title":"Self-Hosted App Monitoring with Prometheus: VPS Latency, Cold Starts, and Health","date":"2026-11-02T03:15","description":"Monitor Tako apps on your VPS with Prometheus: scrape local metrics, query request latency, investigate cold starts, and inspect instance health.","author":null,"image":"d7c0252005dc","imageAlt":"A coral octopus watches Prometheus latency charts and adjusts server controls in a retro mission-control room.","headings":[{"depth":2,"slug":"scrape-the-local-metrics-endpoint","text":"Scrape the local metrics endpoint"},{"depth":2,"slug":"read-latency-alongside-request-volume","text":"Read latency alongside request volume"},{"depth":2,"slug":"separate-cold-starts-from-instance-failures","text":"Separate cold starts from instance failures"}],"markdown":"Your app deployed successfully. An hour later, someone says the dashboard feels slow. The process is running, the homepage loads, and the server still answers SSH. That narrows the mystery slightly. It does not tell you where the time went.\n\nTako exposes Prometheus metrics for its proxy and app lifecycle. We can use those to separate backend request latency, cold starts, and instance health, then follow the evidence into logs. We'll start with one VPS and a local Prometheus process. No dashboard template is required to ask useful questions.\n\n## Scrape the local metrics endpoint\n\nOn a normal Tako server, metrics listen on `127.0.0.1:9898`. The [operations reference](/docs/operations/#metrics) lists the metric families; the [listener implementation](https://github.com/tako-sh/tako/blob/master/tako-server/src/proxy/server.rs) confirms that the address is loopback. From a shell on the VPS, check it:\n\n```bash\ncurl -fsS http://127.0.0.1:9898/metrics\n```\n\nLook for `tako_` metric names and their labels. Some series appear only after the corresponding activity happens, so a quiet server can have fewer samples than you expect. Make a request to a real backend route, then inspect the endpoint again. A static file request exercises a different path.\n\nThe server flag `--metrics-port` selects another port; `0` disables the endpoint and request metrics. Changing that port does not make the listener public. Keep your [server management setup](/docs/servers/) separate from monitoring: Tako's private management listener and its local metrics listener serve different purposes.\n\nFor this example, install Prometheus on the same VPS using its [getting started guide](https://prometheus.io/docs/prometheus/latest/getting_started/). Create `prometheus.yml` in a directory where you'll run the Prometheus binary:\n\n```yaml\nglobal:\n  scrape_interval: 15s\n\nscrape_configs:\n  - job_name: tako\n    metrics_path: /metrics\n    static_configs:\n      - targets: [\"127.0.0.1:9898\"]\n```\n\nValidate the configuration and start a local inspection session:\n\n```bash\n./promtool check config prometheus.yml\n./prometheus --config.file=prometheus.yml \\\n  --web.listen-address=127.0.0.1:9090\n```\n\nThese commands assume the downloaded `prometheus` and `promtool` binaries are in the current directory. The [Prometheus configuration reference](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) explains scrape jobs, and its [server flags](https://prometheus.io/docs/prometheus/latest/command-line/prometheus/) document the explicit web-listener binding. For an ongoing installation, supervise Prometheus and choose its storage location as part of your host operations.\n\nIf you are browsing from your laptop, forward the UI through an existing SSH connection:\n\n```bash\nssh -N -L 9090:127.0.0.1:9090 ubuntu@your-vpn-host\n```\n\nReplace the user and VPN hostname with your server's actual administrator connection, then open `http://127.0.0.1:9090` locally. This example assumes port `9090` is free on your laptop.\n\n```d2\ndirection: right\n\nvps: \"Your VPS\" {\n  app: \"App instances\"\n  tako: \"Tako proxy\\nand lifecycle metrics\" {\n    style.fill: \"#E88783\"\n  }\n  metrics: \"127.0.0.1:9898\\n/metrics\"\n  prometheus: \"Prometheus\\n127.0.0.1:9090\" {\n    style.fill: \"#9BC4B6\"\n  }\n  tako -> app: \"backend requests\"\n  tako -> metrics: \"exposes samples\"\n  prometheus -> metrics: \"scrapes\"\n}\n\nlaptop: \"Your laptop\"\nlaptop -> vps.prometheus: \"SSH-forwarded UI\"\n```\n\nThis keeps the scrape in the host's network namespace. A Prometheus container has its own loopback address; the same target configuration there would point inside that container. Our example runs directly on the VPS so the address means exactly what it says.\n\nStart with this query:\n\n```promql\nup{job=\"tako\"}\n```\n\nA value of `1` means the scrape succeeded; `0` means it failed. That is [Prometheus target health](https://prometheus.io/docs/concepts/jobs_instances/), not an application readiness check. A healthy metrics endpoint can describe an unhealthy app.\n\n## Read latency alongside request volume\n\nTako's [metric definitions](https://github.com/tako-sh/tako/blob/master/tako-server/src/metrics.rs) give each family a `server` label. App metrics also carry `app`; copy actual values from the endpoint rather than guessing their spelling. Keep those dimensions when comparing hosts:\n\n| Metric | Type | Useful question |\n| --- | --- | --- |\n| `tako_http_requests_total` | Counter, with `status` class | Is backend request volume or the 5xx rate changing? |\n| `tako_http_request_duration_seconds` | Histogram | How long is the measured proxy request phase? |\n| `tako_upstream_request_duration_seconds` | Histogram | How long until the upstream returns response headers? |\n| `tako_http_active_connections` | Gauge | How much measured app request activity is in flight? |\n\nFor requests per second, grouped by server, app, and response class:\n\n```promql\nsum by (server, app, status) (\n  rate(tako_http_requests_total{job=\"tako\"}[5m])\n)\n```\n\nThe `status` label contains `2xx`, `3xx`, `4xx`, `5xx`, or `other`, rather than individual codes. Add `status=\"5xx\"` to the selector when investigating server errors. Use `rate()` before aggregation so counter resets are handled per series.\n\nFor an estimated p95 request duration in seconds:\n\n```promql\nhistogram_quantile(\n  0.95,\n  sum by (server, app, le) (\n    rate(tako_http_request_duration_seconds_bucket{job=\"tako\"}[5m])\n  )\n)\n```\n\nThe `le` label preserves classic histogram bucket boundaries. To compare upstream timing, run the same expression with `tako_upstream_request_duration_seconds_bucket`. [Prometheus documents this aggregation pattern](https://prometheus.io/docs/prometheus/latest/querying/functions/). Quantiles estimate a distribution; subtracting two p95 values does not calculate per-request proxy overhead.\n\nRead volume before interpreting a percentile. A quiet app may have insufficient observations for a useful five-minute estimate. A missing series or an undefined percentile is not evidence of zero latency.\n\nThere is another important boundary in Tako's current implementation: the request timer starts after backend resolution succeeds. Static responses, ACME requests, unmatched hosts, cold-start waiting, and errors returned before a backend is selected do not pass through that timer. The [proxy request path](https://github.com/tako-sh/tako/blob/master/tako-server/src/proxy/service/mod.rs) makes that scope explicit. These histograms are not full browser navigation timings or a count of every public error response.\n\nIf upstream timing rises with request timing, investigate application work, database access, and external calls. If the two diverge, inspect request handling and response transfer too. Use the measurements to choose the next question, then read logs for the specific request.\n\n## Separate cold starts from instance failures\n\nAn app configured for on-demand operation can have no running instances between visits. The next request waits for startup. That can affect the user's first response even when steady-state backend latency looks excellent. The [scale-to-zero post](/blog/scale-to-zero-without-containers/) explains the lifecycle; here we want to observe it.\n\n| Metric | Meaning |\n| --- | --- |\n| `tako_cold_starts_total` | Recorded cold-start events, including successful and failed completion paths |\n| `tako_cold_start_duration_seconds` | Startup duration observations for those paths |\n| `tako_cold_start_failures_total` | Failures partitioned by `reason` |\n| `tako_instances_running` | Current running instance count per app |\n| `tako_instance_health` | Health per app instance: `1` healthy, `0` unhealthy |\n\nInspect recent startup failures by reason:\n\n```promql\nsum by (server, app, reason) (\n  increase(tako_cold_start_failures_total{job=\"tako\"}[15m])\n)\n```\n\nThen inspect current state directly:\n\n```promql\ntako_instances_running{job=\"tako\"}\n```\n\n```promql\ntako_instance_health{job=\"tako\"}\n```\n\nWith Prometheus's default label handling, the health metric's exported app-instance label becomes `exported_instance`, because Prometheus supplies its own scrape-target `instance` label. The `server` and `app` labels still distinguish the application. Look at the labels returned by your actual scrape before writing selectors.\n\nZero running instances can be expected in on-demand mode. An unhealthy instance needs investigation, while a removed instance's health series is removed rather than permanently left at zero. Check the desired operating mode and current state together:\n\n```bash\ntako status --json\ntako logs --env production --days 1 --json\n```\n\nRun those from the relevant project and inspect failures or truncation. Correlate the event with [release history and deployment state](/docs/deployment/), then reproduce the failing public request. Metrics locate the symptom; the [operations guide](/docs/operations/) connects it to finite logs and verification.\n\nStart with a successful scrape, a real backend request, and one latency query. Add lifecycle metrics as you investigate startup behavior. Your first useful monitoring setup is the one that helps explain the next slow request."}