Observability (add-on)
Per-minute metric charts, the processes using the most CPU and RAM, and slow or failing site endpoints. The data is recorded and stays on your own server.
Observability is off by default on every account. Your server records nothing until the add-on is enabled. It is free while billing doesn't exist yet. To enable it, contact us through the Enterprise button and mention "Observability add-on"; a Saka admin turns it on per account.
What it is for
The Readiness card and threshold alerts tell you something is wrong. Observability answers why: which process pushed CPU up at 03:12, whether swap started filling before the server slowed down, which endpoint returns the most 5xx errors. It suits servers that host many sites or apps and need investigating, not just up/down monitoring.
What is recorded
| Data | Contents | Interval |
|---|---|---|
| Host metrics | CPU, I/O wait, 1/5/15 minute load average, RAM used, swap used, network in and out, disk read and write, established TCP connections | Every minute |
| Top processes | The 5 biggest processes by CPU and 5 by RAM, with the command line (truncated) and the Docker container name when the process runs in a container | Every minute |
| Site endpoints | From the access log of the web server for sites Saka manages: request count, 4xx and 5xx counts, average duration, p95, and slowest duration per endpoint | Summarized every 5 minutes |
Endpoint paths are normalized so thousands of different addresses count as one: GET /orders/12345?ref=abc and GET /orders/67890 both become GET /orders/:id. Segments that are numbers or long codes are replaced with :id, and everything after a question mark is dropped so tokens or personal data in query strings are never recorded.
Where the data lives
On your server, in the agent's data folder, not at Saka. The panel only reads a summary when you open the card. Data is kept for 7 days; older data deletes itself. If the add-on is turned off, the agent stops recording at once and the remaining data deletes itself after 7 days. Removing the agent with Release a server removes it too.
How light it is
- One read of
/procper minute, plus one scan of the process list. - One small text file per hour (a few megabytes for a week), plus one endpoint summary file per day.
- No code is injected into your apps or sites, no library to install, and no extra process: the recorder runs inside the agent that is already there.
- The access log it uses is already read by the agent for other features, so there is no second read of the log.
- The panel does no polling. Data is fetched when you open the card, change the range, or press Refresh.
Using the Observability menu
Once the add-on is enabled for your account, an Observability menu appears in the top bar (it is hidden while the add-on is off). Pick a server at the top of the page. On a server page, the Observability card also has a button that opens this menu for that server.
- Pick a range from 1 hour to 7 days. Charts are condensed to at most about 240 points.
- Hover (or drag a finger) over any chart. A marker line appears on all charts and the Processes at ... table shows the biggest processes at that moment. It defaults to the moment of highest CPU, so the cause of a spike is visible right away.
- The Top CPU processes over the range table sums up who used CPU most often.
- The Site endpoints table can be sorted by slowest (p95), most 5xx errors, or most requested, and filtered by domain.
After the add-on is enabled, charts need a few minutes to fill in because the agent only starts recording then.
Errors, log search, and alerts
Three more tabs in the same menu:
- Errors: error-level log lines from running containers and the system journal, grouped. Numbers, IDs, and timestamps are ignored, so the same error counts once, with when it was first and last seen. Collected every 5 minutes while the add-on is on.
- Search logs: searches the system journal or one container's log live, by keyword or regex, level, and a range up to 3 days. No index is stored, so it costs nothing while unused. Searches are limited to 20 seconds and 16 MB, and only sources from the list can be chosen.
- Alerts: rules such as server CPU, I/O wait, swap, load per core, TCP connections, one process's CPU (e.g.
mysqld), site p95, or site 5xx ratio held for N minutes. The agent evaluates them every minute and notifies your Telegram, at most once an hour per rule, then sends a recovery notice after 3 normal minutes. Site metrics are judged per 5 minutes and need at least 10 to 20 requests.
PHP file monitoring
The PHP files tab records PHP files (plus .htaccess and .user.ini) that are new, changed, or deleted in web folders. Folders in the usual places are found automatically (Saka sites, /var/www, public_html, aaPanel), and you can add others, e.g. /opt/whmcs. Each event stores the time, size, modification time, and the SHA-256 hash of the content, so you can match it against the access log to see where the file came in from.
- Suspicious: shell-like content (e.g.
eval(base64_decode(...))or commands from visitor input), PHP in upload or cache folders, and an.htaccessthat makes images or text run as PHP. These go to your Telegram right away, at most once per 10 minutes. - Ordinary changes: notified only when few (5 files or fewer). Bulk changes such as plugin updates or deploys are recorded without a notice, so Telegram is not flooded.
- Light: only file names and the folder listing are read, compared to the previous list (size and modification time). Content is read only for new or changed files. A scan runs every 10 minutes, gently, capped at 2 minutes. Framework caches (Laravel, Symfony) and
node_modulesare skipped. - Archive: events are archived to S3 when the archive is on, so the history outlives the 7 days.
Changes are recognized from size and modification time. An attacker who forges the modification time and keeps the size the same is not detected. The first scan only builds a baseline, so files that existed before monitoring was turned on are treated as normal (an old shell is not found; use a separate malware scanner for that).
S3 archive and incident investigation
The S3 archive tab keeps Observability data longer than 7 days in an S3 bucket you own, the same destination as Backup to S3 (IDCloudHost Object Storage, Amazon S3, Cloudflare R2, and other compatible providers). Saka does not provide the storage; the bucket and its cost are on your S3 account. Contents are encrypted with a key that exists only on your server, so the S3 provider cannot read them, and objects end in .enc so they never mix with site backup archives.
- What is archived: metric, endpoint, and error files once their hour has passed. Raw logs only if you choose them: errors and warnings (recommended) or all logs. Sources are the system journal, SSH logins, web access of all Saka sites, nginx, Apache, hosting-panel (aaPanel, cPanel) and PHP logs on that server, and running containers, hourly. Web server log files are found automatically in the usual locations and only new lines are read, starting from when the archive is turned on.
- Retention: you decide: 30, 90, 180, 365 days, any number of days (1 to 3650), or no automatic deletion. Objects older than the retention are deleted automatically, checked every 6 hours. The server disk still keeps only 7 days.
- Data note: application logs often contain emails, tokens, or visitor IPs and can be large. The bucket is yours, so the choice is yours. The default is errors and warnings only, and in that mode web access keeps only 4xx and 5xx statuses.
Investigating an incident. For example a site was hacked and you need to know which IP the attacker came from and which addresses they opened. In the Search logs tab, tick S3 archive, pick the Web server source (for sites outside Saka, such as WHMCS on nginx or Apache), Web access (Saka sites), or SSH logins (or all sources), choose a range of up to 31 days ending on a given date, and search for an IP or an address (e.g. wp-login.php or /shell). For a complete web access trail, pick All logs before an incident happens: the archive only holds what was recorded since it was turned on, so it cannot look back before that. The Errors tab can also use 30 and 90 day ranges from the archive.
Slow queries and scripts
The Slow tab answers "why is my site slow" by reading logs that already exist on the server: the MySQL/MariaDB slow query log and the PHP-FPM slowlog. Queries are grouped after their values are replaced with question marks, so a thousand WHERE id = 42 with different numbers become one row with the count, total time, slowest run, and rows examined. PHP-FPM traces are grouped by script and the function where they stopped.
Saka does not change your database or PHP configuration. When a log is off, the tab shows how to turn it on (slow_query_log for MySQL, request_slowlog_timeout and slowlog for PHP-FPM). Log locations are found automatically in the usual places and from the PHP-FPM pool files.
Events on the charts
Coloured dashed lines on the Overview charts mark reboots, Saka agent updates, system package changes (apt, dnf, yum), PHP file changes, and alerts that fired. The list under the charts holds the same events; click one to jump to its time. The point is to answer "what changed right before this spike".
History beyond 7 days
When the S3 archive is on, the Overview range can be 14 and 31 days. Each finished day is summarised into 5-minute averages and uploaded (encrypted) to your bucket; the part of a chart older than 7 days is read from there. Process lists and the endpoint table stay limited to the last 7 days. Without the S3 archive the range is capped at 7 days. Days before the archive was turned on have no history.
Anomaly alerts
Besides fixed thresholds, alert rules can be habit-based: CPU, TCP connections, or network traffic compared with the average of the same hour on previous days. The threshold is a multiple (for example 3 times usual) plus a minimum value so small spikes trigger nothing. A server that is busy every morning does not alarm, but a spike in an hour that is normally quiet does. At least 3 days of data are needed before an anomaly rule can fire.
Saved searches and live follow
In the Search logs tab, Save this search stores the source, search text, level, and range under a name (up to 30 per server), for example "failed SSH logins" or "5xx on domain X", so the next investigation is a pick from the list. Follow live reloads the results every 10 seconds for the last hour and stops when you leave the tab.
Limits
Observability does not follow a single request across several services (distributed tracing) and does not inject code into your app. It is the view from the server and web server side.
- Duration is the time the server took to process the request, not network time or load time on the visitor's device.
- p95 is the upper bound of a duration bucket (e.g. 100 ms, 250 ms, 500 ms), so it is an estimate, not an exact number.
- Endpoints are counted only for sites Saka manages on that server. Other apps on the server that don't go through Saka's site web server show up in host metrics and the process list, but not in the endpoint table.
- Endpoints are counted per domain. If one domain has hundreds of distinct paths within 5 minutes, the excess paths are merged as
(lainnya). - History is 7 days and only from when the add-on was enabled.
Related
CPU, RAM, and disk alerts with Telegram notifications are set on the thresholds card of the server page. Basic per-site traffic summaries are on the site page without the add-on. How Saka treats your data: Security & privacy.