The sar command in Linux: historical system performance

The sar command in Linux: historical system performance

The sar command (System Activity Reporter) is part of the sysstat package and allows collecting, reporting, and saving information about system activity over time. Unlike tools that show an instant, sar stores historical data that can be queried later to identify trends, bottlenecks, or sporadic problems.

Sar was originally developed by Sebastien Godard and is distributed under the GPL license. It is included in most Linux distributions via the sysstat package, which also provides the sa1 and sa2 daemons responsible for periodic sampling and storage in binary files under /var/log/sa/.

On Debian/Ubuntu-based distributions it is installed with sudo apt-get install sysstat. On RHEL, CentOS, or Fedora use sudo dnf install sysstat or yum. After installation, the file /etc/default/sysstat allows enabling collection by setting ENABLED=”true” and defining the sampling interval (default 10 minutes).

The general format is sar [options] [interval [count]]. Without arguments, sar displays data for the current day. The most common parameters include -u for CPU, -r for memory, -B for paging, -n for network, -d for disks, and -A to obtain all available reports.

• -u: shows percentage of CPU time in user, nice, system, iowait, stolen, and idle modes.
• -r: reports free and used memory, as well as buffers and cache.
• -B: details paging and swap activities, including pgpgin/pgpgout and fault/s.
• -n: with suboptions such as DEV for interface traffic, EERROR for errors, and FCOLL for collisions.
• -d: block device statistics, such as tps, kB_read/s, and kB_wrtn/s.
• -A: equivalent to specifying all the previous options, useful for a complete overview.

To observe CPU utilization every two seconds for ten iterations, run sar -u 2 10. The output shows columns like %user, %nice, %system, %iowait, %steal, and %idle, allowing detection of consumption spikes or I/O waits.

Each line corresponds to a sampling interval; the values are averages over that period. A sustained high %iowait indicates the CPU is waiting for disk operations, while an elevated %system may signal excessive system call or interrupt activity.

The sa1 daemon runs via cron every 10 minutes (configurable) and saves binary data in /var/log/sa/saDD, where DD is the day of the month. Subsequently, sa2 generates a daily summary in readable format and stores it in the same directory, facilitating monthly report generation.

In /etc/cron.d/sysstat is the task that calls /usr/lib/sysstat/sa1 1 1 to take samples and /usr/lib/sysstat/sa2 6 0 0 to create the summary. Modifying these files allows changing the frequency or retaining more days of history according to auditing needs.

The binary files can be read again with sar -f /var/log/sa/saDD to obtain the report for a specific day. Tools such as ksar (KDE System Activity Reporter) or scripts with gnuplot transform those data into trend graphs, facilitating presentation to operations teams or management.

It is recommended to adjust the sampling interval according to load: on critical production servers a 30-second interval provides greater detail, while on less demanding systems 5-10 minutes reduces disk usage. Moreover, sar does not capture sporadic events shorter than an interval, so it is complemented with tools like perf or eBPF for finer analysis.

To analyze disk I/O bottlenecks, sar -d can be combined with sar -u and observe if %iowait increases while %user and %system remain low. A typical example is running sar -d 1 5 to see transfers per second (tps) and kilobytes read and written. Likewise, sar -n DEV shows traffic for each network interface; if collisions (FCOLL) or errors (EERROR) are detected as high, it could indicate cabling or duplex configuration issues. These metric crossings allow identifying whether the limitation stems from storage, network, or the CPU itself.

Many production environments integrate sar with continuous monitoring systems to create historical dashboards. The binary data from sa1 can be exported to CSV using sar -A -f /var/log/sa/saDD > metrics.csv and then ingested by tools like Prometheus via a custom exporter or via scripts that run sar each minute and push the values to a Pushgateway. In Grafana, panels can be created showing the evolution of CPU, memory, and network usage over weeks, facilitating detection of growth trends and capacity planning based on evidence.

The sar command constitutes an essential tool for any Linux administrator who needs to understand the system’s historical behavior. Its ability to store and query CPU, memory, I/O, and network metrics over days or months enables capacity analysis, upgrade planning, and proactive anomaly detection.

This post is also available in ESPAÑOL.