The quick resolution of problems is a fundamental activity for any sysadmin. Let’s discover some tricks and advices to become better and faster at diagnosing and resolving troubles of a Linux system.

In the last issues of GURU advisor we dedicated an article to those commands suitable for a basic Linux troubleshooting in a VPS context, but when top, ps and netstat are not enough, we must aim to more advanced tools. Troubleshooting is an activity that can easily transform into a waste of time and resources, proportional to the difficulties of the problems to be solved. In order to optimize the results and limit unuseful trials and tests it’s mandatory to act following well-structured procedures.

2000px Webpage icon powered by linux.svg

The USE method -which stands for Utilization, Saturation and Errors- was originally conceived by Brendan Gregg (who wrote an omniscient book on the topic: Systems Performance - Prentice Hall, 2013) and has its basis in a simple and efficient idea: defining a workflows that determines the level of utilization, saturation and possible errors for each available resource. In this way we can gradually reduce the number of possible causes of the misbehaviour until we can precisely identify the cause of the loss of performance.
Defining resources is quite easy: they are the physical components of a machine (we can apply the method to software components, but it would get very complicated and detailed -- perhaps a bit too overkill given the scenario), a server in our case -and the equivalent virtual version. Processor, memory, disk, controllers, network interfaces, buses, etc..
Naturally the better the knowledge of the architecture, and the faster and more precise the results will be, albeit implying a certain complexity and a steep learning curve.

Usually the analysis of hardware macro-elements (processor, memory, etc. without going into details) and the exclusion of the software environment, given its additional difficulties, it’s enough to put the USE method into practice.

The definition of utilization, saturation and error is quite subtle, however we can provide some standard definitions and state that: utilization means the proportion of use of a resource, once we set the edges at 0% (the resource is not in use) and 100% (the resource cannot accept any work without adding latency or queue), saturation means the degree of work the resource can’t execute (so it involves latency and queue. Note how utilization and saturation are strictly bounded). For the definition of error, we can refer to the common mean of the term.

Available tools per resource


As the USE method teaches, a proper problem identification and solving job must be based on the analysis of the state of the different system resources by means of specific tools. An overall observation is useful only in the initial phase to have an idea of the steps to follow.
As an Operating System, Linux natively provides several diagnostics tools as command line commands that can be coupled with other tools, like the ones of the sysstat package that we are about to investigate. A detailed installation procedure and the availability depends on the distro in use.

uptime [Load Average]

uptime2

uptime is a tool that provides information about load average (that is, the average computational work performed by the system) in the last 1, 5 and 15 minutes. Load average considers both processes in runnable state (which use CPU or are waiting to be elaborated) and non-interruptable state (processes waiting for and I/O access, like disk reading data). A load average may be high even with an unloaded processor, for instance when storage is a bottleneck (slow network share, faulty disk, USB peripherals with scarce performances, etc..) and leaves several processes in uninterruptable sleep state waiting for the peripheral.
The number that is showed must be compared to the number of cores available to the system: a value lower than the number of cores indicates a situation without problems (the system can manage the work load without effort), but a value greater that the number of cores indicate a saturation or overload state (that can be, more or less, temporary).
In the practical example shown here, the load average is measured during a stress test with a dedicated tool: in the last minute is 42, which means that the system, a four core server, would need 38 additional cores to execute the work load while keeping the system not overloaded. Once the test is over, the machine shows a way lower value: 0.13, it’s almost in idle state.

dmesg -T | tail [Kernel errors]

dmesg

dmesg is the command that shows the kernel buffer, which contains information on errors. By concatenating it with the tail command, we can easily consult the last 10 system messages (peripherals, errors, interrupts, etc..) which could offer a clue on why the system has a loss of performances.

vmstat [Virtual memory and detailed information]

vmstat

The two previous command show generic results about the state of the system, vmstat is more detailed: it offers information about processes, memory, paging, block I/O, disk and CPU activity. The command run without any argument shows information collected since last machine’s boot, while adding the parameter 1 refreshes information every 1 second. Interrupt the screen output by pressing ctrl+C.
Visualization is pretty schematic, the output is organized into groups of columns, each regarding a specific area of the system (processes, memory, swap, I/O, etc..), while the columns to consider are:

  • r: the number of active and queued process for each core, a better indicator than load averages found with uptime as it doesn’t include I/O requests. An ‘r’ number greater than the number of cores indicates saturation.
  • free: it’s the free (idle) memory in kilobytes. The free command provides additional details.
  • si, so: swap in and swap out, respectively, if different from zero mean that memory is swapping to disk.
  • us, sy, id, wa, st: there parameters are related to CPU time, as an average on all available CPUs.
    They refer, respectively, to user time (time spent executing code not from kernel), system time (time spent executing kernel code), idle time (time spent waiting for an IO request) and stolen time (time ‘stolen’ by a virtual machine).

The sum of the us+sy times indicates whether a CPU is overloaded or not, a constant ‘wa’ number indicates a disk bottleneck: in this case the processor is at rest because tasks are waiting for a disk IO operation that isn’t coming. In order to obtain more detailed information of the different areas of the system, additional commands can be used.

mpstat -P ALL 1 [CPU state]

mpstat

The mpstat command integrates and completes the information about processors by providing detailed data on the activities of each available core (option -P ALL, as a momentary snapshot or each N seconds if a numeric argument is provided). The fields to look for are usr (execution percentage per user), sys (percentage on a kernel basis), iowait (idle time spent waiting for I/O requests), guest (time spent for virtual processes) and idle (time that processor was idle and system didn’t have any disk I/O request).

pidstat [processes]

pidstat

pidstat provides information on active processes managed by the kernel, the parameter 1 -as it happens with other commands- states the amount of time between a report and the following one. The output resembles top’s, but the visualization is structured in momentary tabs instead of a continuous list of reports. In this way we can investigate on patterns that repeat through time, in addition to note results with copy-and-paste.

iostat -xz [Disk I/O]

 

iostat

iostat provides information about block devices (disks), useful to understand work loads and performances derived to I/O systems. In the upper part of the output we can find parameter related to the processor, but the area that is of our interest is the next one with the following columns:

  • r/s, w/s rkB/s, wkB/s: values about reading and writing operations, Kbyte read and writes per second. They are useful indicators of the work load. Bad performances may depend upon an excessive load.
  • await: it’s the time that penalizes an application and includes both queue time and usage time. Values greater than the expected indicate saturation or faults.
  • avgqu-sz: it indicates the average requests sent to the device. Value greater than 1 might indicate saturation, while we must keep in mind that some devices can operates with parallel requests, in particular virtual devices that leverage several back-end disks.
  • %util: it’s the device usage percentage. A value greater than 60 is a signal of bad performances, 100 indicates saturation, like in the case of a single I/O process continuously accessing the same resource.

The -x argument is used to show extended information, while the -z option excludes devices with no activity in order to offer a better and simpler output consultation. iostat doesn’t work with OpenVZ/Virtuozzo containers.

free –m [memory]

free

The free command shows several information about the usage state of RAM memory, the -m parameter specified the unit of measure in Megabytes.
Be careful because a superficial analysis of results may be lead to a wrong interpretation. The values of used and free -which are the headers of columns- can be misleading as they indicate, in the case of used memory, the total amount of memory actually used and the memory reserved as buffer for I/O operations. Buffer reserved memory is, indeed, actually available to the system, albeit being “pre-allocated”. Actual usage values are found in the second row, the one identified by the -/+ buffers/cache field.

To make a practical example, an high value of buffers indicates problems on a I/O level (which can be further investigated with the iostat command9. The value used in the Swap row shows if and how much the system is swapping memory on disk.
Some particular situations, like systems with the ZFS file system, show even more unexpected results as they implement a RAM caching system.

sar -n DEV 1 [Network I/O]

sarDEV

Up to now we’ve dealt with tools that allow to control computing resources, mainly processor, disk and memory, but we often need to evaluate the working state of network connections, and the sar command comes handy. This tool shows statistics on throughput of network interfaces: rxpck/s and txpck/s (received and sent packets per second), rxkB/s and txkB/s (sent and received kilobytes per second), %iful (use percentage of the interface). This last parameter shows the sum of sent and received kB per second, expressed as a percentage on the interface’s speed in case of half-duplex. In case of full-duplex, the parameter to consider is the greater between rxkB/s and trkB/s.

sar -n TCP,ETCP 1 [TCP connections]

sarTCP

The TCP and ETCP (ErrorTCP) options offer metrics about TCP connections that are started per second (active/s), accepted (passive/s) and retransmissions per second (retrans/s). The number of started/accepted connections offer a gross measure of the server’s usage degree, and a high value of retrans means a server-side or network problem: for instance a situation where the server is overloaded and doesn’t accept any more packets, or the Internet connection isn’t stable.

atop [main overview]

atop

atop (the evoluted version of top) provides an overview of the system. The output is divided into two parts: the upper part provide information on a system level, while the lower on single process level. Being an all-around diagnostics tool, some information is the same provided by other tools. For instance avg1, avg5 and avg15 of the CPL (CPU Load) field are the same load averages of the last 1, 5 and 15 minutes offered by uptime. We can also get additional information on processes by pressing some keys: m (information about memory usage), d (disk), n (network, but the netatop module must be installed. See http://atoptool.nl/downloadnetatop.php), c (command line), v (charateristics of processes), u (activities per user) and g (generic, which is the default output). Pressing r reloads the page.
Among the capabilities of atop we can also find logging (logs are usually collected in /var/log/atop/) which makes performances analysis possible also for moments in the past, a feature which is not offered by top.
Furthermore, atop collects information about processes in such a way that almost every process is caught, while top has a system that misses many short-lived process that could make up most of the resource utilization in certain situations.

In this article we have introduced the USE method and analyzed a number of tools to perform troubleshooting in a Linux scenario. With this tools you can identify with a good precision level the causes of loss of performances as well as use use them for planned maintenance routines and system monitoring.

banner eng

fb icon evo twitter icon evo

Word of the Day

The term Edge Computing refers, when used in the cloud-based infrastructure sphere, the set of devices and technologies that allows...

>

The acronym SoC (System on Chip) describes particular integrated circuit that contain a whole system inside a single physical chip:...

>

The acronym PtP (Point-to-Point) indicates point-to-point radio links realized with wireless technologies. Differently, PtMP links connects a single source to...

>

Hold Down Timer is a technique used by network routers. When a node receives notification that another router is offline...

>

In the field of Information Technology, the term piggybacking refers to situations where an unauthorized third party gains access to...

>
Read also the others...

Download of the Day

Netcat

Netcat is a command line tool that can be used in both Linux and Windows environments, capable of...

>

Fiddler

Fiddler is a proxy server that can run locally to allow application debugging and control of data in...

>

Adapter Watch

Adapter Watch is a tool that shows a complete and detailed report about network cards. Download it here.

>

DNS DataView

DNS DataView is a graphical-interface software to perform DNS lookup queries from your PC using system-defined DNS, or...

>

SolarWinds Traceroute NG

SolarWinds Traceroute NG is a command line tool to perform advanced traceroute in Windows environment, compared to the...

>
All Download...

Issues Archive

  • GURU advisor: issue 18 - April 2018

    GURU advisor: issue 18 - April 2018

  • GURU advisor: issue 17 - January 2018

    GURU advisor: issue 17 - January 2018

  • GURU advisor: issue 16 - october 2017

    GURU advisor: issue 16 - october 2017

  • GURU advisor: issue 15 - July 2017

    GURU advisor: issue 15 - July 2017

  • GURU advisor: issue 14 - May 2017

    GURU advisor: issue 14 - May 2017

  • GURU advisor: issue 13 - March 2017

    GURU advisor: issue 13 - March 2017

  • GURU advisor: issue 12 -  January 2017

    GURU advisor: issue 12 - January 2017

  • GURU advisor: issue 11 -  October 2016

    GURU advisor: issue 11 - October 2016

  • 1
  • 2
  • 3
  • BYOD: your devices for your firm

    The quick evolution of informatics and technologies, together with the crisis that mined financial mines, has brought to a tendency inversion: users that prefer to work with their own devices as they’re often more advanced and modern than those the companies would provide. Read More
  • A switch for datacenters: Quanta LB4M

    You don’t always have to invest thousands of euros to build an enterprise-level networking: here’s our test of the Quanta LB4M switch Read More
  • Mobile World Congress in Barcelona

    GURU advisor will be at the Mobile World Congress in Barcelona from February 22nd to 25th 2016!

    MWC is one of the biggest conventions about the worldwide mobile market, we'll be present for the whole event and we'll keep you posted with news and previews from the congress.

    Read More
  • 1