DevOps / Sys Admin Q & A #15 : Why is the Server slow?

bogotobogo.com site search:

Introduction

Tracking down the sluggish/unresponsive host server issue could be very tricky. That's because any networking problems often create the same set of symptoms as caused by the server itself.

When a machine is slow, it is often because we have consumed a particular resource on the system.

The main resources are CPU, RAM, disk I/O, and network.

System Load

System load average is probably the fundamental metric we can use when troubleshooting a sluggish system.

One of the first commands is uptime:

$ uptime
 21:21:09 up 2 days, 11:23,  3 users,  load average: 0.92, 1.33, 1.30

The three numbers after load average - 0.92, 1.33, and 1.30 - represent the 1-, 5-, and 15-minute load averages on the machine, respectively.

A system load average is equal to the average number of processes in a runnable or uninterruptible state.

Runnable processes are either currently using the CPU or waiting to do so, and uninterruptible processes are waiting for I/O.

Picture info DevOps / Sys Admin Q & A #10 : Trouble Shooting

High Load?

Here are the metrics for each type of loads:

CPU-bound (processes waiting on CPU resources)
RAM-bound (specifically, high RAM usage that has moved into swap)
I/O-bound (processes fighting for disk or network I/O)

Typically systems seem to be more responsive when under CPU-bound load than when under I/O-bound load.

Even though a system is CPU-bound, and it could still run diagnostic tools on the system with pretty good response times.

On the other hand, a system with relatively low I/O-bound loads on which just logging in took a while because the disk I/O was completely saturated.

A system that runs out of RAM resources often appears to have I/O-bound load, since once the system starts using swap storage on the disk, it can consume disk resources and cause a downward spiral as processes slow to a halt.

top for what?

The basic steps are to examine the top output to identify what resources we're running out of (CPU, RAM, disk I/O).

Once we have figured that out, we can try to identify what processes are consuming those resources the most.

High User CPU Time - If we see high user CPU time but low I/O wait times, we simply need to identify which processes on the system are consuming the most CPU.
Out-of-Memory
High I/O Wait - When we see high I/O wait, one of the first things we should check is whether the machine is using a lot of swap. If we have plenty of RAM, we will need to figure out which program is consuming the most I/O.
It can sometimes be difficult to figure out exactly which process is using the I/O, but if we have multiple partitions on our system, we can narrow it down by figuring out which partition most of the I/O is on.
To do this, we can use iostat program, which is provided by the sysstat:
```
$ iostat -p sda
Linux 3.13.0-40-generic (laptop) 	11/30/2015 	_x86_64_	(2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          29.46    0.40    8.94    1.38    0.00   59.82

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              10.46        87.83        69.78   19248132   15291117
sda1              0.01         0.04         0.00       9751          1
sda2              8.91        64.39        49.27   14109557   10797156
sda3              1.24        23.37        20.51    5120684    4493960
```
When we have a system under heavy I/O load, the first step is to look at each of the partitions and identify which partition is getting the heaviest I/O load.
In addition to iostat, we have a much simpler tool available in newer distributions called iotop. In effect, it is a blend of top and iostat in that it shows us all of the running processes on the system sorted by their I/O statistics.