# Detail Data

### Buddy (Memory Fragmentation) Data, collectl -sB

    # MEMORY FRAGMENTATION (4K pages)
    #Node    Zone      1Pg    2Pgs    4Pgs    8Pgs   16Pgs   32Pgs   64Pgs  128Pgs  256Pgs  512Pgs 1024Pgs

This table shows the number of memory fragments by pagesize in increasing powers of 2 for various types of memory defined by the combination of Node and Zone.

### CPU Data, collectl -sC

    # SINGLE CPU STATISTICS
    #   Cpu  User Nice  Sys Wait IRQ  Soft Steal Guest NiceG Idle INTRPT

These are the same fields as reported for verbose CPU Summary, each preceeded by the CPU number. If running collectl V2.5.0 or greater AND you request interrupt summary data, the INTRPT field will also be included.

CPU | The CPU number which the stats are associated with
User | Time spent in User mode, not including time spend in "nice" mode.
Nice | Time spent in Nice mode, that is lower priority as adjusted by the nice command and have the "N" status flag set when examined with "ps".
Sys | This is time spent in "pure" system time.
Wait | Also known as "iowait", this is the time the CPU was idle during an outstanding disk I/O request. This is not considered to be part of the total or system times reported in brief mode.
Irq | Time spent processing interrupts and also considered to be part of the summary system time reported in "brief" mode.
Soft | Time spent processing soft interrupts and also considered to be part of the summary system time reported in "brief" mode.
Steal | Time spend in involuntary wait state while the hypervisor was servicing another virtual processor.
Guest | Time spent running a virtual CPU for guest operating systems under the control of the Linux kernel, new since 2.6.24
NiceG | Time spent running a niced guest (virtual CPU for guest operating systems under the control of the Linux kernel), new since 2.6.33
Idle | Time spent idle, nothing that since CPU numbers are rounded off, they may not always add up to 100%
Intrpt | If the interrupt summary stats were requested at the same time, this will be included which is the aggregate number of interrupts for each CPU.

### Disk Data, collectl -sD

If you specify filtering with --dskfilt, the disk names that match the pattern(s) will either be included or excluded from the the summary data. However, the data will still be collected so if recorded to a file can later be viewed. Note: if you specify --dskopts f, fractional values will be reported for some of the fields for more precision.

    # DISK STATISTICS (/sec)
    #          <---------reads---------><---------writes---------><--------averages--------> Pct
    #Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util

Name | Name of the disk the statistics are being reported for.
KBytes | KB read/sec
Merged | Read requests merged per second when being dequeued.
IOs | Number of reads/sec
Size | Average read I/O size in KBytes
KBytes | KB written/sec
Merged | Write requests merged per second when being dequeued.
IOs | Number of writes/sec
Size | Average write I/O size in KBytes
RWSize | Average combined read and write I/O size in KBytes. This is not the average of the read and write sizes but rather the sum of the reads/write divided by the number of I/Os
QLen | Average number of requests queued
Wait | Average time in msec for a request has been waiting in the queue
SvcTim | Average time in msec for a request to be serviced by the device
Util | Percentage of CPU time during which I/O requests were issued

### Infiniband, collectl -sX

    # INFINIBAND STATISTICS (/sec)
    #HCA    KBIn   PktIn  SizeIn   KBOut  PktOut SizeOut  Errors

HCA | HCA instance name
KBIn | KB received/sec.
PktIn | Received packets/sec.
SizeIn | Average incoming packet size in KB
KBOut | KB transmitted/sec.
PktOut | Transmitted packets/sec.
SizeOut | Average outgoing packet size in KB
Errs | Count of current errors. Since these are typically infrequent, it is felt that reporting them as a rate would result in either not seeing them OR round-off hiding their values.

### Interrupts, collectl -sJ

    # INTERRUPT DETAILS
    # Int    Cpu0   [Cpu...]   Type            Device(s)

Int | Interrupt number within the range 0-255. Note that only those interrupts that have had any activity since the last monitoring interval will be reported
CPUn... | The CPU for which the interrupt count is being reported. There will be one column/CPU
Type | Interrupt type, whitespace removed
Device | The names of the devices which are generating this interrut as a comma separated list

### Lustre Data, collectl -sL

There are several formats the lustre detail data can take based on whether you're looking at a client or an OSS (there is not any MDS specific detail data, though it does share the same disk-level buffer size data as the OSS). Furthermore, if one specifies the -sLL form of the detail switch OST level details will be reported where appropriate.

Lustre Client, collectl -sL

    # LUSTRE CLIENT DETAIL (/sec)
    #Fils  KBRead  Reads SizeKB KBWrite Writes SizeKB

Filsys | Name of the filesystem these stats apply to
KBRead | KBs read/sec
SizeKB | Average read size
Reads | Reads/sec
KBWrite | KBs written/sec
Writes | Writes/sec
SizeKB | Average write size

Lustre Client, collectl --lustops O

    # LUSTRE CLIENT DETAIL (/sec)
    #Fils  Ost     KBRead  Reads SizeKB KBWrite Writes SizeKB

The data here is the same as that reported for the standard client side lustre data except now it is broken down by OST within the file system.

Filsys | Name of the filesystem these stats apply to
Ost | OST name within the filesystem
KBRead | KBs read/sec
Reads | Reads/sec
SizeKB | Average read size
KBWrite | KBs written/sec
Writes | Writes/sec
SizeKB | Average write size

Lustre Client RPB-Buffer Stats, collectl --lustopts B

    # LUSTRE CLIENT DETAIL: RPC-BUFFERS (pages)
    #Filsys  Ost   RdK  Rds   1K   2K   ...   WrtK Wrts   1K   2K   ...

This form also includes the reads/writs within the filesystem, but also add the sizes of the RPM buffers. Since these numbers always apply to OSTs you need to use the -sLL form of the subsystem switch.

Filsys | Name of the filesystem these stats apply to
Ost | OST name within the filesystem
RdK | KBs read/sec
Rds | Reads/sec
nK | Number of pages of of this size read
WrtK | KBs written/sec
Wrts | Writes/sec
nK | Number of pages of of this size written

Lustre Client Metadata, collectl -sL --lustopts M

    # LUSTRE CLIENT DETAIL: METADATA
    #Filsys   KBRead  Reads KBWrite  Writes  Open Close GAttr SAttr  Seek Fsync DrtHit DrtMis

Filsys | Name of the filesystem these stats apply to
KBRead | KBs read/sec
Reads | Reads/sec
KBWrite | KBs written/sec
Writes | Writes/sec
Open | Opens/sec
Close | Closes/sec
GAttr | Get Attributes/sec
SAttr | Set Attributes/sec
Seek | Seeks/sec
Fsync | FSyncs/sex
DrtHit | Dirty Hits/sec
DrtMis | Dirty Misses/sec

Lustre Client Readhead, collectl -sL --lustopts R

    # LUSTRE CLIENT DETAIL: READAHEAD
    #Filsys   KBRead Reads  KBWrite Writes  Pend  Hits Misses NotCon MisWin LckFal  Discrd ZFile ZerWin RA2Eof HitMax

Filsys | Name of the filesystem these stats apply to
KBRead | KBs read/sec
Reads | Reads/sec
KBWrite | KBs written/sec
Writes | Writes/sec
Pend | Pending issued pages
Hits | Hits
Misses | Misses
NotCon | Readpage not consecutive
MisWin | Miss inside window
LckFal | Failed lock match
Discrd | Read but discarded
ZFile | Zero length file
ZerWin | Zero size window
RA2Eof | Read-ahead to EOF
HitMax | Hit max r-a issue

Lustre OSS, collectl -sL

    # LUSTRE FILESYSTEM SINGLE OST STATISTICS (/sec)
    #Ost            KBRead   Reads  SizeKB    KBWrite  Writes  SizeKB

Ost | OST name
KBRead | KBs read/sec
Reads | Reads/sec
SizeKB | Average read size
KBWrite | KBs written/sec
Writes | Writes/sec
SizeKB | Average write size

Lustre OSS RPC Buffers, collectl -sL --lustopts B

    # LUSTRE FILESYSTEM SINGLE OST STATISTICS
    #Ost            RdK  Rds   1P   2P  ...  WrtK Wrts   1P   2P  ...

Filsys | Name of the filesystem these stats apply to
Ost | OST name within the filesystem
RdK | KBs read/sec
Rds | Reads/sec
nP | Number of pages of of this size read
WrtK | KBs written/sec
Wrts | Writes/sec
nP | Number of pages of of this size written

Lustre OSS and MDS Disk Buffers, collectl -sL --lustopts D

This display is very similar the the RPC buffers in that the sizes of different size I/O requests are reported. In this case there are requests send to the disk driver. Note that this report is only available for HP's SFS.

    # LUSTRE DISK BLOCK LEVEL DETAIL (units are 512 bytes)
    #DISK RdK  Rds 0.5K   1K   2K   ...   WrtK Wrts 0.5K   1K   2K   ...

Disk | Name of the disk these stats apply to
RdK | Reads/sec
Rds | KBs read/sec
nK | Number of blocks of of this size read
WrtK | Writes/sec
Wrts | KBs written/sec
nK | Number of blocks of of this size written

### Memory Data, collectl -sM

This is also known as numa data and provides detail information about memory utilization in each numa node.

    # MEMORY STATISTICS
    # Node    Total     Used     Free     Slab   Mapped     Anon    Locked  Inact HitPct

Node | Numa node number, which is usually the same as a physical socket
Total | Total physical memory
Used | Used physical memory. This does not include memory used by the kernel itself.
Free | Unallocated memory
Slab | Memory used for slabs, see collectl -sY
Mapped | Memory mapped by processes
Anon | Anonymous memory
Locked | Locked memory
Inactive | Inactive pages, which is the sum of Inactive(anon) and Inactive(file). Note that Inactive(anon) is not considered nor included in the previous anonynous memory field.

Hit%

It is currenlty not entirely clear how useful this number actually is as it refers to the hit percentages for both local and foreign memory as a single number. Most importat, it does not refer to memory access but rather memory allocation. In other words, it does not differentiate between one failing to allocation memory and only referencing it a small number of time vs a very large number of times. Clearly the latter would be more interesting from a performance perspective.

### Network Data, collectl -sN

If you specify filtering with --netfilt, the names that match the pattern(s) will either be included or excluded from the the summary data. However, the data will still be collected so if recorded to a file can later be viewed.

    # NETWORK STATISTICS (/sec)
    #Num    Name  KBIn  PktIn SizeIn  MultI   CmpI  ErrsI  KBOut PktOut  SizeO   CmpO   ErrsO

Num | Each network interface is numbered, starting with 0
Name | Name of the interface
KBIn | Incoming KB/sec
PktIn | Incoming packets/sec
SizeI | Average incoming packet size in bytes
MultI | Incoming multicast packets/sec
CmpI | Incoming compressed packets/sec
ErrsI | Total incoming errors/sec. This is an aggregration of incoming errors. To see explicit error counters use --netopts e
KBOut | Outgoing KB/sec
PktOut | Outgoing packets/sec
SizeO | Average outgoing packet size in bytes
CmpO | Outgoing compressed packets/sec
ErrsO | Total outgoing errors/sec. This is an aggregation of outgoing errors. To see explicit error counters use --netopts e

### Network Data, collectl -sN --netopts e

    # NETWORK ERRORS SUMMARY (/sec)
    #Num    Name   ErrIn  DropIn  FifoIn FrameIn    ErrOut DropOut FifoOut CollOut CarrOut

Num | Each network interface is numbered, starting with 0
Name | Name of the interface
ErrIn | Receive errors/sec detected by the device driver
DropIn | Receive packets dropped/sec
FifoIn | Receive packet FIFO buffer errors/sec
FrameIn | Receive packet framing errors/sec
ErrOut | Transmit errors/sec detected by the device driver
DropOut | Transmit packets dropped/sec
FifoOut | Transmit packet FIFO buffer errors/sec
CollOut | Transmit collisions/sec detected on the interface
CarrOut | Transmit packet carrier loss errors detected/sec

### NFS Data, collectl -sF

By default, collectl will report all 6 types of nfs data (clients and servers for all 3 versions of nfs) unless one has limited the reporting with --nfsfilt. For versions prior to 3.2.1 which only collected a single type of data, collectl will only report that single type.

Also note that nfs V2 records 2 fields that V3 doesn't record and V4 records many more. At this time and detail data is standardized on the V3 format and fields not collected will be left blank. These fields map onto those reported by nfsstat as indicated and are reported as rates.

    # NFS SERVER/CLIENT DETAILS (/sec)
    #Type Read Writ Comm Look Accs Gttr Sttr Rdir Cre8 Rmov Rnam Link Rlnk Null Syml Mkdr Rmdr Fsta Finf Path Mknd Rdr+

Type | a combination of Clt or Svr and one of 2, 3 or 4
Read | Reads
Writ | Writes
Comm | Commits
Look | Lookups
Accs | Accesses
Gttr | Getattrs
Sttr | Setattrs
Rdir | Readdirs
Cre8 | Creates
Rmov | Removes
Rnam | Renames
Link | Links
Rlnk | Readlinks
Null | Nulls
Syml | Symlinks
Mkdr | Mkdirs
Rmdr | Rmdirs
Fsta | Fsstats
Finf | Fsinfos
Path | Pathconfs
Mknd | Mknods
Rdr+ | Readdirpluses

### Process Data, collectl -sZ

There are actually multiple formats process data can be displayed in, the default being the one shown immediately below. By using --procopts as shown in later examples, you can change what is displayed, noting that when playing back the data from a raw file, all information has been recorded and so you can actually play it back multiple times and see different views.

These switched can also be use in conjunction with --top.

    # PROCESS SUMMARY (faults are /sec)
    # PID  User     PR  PPID S   VSZ   RSS  CP  SysT  UsrT Pct  AccuTime  RKB  WKB MajF MinF Command

PID | Pid of the process
User | Name of user which this process is running under. In playback mode on a different machine, use -oP to direct collectl to use the password file named in collectl.conf (default is /etc/passwd) to lookup the corresponding username. Otherwise the UID will be reported instead.
PR | Process priority
PPID | PID of this process's parent
S | Process State: S - Sleeping, D - Uninterruptable Sleep, R - Running, Z - Zombie or T - Stopped/Traced
VSZ | This is the amount of VS memory used by this process
RSS | This is the amount of RSS memory used by this process
CP | CPU number this process is currently running on
SysT | The amount of System Time this process used during this interval
UsrT | The amount of User Time this process used during this interval
Pct | Percentage of the current interval taken up by this task (the User and System time are used for this calculation)
AccuTime | Total accumulated System and User time since the process began execution
RKB | This is the number of kilobytes of data written by each process. Both this and the WKB field are only present if the kernel had proces I/O monitoring enabled which is not the default as of 2.6.23.
WKB | This is the number of kilobytes of data read by each process
MajF | Major Page Faults per second
MinF | Minor Page Faults per second
Command | Command that is running. Path and command line options are NOT included unless --procopts w

### Process Data, collectl -sZ --procopts x

This format is essentially idential to the last except that it adds extended information to the display, specifically VCtx and NCtx.

    # PROCESS SUMMARY (counters are /sec)
    # PID  User     PR  PPID THRD S   VSZ   RSS CP  SysT  UsrT Pct  AccuTime  RKB  WKB VCtx NCtx MajF MinF Command

VCtx | Voluntary context switches
NCtx | Non-voluntary context switches

### Process I/O Data, collectl --procopts i

    # PID  User    PPID S  SysT  UsrT   Pct AccuTime   RKB   WKB  RKBC  WKBC  RSys  WSys  Cncl  Command

PID | Pid of the process
User | Name of user which this process is running under. In playback mode on a different machine, use -oP to direct collectl to use the password file named in collectl.conf (default is /etc/passwd) to lookup the corresponding username. Otherwise the UID will be reported instead.
PPID | PID of this process's parent
S | Process State: S - Sleeping, D - Uninterruptable Sleep, R - Running, Z - Zombie or T - Stopped/Traced
SysT | The amount of System Time this process used during this interval
UsrT | The amount of User Time this process used during this interval
Pct | Percentage of the current interval taken up by this task (the User and System time are used for this calculation)
AccuTime | Total accumulated System and User time since the process began execution
RKB | Attempt to count the number of bytes which this process really did cause to be fetched from the storage layer by doing calls to read_bytes. This is done at the submit_bio() level, so it is accurate for block-backed filesystems.
WKB | Attempt to count the number of bytes which this process caused to be sent to the storage layer by doing calls to write_bytes. This is done at page-dirtying time.
RKBC | Number of bytes which were read via read, readv, pread and sendfile. Since these requests are satisfied from kernel pagecache they won't be accounted for by RKB, because they didn't require any I/O.
WKBC | Number of bytes which were written via write, writev, pwrite and sendfile. Like RKBC, since the I/O uses the pagecache these values won't be accounted for by WKB.
RSys | Number of read syscalls, specifically: read, pread, readv and sendfile
WSys | Number of write syscalls, specifically: write, pwrite, writev and sendfile
Cncl | Number of cancelled write bytes.

### Process Memory Data, collectl --procmem

    # PID  User     S VmSize  VmLck  VmRSS VmData  VmStk  VmExe  VmLib  VmSwp MajF MinF Command

PID | Pid of the process
User | Name of user which this process is running under. In playback mode on a different machine, use -oP to direct collectl to use the password file named in collectl.conf (default is /etc/passwd) to lookup the corresponding username. Otherwise the UID will be reported instead.
S | Process State: S - Sleeping, D - Uninterruptable Sleep, R - Running, Z - Zombie or T - Stopped/Traced
VmSize | Size of Virtual memory used by the entire process
VmLck | Size of Locked Virtual Memory
VmRSS | Size of Resident Virtual Memory
VmData | Size of Virtual Memory used for heap
VmStk | Size of Virtual Memory used for stack
VmExe | Size of Virtual Memory used for exe and statically linked libraries
VmLib | Size of Virtual Memory used for dynamically linked libraries
VmSwp | Size of Virtual Memory used for swapping. This does not necessarily mean a process is actively swapping but only that the memory has been mapped.
MajF | Major Page Faults per second
MinF | Minor Page Faults per second
Command | Command that is running. Path and command line options are NOT included unless --procopt w

### Slab Data, collectl -sY

There are actualy 3 different formats for slab data. The first applies to all kernels prior to 2.6.22 and contains the same fields as the Summary report for each named slab and loses the caches fields.

    # SLAB DETAIL
    #          <-----------Objects----------><---------Slab Allocation------><----Change-->
    #Name      InUse   Bytes   Alloc   Bytes   InUse   Bytes   Total   Bytes   Diff    Pct

Objects</td</tr>
InUse | Total number of objects that are currently in use.
Bytes | Total size of all the objects in use.
Alloc | Total number of objects that have been allocated but not necessarily in use.
Bytes | Total size of all the allocated objects whether in use or not.
Slab Allocation</td</tr>
InUse | Number of slabs that have at least one active object in them.
Bytes | Total size of all the slabs.
Total | Total number of slabs that have been allocated whether in use or not.
Bytes | Total size of all the slabs that have been allocted whether in use or not.
Diff | Change in size of this slab since last sample in bytes
Pct | Percentage change in size of this slab

This second format applies to the new SLUB allocator starting with the 2.6.22 kernel. As with the old format slab detail report, the same fields as are found in the Slab Summary Report are shown for each named slab.

    # SLAB DETAIL
    #                             <----------- objects --------><--- slabs ---><---------allocated memory-------->
    #Slab Name                    Size  /slab   In Use    Avail  SizeK  Number     UsedK    TotalK   Change    Pct

Objects</td</tr>
Size | Size of a single slab object
/Slab | The number of objecs in a single slab
InUse | The total number of objects that have been allocated to processes.
Avail | The total number of objects that are available in the currently allocated slabs. This includes those that have already been allocated toprocesses.
Slabs
SizeK | The size of one slab, which typically contains multiple objects
Number | This is the number of individual slabs that have been allocated and taking physical memory.
Memory
UsedK | Memory used by those objects that have been allocated to processes.
TotalK | Total physical memory allocated to processes. When there is no filtering in effect, this number will be equal to the Slabs field reported by -sm.
Change | Change in size of this slab since last sample in bytes
Pct | Percentage change in size of this slab

The third format uses a common format based on the slabtop utility for displaying top slab data sorted by any of the listed column headers. All on need to do is use one of these names as the argument to the --top switch in lower case (use --showtopopts for full list of options to the --top switch). All the same rules apply for controlling the number of lines in the data section, mixing in subsystem data and even using with playback mode.

    # TOP SLABS 15:38:08
    #NumObj  ActObj  ObjSize  NumSlab  Obj/Slab  TotSize  TotChg  TotPct  Name

NumObj | Total number of objects that are available, which includes those in use
ActObj | Number of objects that are in use
ObjSize | Size of an individual slab object
NumSlab | Total number of slabs that have been allocated
Obj/Slab | This is the constant number of objects that fit into one slab
TotSize | Amount of memory consumed by this slab even if not all objects are actualy allocated
TotChg | Change in size of this slab since last sample in bytes
TotPct | Percentage change in size of this slab
Name | Slab Name

updated July 23, 2014