NAME¶

biostacks - Show disk I/O latency with initialization stacks. Uses bpftrace/eBPF.

SYNOPSIS¶

biostacks

DESCRIPTION¶

This tool shows disk I/O latency histograms for each block I/O initialization path. This can help reveal the reason for different latencies, as some may be created by log flushing, others by application reads, etc.

This works by tracing the blk_account_io_start() and the blk_start_request() or blk_mq_start_request() functions using dynamic instrumentation. Linux 5.0 removed the classic I/O scheduler, so the blk_start_request() probe can be removed from the tool (just delete it). This tool may need other maintenance to keep working if these functions change in later kernels.

Since this uses BPF, only the root user can use this tool.

REQUIREMENTS¶

CONFIG_BPF and bpftrace.

EXAMPLES¶

Trace disk I/O latency with initialization stacks:

# biostacks.bt

FIELDS¶

0th

An initialization kernel stack trace (shown in "@[...]") is printed before each I/O histogram.

1st, 2nd

This is a range of I/O latency, in microseconds (shown in "[...)" set notation).

3rd

A column showing the count of I/O in this range.

4th

This is an ASCII histogram representing the count column.

OVERHEAD¶

The rate of biostacks should be low (bounded by device IOPS), such that the overhead of this tool is expected to be negligible.

SOURCE¶

This tool originated from the book "BPF Performance Tools", published by Addison Wesley (2019):

http://www.brendangregg.com/bpf-performance-tools-book.html

See the book for more documentation on this tool.

This version is in the bpftrace repository:

https://github.com/iovisor/bpftrace

Also look in the bpftrace distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

OS¶

Linux

STABILITY¶

Unstable - in development.

AUTHOR¶

Brendan Gregg

SEE ALSO¶

biosnoop.bt(8)