名前¶

guestfs-performance - engineering libguestfs for greatest performance

説明¶

This page documents how to get the greatest performance out of libguestfs, especially when you expect to use libguestfs to manipulate thousands of virtual machines or disk images.

Three main areas are covered. Libguestfs runs an appliance (a small Linux distribution) inside qemu/KVM. The first two areas are: minimizing the time taken to start this appliance, and the number of times the appliance has to be started. The third area is shortening the time taken for inspection of VMs.

BASELINE MEASUREMENTS¶

Before making changes to how you use libguestfs, take baseline measurements.

 time guestfish -a /dev/null run

 make
 ./run utils/boot-benchmark/boot-benchmark

 time guestfish --ro -a disk.img -i exit
 time guestfish --ro -d GuestName -i exit

UNDERSTANDING THE APPLIANCE AND WHEN IT IS BUILT/CACHED¶

The first time you use libguestfs, it will build and cache an appliance. This is usually in /var/tmp/.guestfs-*, unless you have set $TMPDIR or $LIBGUESTFS_CACHEDIR in which case it will be under that temporary directory.

For more information about how the appliance is constructed, see "SUPERMIN APPLIANCES" in supermin(1).

Every time libguestfs runs it will check that no host files used by the appliance have changed. If any have, then the appliance is rebuilt. This usually happens when a package is installed or updated on the host (eg. using programs like "yum" or "apt-get"). The reason for reconstructing the appliance is security: the new program that has been installed might contain a security fix, and so we want to include the fixed program in the appliance automatically.

These are the performance implications:

• The process of building (or rebuilding) the cached appliance is slow, and you can avoid this happening by using a fixed appliance (see below).
• If not using a fixed appliance, be aware that updating software on the host will cause a one time rebuild of the appliance.
• /var/tmp (or $TMPDIR, $LIBGUESTFS_CACHEDIR) should be on a fast disk, and have plenty of space for the appliance.

固定アプライアンスの使用法¶

To fully control when the appliance is built, you can build a fixed appliance. This appliance should be stored on a fast local disk.

アプライアンスを構築するには、以下のコマンドを実行します:

 libguestfs-make-fixed-appliance <directory>

replacing "<directory>" with the name of a directory where the appliance will be stored (normally you would name a subdirectory, for example: /usr/local/lib/guestfs/appliance or /dev/shm/appliance).

Then set $LIBGUESTFS_PATH (and ensure this environment variable is set in your libguestfs program), or modify your program so it calls "guestfs_set_path". For example:

 export LIBGUESTFS_PATH=/usr/local/lib/guestfs/appliance

Now you can run libguestfs programs, virt tools, guestfish etc. as normal. The programs will use your fixed appliance, and will not ever build, rebuild, or cache their own appliance.

(この話題の詳細は libguestfs-make-fixed-appliance(1) を参照してください)。

REDUCING THE NUMBER OF TIMES THE APPLIANCE IS LAUNCHED¶

By far the most effective, though not always the simplest way to get good performance is to ensure that the appliance is launched the minimum number of times. This will probably involve changing your libguestfs application.

Try to call "guestfs_launch" at most once per target virtual machine or disk image.

Instead of using a separate instance of guestfish(1) to make a series of changes to the same guest, use a single instance of guestfish and/or use the guestfish --listen option.

Consider writing your program as a daemon which holds a guest open while making a series of changes. Or marshal all the operations you want to perform before opening the guest.

You can also try adding disks from multiple guests to a single appliance. Before trying this, note the following points:

1.

Adding multiple guests to one appliance is a security problem because it may allow one guest to interfere with the disks of another guest. Only do it if you trust all the guests, or if you can group guests by trust.

2.

There is a hard limit to the number of disks you can add to a single appliance. Call "guestfs_max_disks" in guestfs(3) to get this limit. For further information see "LIMITS" in guestfs(3).

3.

Using libguestfs this way is complicated. Disks can have unexpected interactions: for example, if two guests use the same UUID for a filesystem (because they were cloned), or have volume groups with the same name (but see "guestfs_lvm_set_filter").

virt-df(1) adds multiple disks by default, so the source code for this program would be a good place to start.

SHORTENING THE TIME TAKEN FOR INSPECTION OF VMs¶

The main advice is obvious: Do not perform inspection (which is expensive) unless you need the results.

If you previously performed inspection on the guest, then it may be safe to cache and reuse the results from last time.

Some disks don’t need to be inspected at all: for example, if you are creating a disk image, or if the disk image is not a VM, or if the disk image has a known layout.

Even when basic inspection ("guestfs_inspect_os") is required, auxiliary inspection operations may be avoided:

• Mounting disks is only necessary to get further filesystem information.
• Listing applications ("guestfs_inspect_list_applications") is an expensive operation on Linux, but almost free on Windows.
• Generating a guest icon ("guestfs_inspect_get_icon") is cheap on Linux but expensive on Windows.

PARALLEL APPLIANCES¶

Libguestfs appliances are mostly I/O bound and you can launch multiple appliances in parallel. Provided there is enough free memory, there should be little difference in launching 1 appliance vs N appliances in parallel.

On a 2-core (4-thread) laptop with 16 GB of RAM, using the (not especially realistic) test Perl script below, the following plot shows excellent scalability when running between 1 and 20 appliances in parallel:

  12 ++---+----+----+----+-----+----+----+----+----+---++
     +    +    +    +    +     +    +    +    +    +    *
     |                                                  |
     |                                               *  |
  11 ++                                                ++
     |                                                  |
     |                                                  |
     |                                          *  *    |
  10 ++                                                ++
     |                                        *         |
     |                                                  |
 s   |                                                  |
   9 ++                                                ++
 e   |                                                  |
     |                                     *            |
 c   |                                                  |
   8 ++                                  *             ++
 o   |                                *                 |
     |                                                  |
 n 7 ++                                                ++
     |                              *                   |
 d   |                           *                      |
     |                                                  |
 s 6 ++                                                ++
     |                      *  *                        |
     |                   *                              |
     |                                                  |
   5 ++                                                ++
     |                                                  |
     |                 *                                |
     |            * *                                   |
   4 ++                                                ++
     |                                                  |
     |                                                  |
     +    *  * *    +    +     +    +    +    +    +    +
   3 ++-*-+----+----+----+-----+----+----+----+----+---++
     0    2    4    6    8     10   12   14   16   18   20
               number of parallel appliances

It is possible to run many more than 20 appliances in parallel, but if you are using the libvirt backend then you should be aware that out of the box libvirt limits the number of client connections to 20.

The simple Perl script below was used to collect the data for the plot above, but there is much more information on this subject, including more advanced test scripts and graphs, available in the following blog postings:

http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-1/ http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-2/ http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-3/ http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-4/

 #!/usr/bin/env perl
 
 use strict;
 use threads;
 use warnings;
 use Sys::Guestfs;
 use Time::HiRes qw(time);
 
 sub test {
     my $g = Sys::Guestfs->new;
     $g->add_drive_ro ("/dev/null");
     $g->launch ();
     
     # You could add some work for libguestfs to do here.
     
     $g->close ();
 }
 
 # Get everything into cache.
 test (); test (); test ();
 
 for my $nr_threads (1..20) {
     my $start_t = time ();
     my @threads;
     foreach (1..$nr_threads) {
         push @threads, threads->create (\&test)
     }
     foreach (@threads) {
         $_->join ();
         if (my $err = $_->error ()) {
             die "launch failed with $nr_threads threads: $err"
         }
     }
     my $end_t = time ();
     printf ("%d %.2f\n", $nr_threads, $end_t - $start_t);
 }

USING USER-MODE LINUX¶

Since libguestfs 1.24, it has been possible to use the User-Mode Linux (uml) backend instead of KVM (see "USER-MODE LINUX BACKEND" in guestfs(3)). This section makes some general remarks about this backend, but it is highly advisable to measure your own workload under UML rather than trusting comments or intuition.

• UML usually performs the same or slightly slower than KVM, on baremetal.
• However UML often performs the same under virtualization as it does on baremetal, whereas KVM can run much slower under virtualization (since hardware virt acceleration is not available).
• Upload and download is as much as 10 times slower on UML than KVM. Libguestfs sends this data over the UML emulated serial port, which is far less efficient than KVM’s virtio-serial.
• UML lacks some features (eg. qcow2 support), so it may not be applicable at all.

For some actual figures, see: http://rwmj.wordpress.com/2013/08/14/performance-of-user-mode-linux-as-a-libguestfs-backend/#content

性能劣化のトラブルシューティング¶

DETAILED ANALYSIS¶

 make
 ./run utils/boot-analysis/boot-analysis

 $ guestfish -a /dev/null run -v |& ts -i '%.s'
 0.000022 libguestfs: launch: program=guestfish
 0.000134 libguestfs: launch: version=1.29.31fedora=23,release=2.fc23,libvirt
 0.000044 libguestfs: launch: backend registered: unix
 0.000035 libguestfs: launch: backend registered: uml
 0.000035 libguestfs: launch: backend registered: libvirt
 0.000032 libguestfs: launch: backend registered: direct
 0.000030 libguestfs: launch: backend=libvirt
 0.000031 libguestfs: launch: tmpdir=/tmp/libguestfsw18rBQ
 0.000029 libguestfs: launch: umask=0002
 0.000031 libguestfs: launch: euid=1000
 0.000030 libguestfs: libvirt version = 1002012 (1.2.12)
 [etc]

 global last;
 
 function display_time () {
       now = gettimeofday_us ();
       delta = 0;
       if (last > 0)
             delta = now - last;
       last = now;
 
       printf ("%d (+%d):", now, delta);
 }
 
 probe begin {
       last = 0;
       printf ("ready\n");
 }
 
 /* Display all calls to static markers. */
 probe process("/usr/lib*/libguestfs.so.0")
           .provider("guestfs").mark("*") ? {
       display_time();
       printf ("\t%s %s\n", $$name, $$parms);
 }
 
 /* すべての guestfs_* 関数の呼び出しを一覧表示します。 */
 probe process("/usr/lib*/libguestfs.so.0")
           .function("guestfs_[a-z]*") ? {
       display_time();
       printf ("\t%s %s\n", probefunc(), $$parms);
 }

 # stap time.stap
 ready

 $ guestfish -a /dev/null run

 xxxx (+0):     guestfs_create 
 xxxx (+29):    guestfs_set_pgroup g=0x17a9de0 pgroup=0x1
 xxxx (+9):     guestfs_add_drive_opts_argv g=0x17a9de0 [...]
 xxxx (+8):     guestfs_int_safe_strdup g=0x17a9de0 str=0x7f8a153bed5d
 xxxx (+19):    guestfs_int_safe_malloc g=0x17a9de0 nbytes=0x38
 xxxx (+5):     guestfs_int_safe_strdup g=0x17a9de0 str=0x17a9f60
 xxxx (+10):    guestfs_launch g=0x17a9de0
 xxxx (+4):     launch_start 
 [etc]

 #!/bin/bash -
 
 # 実際の QEMU バイナリーを指し示すようこれを設定してください。
 qemu=/usr/bin/qemu-kvm
 
 if [ "$1" != "-global" ]; then
     # ヘルプの出力などを解析します。
     exec $qemu "$@"
 else 
     # Really running qemu.
     exec $qemu -S -s "$@"
 fi

 LIBGUESTFS_HV=/path/to/qemu-wrapper guestfish -a /dev/null -v run

 $ gdb
 (gdb) set architecture i8086
 The target architecture is assumed to be i8086
 (gdb) target remote :1234
 Remote debugging using :1234
 0x0000fff0 in ?? ()
 (gdb) cont

PERFORMANCE REGRESSIONS IN OTHER PROGRAMS¶

Sometimes performance regressions happen in other programs (eg. qemu, the kernel) that cause problems for libguestfs.

In the libguestfs source, utils/boot-benchmark/boot-benchmark-range.pl is a script which can be used to benchmark libguestfs across a range of git commits in another project to find out if any commit is causing a slowdown (or speedup).

To find out how to use this script, consult the manual:

 ./utils/boot-benchmark/boot-benchmark-range.pl --man

関連項目¶

supermin(1), guestfish(1), guestfs(3), guestfs-examples(3), guestfs-internals(1), libguestfs-make-fixed-appliance(1), stap(1), qemu(1), gdb(1), http://libguestfs.org/.

著者¶

Richard W.M. Jones ("rjones at redhat dot com")

COPYRIGHT¶

Copyright (C) 2012-2019 Red Hat Inc.

LICENSE¶

BUGS¶

To get a list of bugs against libguestfs, use this link: https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtualization+Tools

To report a new bug against libguestfs, use this link: https://bugzilla.redhat.com/enter_bug.cgi?component=libguestfs&product=Virtualization+Tools

When reporting a bug, please supply:

• The version of libguestfs.
• Where you got libguestfs (eg. which Linux distro, compiled from source, etc)
• Describe the bug accurately and give a way to reproduce it.
• Run libguestfs-test-tool(1) and paste the complete, unedited output into the bug report.