[Drbd-dev] [PATCH v4 01/11] block: make generic_make_request handle arbitrarily sized bios

[Ming Lin]

On Tue, May 26, 2015 at 9:04 AM, Mike Snitzer <snitzer at redhat.com> wrote:
> On Tue, May 26 2015 at 11:02am -0400,
> Ming Lin <mlin at kernel.org> wrote:
>
>> On Tue, May 26, 2015 at 7:36 AM, Mike Snitzer <snitzer at redhat.com> wrote:
>> > On Fri, May 22 2015 at  2:18pm -0400,
>> > Ming Lin <mlin at kernel.org> wrote:
>> >
>> >> From: Kent Overstreet <kent.overstreet at gmail.com>
>> >>
>> >> The way the block layer is currently written, it goes to great lengths
>> >> to avoid having to split bios; upper layer code (such as bio_add_page())
>> >> checks what the underlying device can handle and tries to always create
>> >> bios that don't need to be split.
>> >>
>> >> But this approach becomes unwieldy and eventually breaks down with
>> >> stacked devices and devices with dynamic limits, and it adds a lot of
>> >> complexity. If the block layer could split bios as needed, we could
>> >> eliminate a lot of complexity elsewhere - particularly in stacked
>> >> drivers. Code that creates bios can then create whatever size bios are
>> >> convenient, and more importantly stacked drivers don't have to deal with
>> >> both their own bio size limitations and the limitations of the
>> >> (potentially multiple) devices underneath them.  In the future this will
>> >> let us delete merge_bvec_fn and a bunch of other code.
>> >
>> > This series doesn't take any steps to train upper layers
>> > (e.g. filesystems) to size their bios larger (which is defined as
>> > "whatever size bios are convenient" above).
>> >
>> > bio_add_page(), and merge_bvec_fn, served as the means for upper layers
>> > (and direct IO) to build up optimally sized bios.  Without a replacement
>> > (that I can see anyway) how is this patchset making forward progress
>> > (getting Acks, etc)!?
>> >
>> > I like the idea of reduced complexity associated with these late bio
>> > splitting changes I'm just not seeing how this is ready given there are
>> > no upper layer changes that speak to building larger bios..
>> >
>> > What am I missing?
>>
>> See: [PATCH v4 02/11] block: simplify bio_add_page()
>> https://lkml.org/lkml/2015/5/22/754
>>
>> Now bio_add_page() can build lager bios.
>> And blk_queue_split() can split the bios in ->make_request() if needed.
>
> That'll result in quite large bios and always needing splitting.
>
> As Alasdair asked: please provide some performance data that justifies
> these changes.  E.g use a setup like: XFS on a DM striped target.  We
> can iterate on more complex setups once we have established some basic
> tests.

Here are fio results of XFS on a DM stripped target with 2 SSDs + 1 HDD.
Does it make sense?

                                         4.1-rc4                 4.1-rc4-patched
                                         ------------------
-----------------------
                                          (KB/s)                 (KB/s)
sequential-read-buf:            150822                 151371
sequential-read-direct:         408938                 421940
random-read-buf:                3404.9                  3389.1
random-read-direct:             4859.8                 4843.5
sequential-write-buf:            333455                 335776
sequential-write-direct:        44739                   43194
random-write-buf:                7272.1                 7209.6
random-write-direct:             4333.9                 4330.7



root at minggr:~/tmp/test# cat t.job
[global]
size=1G
directory=/mnt/
numjobs=8
group_reporting
runtime=300
time_based
bs=8k
ioengine=libaio
iodepth=64

[sequential-read-buf]
rw=read

[sequential-read-direct]
rw=read
direct=1

[random-read-buf]
rw=randread

[random-read-direct]
rw=randread
direct=1

[sequential-write-buf]
rw=write

[sequential-write-direct]
rw=write
direct=1

[random-write-buf]
rw=randwrite

[random-write-direct]
rw=randwrite
direct=1


root at minggr:~/tmp/test# cat run.sh
#!/bin/bash

jobs="sequential-read-buf sequential-read-direct random-read-buf
random-read-direct"
jobs="$jobs sequential-write-buf sequential-write-direct
random-write-buf random-write-direct"

#each partition is 100G
pvcreate /dev/sdb3 /dev/nvme0n1p1 /dev/sdc6
vgcreate striped_vol_group /dev/sdb3 /dev/nvme0n1p1 /dev/sdc6
lvcreate -i3 -I4 -L250G -nstriped_logical_volume striped_vol_group

for job in $jobs ; do
        umount /mnt > /dev/null 2>&1
        mkfs.xfs -f /dev/striped_vol_group/striped_logical_volume
        mount /dev/striped_vol_group/striped_logical_volume /mnt

        fio --output=${job}.log --section=${job} t.job
done