[DRBD-user] Massive starvation in diskless state doing direct IO reads

[Roland Friedwagner]

Am Freitag 03 September 2010 schrieb Lars Ellenberg:
> >
> > Starvation is gone with disabled sendpage feature!
> >
> > [secondary]# cat /sys/module/drbd/parameters/disable_sendpage
> > Y
> > [diskless primary]# dd if=/dev/drbd0 of=/dev/null iflag=direct
> > bs=9M count=50 50+0 records in
> > 50+0 records out
> > 471859200 bytes (472 MB) copied, 4.91869 seconds, 95.9 MB/s
> >
> > I did not see any performance degradation not using sendpage
> > with 1Gbit Link between Primary and Secondary (works like a
> > charm;-).
> >
> > Does using sendpage makes a difference when it comes to >= 10Gbit
> > Links?
> >
> > > As long as you have _occasional_ write requests while you do the
> > > sequential read, you should still be fine, too.
> >
> > Doing writes during big direct io reads does _not_ fix it:
> >
> > I run this in background to produce about 10 writes/sec:
> >
> >   while :; do date --iso-8601=sec; dd if=/dev/zero of=/dev/drbd0
> > bs=1k count=1 conv=fsync 2> /dev/null; sleep .1; done
> >
> > But direct io reads are still stucked (and the writes also get
> > stucked):
>
> Now that you mention it, that makes sense, considering where the hang
> occurs (receiver trying to allocate buffer pages for the read...).
>
> please try below patch, and see how much it improves the situation.
>
> diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c
> index 33e0541..a60ebfa 100644
> --- a/drbd/drbd_receiver.c
> +++ b/drbd/drbd_receiver.c
> @@ -299,7 +299,11 @@ STATIC struct page *drbd_pp_alloc(struct
> drbd_conf *mdev, unsigned number, bool break;
>  		}
>
> -		schedule();
> +		/* If everything is on the net_ee (still referenced by tcp),
> +		 * there won't be a wake_up() until the next process_done_ee
> +		 * which may be ping-int (typically 10 seconds) away.
> +		 * Retry ourselves a bit faster. */
> +		schedule_timeout(HZ/10);
>  	}
>  	finish_wait(&drbd_pp_wait, &wait);


Great!!! - I applied you patch and direct io read performance is normal now:

# dd if=/dev/drbd0 of=/dev/null iflag=direct bs=7M count=50
50+0 records in
50+0 records out
367001600 bytes (367 MB) copied, 4.42229 seconds, 83.0 MB/s
# dd if=/dev/drbd0 of=/dev/null iflag=direct bs=8M count=50
50+0 records in
50+0 records out
419430400 bytes (419 MB) copied, 4.38405 seconds, 95.7 MB/s
# dd if=/dev/drbd0 of=/dev/null iflag=direct bs=9M count=50
50+0 records in
50+0 records out
471859200 bytes (472 MB) copied, 5.59782 seconds, 84.3 MB/s
# dd if=/dev/drbd0 of=/dev/null iflag=direct bs=10M count=50
50+0 records in
50+0 records out
524288000 bytes (524 MB) copied, 6.56386 seconds, 79.9 MB/s


But disable sendpage does outperfom these results:

# dd if=/dev/drbd0 of=/dev/null iflag=direct bs=16M count=50
50+0 records in
50+0 records out
838860800 bytes (839 MB) copied, 10.8062 seconds, 77.6 MB/s
# echo 1 > /sys/module/drbd/parameters/disable_sendpage
# dd if=/dev/drbd0 of=/dev/null iflag=direct bs=16M count=50
50+0 records in
50+0 records out
838860800 bytes (839 MB) copied, 8.23132 seconds, 102 MB/s

ThxALot & Kind Regards,
Roland

-- 
Roland.Friedwagner at wu.ac.at            Phone: +43 1 31336 5377
IT Services - WU (Vienna University of Economics and Business)