Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Am Freitag 03 September 2010 schrieb Lars Ellenberg: > > > > Starvation is gone with disabled sendpage feature! > > > > [secondary]# cat /sys/module/drbd/parameters/disable_sendpage > > Y > > [diskless primary]# dd if=/dev/drbd0 of=/dev/null iflag=direct > > bs=9M count=50 50+0 records in > > 50+0 records out > > 471859200 bytes (472 MB) copied, 4.91869 seconds, 95.9 MB/s > > > > I did not see any performance degradation not using sendpage > > with 1Gbit Link between Primary and Secondary (works like a > > charm;-). > > > > Does using sendpage makes a difference when it comes to >= 10Gbit > > Links? > > > > > As long as you have _occasional_ write requests while you do the > > > sequential read, you should still be fine, too. > > > > Doing writes during big direct io reads does _not_ fix it: > > > > I run this in background to produce about 10 writes/sec: > > > > while :; do date --iso-8601=sec; dd if=/dev/zero of=/dev/drbd0 > > bs=1k count=1 conv=fsync 2> /dev/null; sleep .1; done > > > > But direct io reads are still stucked (and the writes also get > > stucked): > > Now that you mention it, that makes sense, considering where the hang > occurs (receiver trying to allocate buffer pages for the read...). > > please try below patch, and see how much it improves the situation. > > diff --git a/drbd/drbd_receiver.c b/drbd/drbd_receiver.c > index 33e0541..a60ebfa 100644 > --- a/drbd/drbd_receiver.c > +++ b/drbd/drbd_receiver.c > @@ -299,7 +299,11 @@ STATIC struct page *drbd_pp_alloc(struct > drbd_conf *mdev, unsigned number, bool break; > } > > - schedule(); > + /* If everything is on the net_ee (still referenced by tcp), > + * there won't be a wake_up() until the next process_done_ee > + * which may be ping-int (typically 10 seconds) away. > + * Retry ourselves a bit faster. */ > + schedule_timeout(HZ/10); > } > finish_wait(&drbd_pp_wait, &wait); Great!!! - I applied you patch and direct io read performance is normal now: # dd if=/dev/drbd0 of=/dev/null iflag=direct bs=7M count=50 50+0 records in 50+0 records out 367001600 bytes (367 MB) copied, 4.42229 seconds, 83.0 MB/s # dd if=/dev/drbd0 of=/dev/null iflag=direct bs=8M count=50 50+0 records in 50+0 records out 419430400 bytes (419 MB) copied, 4.38405 seconds, 95.7 MB/s # dd if=/dev/drbd0 of=/dev/null iflag=direct bs=9M count=50 50+0 records in 50+0 records out 471859200 bytes (472 MB) copied, 5.59782 seconds, 84.3 MB/s # dd if=/dev/drbd0 of=/dev/null iflag=direct bs=10M count=50 50+0 records in 50+0 records out 524288000 bytes (524 MB) copied, 6.56386 seconds, 79.9 MB/s But disable sendpage does outperfom these results: # dd if=/dev/drbd0 of=/dev/null iflag=direct bs=16M count=50 50+0 records in 50+0 records out 838860800 bytes (839 MB) copied, 10.8062 seconds, 77.6 MB/s # echo 1 > /sys/module/drbd/parameters/disable_sendpage # dd if=/dev/drbd0 of=/dev/null iflag=direct bs=16M count=50 50+0 records in 50+0 records out 838860800 bytes (839 MB) copied, 8.23132 seconds, 102 MB/s ThxALot & Kind Regards, Roland -- Roland.Friedwagner at wu.ac.at Phone: +43 1 31336 5377 IT Services - WU (Vienna University of Economics and Business)