[DRBD-user] Low performance over NFS after upgrade from 0.7.25 to 8.2.7

Håkan Engblom zyber_cynic at hotmail.com
Thu Nov 4 13:55:56 CET 2010


Hi drbd-users,
 
We use DRBD on a system with two fileservers. We have three replicated DRBD-partitions, and on top of that we use ext3 filesystem, and these are exported over NFS,
pretty basic setup. NFS-server is in kernel. Linux kernel is 2.6.27.39, running on a x86_64 system. C-protocol for drbd.
 
Picture of setup:
 
Primary server:                  Secondary server:                         NFS-client:
 
 
 
    |.........................................................................|
    |                                                                         |
NFS-server                                                                 NFS-cleint
    |
ext3-filesystem
    |
drbd-partition ................. drbd-partition
    |                                    |
/dev/sda? (sas-disk)             /dev/sda? (disk identical to the primary)
 
 
 
The problem we have is that after the upgrade to 8.2.7, the performance, when creating many small files from an NFS-client, has decreased drastically. 
When using 0.7.25 it takes 1:20 (1 minute 20 seconds) to do "pax -r -f 10000_files_each_5k.tar" on the NFS-client, over NFS and to a drbd-partition.
The time for the same test when using 8.2.7 is 19:22, nothin except drbd is changed. Config for DRBD is somwhat changed when going from 0.7.25 to 8.2.7,
but I cannot see those changes should cause such a performance degradation.
 
If the same tar-archive is unpacked directly on the Primary server, not using NFS but still on a drbd-partition, it takes 0.3 seconds, 
regardless of drbd-version.
 
When doing the tests above, all drbd-partitions are consistent and up to date.
 
If the test is done from an NFS-client to a drbd-partition, but the secondary FSB is removed, so no replication has to be performed,
the time is app. 40 seconds regardless of DRBD-version.
 
If we unpack the same tar-archive over NFS but to a "non-drbd" partition (ordinary ext3 on top of a /dev/sda? partition),
the test takes app. 40 seconds (regardless of DRBD-version, of course, since it is not used)
 
So the problem is only seen when NFS and drbd are combined, that is, NFS-server/client on top of a drbd-partition.
 
At first I thought it was some buffer/cache that ran full, and that the problem occurd after that, some queing problem,
but when doing a packet capture of the NFS-traffic on the primary server, it shows that the problem is there already from the
very first file created during the "pax-operation".
 
This is a snippet of the opacket-capture, with some RTT-statistics added (the right-most column is the time since previous packet in micro seconds):
 
DRBD 8.2.7

















10.4.149.50
->
10.4.145.13
NFS
V3
LOOKUP
Call,
DH:0x1bd94dba/file.47









10.4.145.13
->
10.4.149.50
NFS
V3
LOOKUP
Reply
(Call
In
11358)
Error:NFS3ERR_NOENT




32

10.4.149.50
->
10.4.145.13
NFS
V3
CREATE
Call,
DH:0x1bd94dba/file.47
Mode:EXCLUSIVE






250

10.4.145.13
->
10.4.149.50
NFS
V3
CREATE
Reply
(Call
In
11360)





35572

10.4.149.50
->
10.4.145.13
NFS
V3
SETATTR
Call,
FH:0x43dd169c







349

10.4.145.13
->
10.4.149.50
NFS
V3
SETATTR
Reply
(Call
In
11364)





23581

10.4.149.50
->
10.4.145.13
NFS
V3
GETATTR
Call,
FH:0x43dd169c







279

10.4.145.13
->
10.4.149.50
NFS
V3
GETATTR
Reply
(Call
In
11366)
Regular
File
mode:0644
uid:0
gid:0
30

10.4.149.50
->
10.4.145.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=0)





207

10.4.149.50
->
10.4.145.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=1480)





4

10.4.149.50
->
10.4.145.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=2960)





9

10.4.149.50
->
10.4.145.13
NFS
V3
WRITE
Call,
FH:0x43dd169c
Offset:0
Len:5000
UNSTABLE




12

10.4.145.13
->
10.4.149.50
NFS
V3
WRITE
Reply
(Call
In
11371)
Len:5000
UNSTABLE


65

10.4.149.50
->
10.4.145.13
NFS
V3
COMMIT
Call,
FH:0x43dd169c







219

10.4.145.13
->
10.4.149.50
NFS
V3
COMMIT
Reply
(Call
In
11373)





41264

10.4.149.50
->
10.4.145.13
NFS
V3
ACCESS
Call,
FH:0x1bd94dba







395

10.4.145.13
->
10.4.149.50
NFS
V3
ACCESS
Reply
(Call
In
11375)





21

10.4.149.50
->
10.4.145.13
NFS
V3
SETATTR
Call,
FH:0x43dd169c







174

10.4.145.13
->
10.4.149.50
NFS
V3
SETATTR
Reply
(Call
In
11377)





17469



































DRBD 0.7.25
















10.4.56.18
->
10.4.49.13
NFS
V3
LOOKUP
Call,
DH:0x6ee6e4ba/file.108









10.4.49.13
->
10.4.56.18
NFS
V3
LOOKUP
Reply
(Call
In
237)
Error:NFS3ERR_NOENT




31

10.4.56.18
->
10.4.49.13
NFS
V3
CREATE
Call,
DH:0x6ee6e4ba/file.108
Mode:EXCLUSIVE






94

10.4.49.13
->
10.4.56.18
NFS
V3
CREATE
Reply
(Call
In
239)





2202

10.4.56.18
->
10.4.49.13
NFS
V3
SETATTR
Call,
FH:0x7919d20a







178

10.4.49.13
->
10.4.56.18
NFS
V3
SETATTR
Reply
(Call
In
241)





1004

10.4.56.18
->
10.4.49.13
NFS
V3
GETATTR
Call,
FH:0x7919d20a







250

10.4.49.13
->
10.4.56.18
NFS
V3
GETATTR
Reply
(Call
In
243)
Regular
File
mode:0644
uid:0
gid:0
22

10.4.56.18
->
10.4.49.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=0)





222

10.4.56.18
->
10.4.49.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=1480)





4

10.4.56.18
->
10.4.49.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=2960)





2

10.4.56.18
->
10.4.49.13
NFS
V3
WRITE
Call,
FH:0x7919d20a
Offset:0
Len:5000
UNSTABLE




2

10.4.49.13
->
10.4.56.18
NFS
V3
WRITE
Reply
(Call
In
248)
Len:5000
UNSTABLE


59

10.4.56.18
->
10.4.49.13
NFS
V3
COMMIT
Call,
FH:0x7919d20a







104

10.4.49.13
->
10.4.56.18
NFS
V3
COMMIT
Reply
(Call
In
250)





1903

10.4.56.18
->
10.4.49.13
NFS
V3
ACCESS
Call,
FH:0x6ee6e4ba







273

10.4.49.13
->
10.4.56.18
NFS
V3
ACCESS
Reply
(Call
In
252)





30

10.4.56.18
->
10.4.49.13
NFS
V3
SETATTR
Call,
FH:0x7919d20a







210

10.4.49.13
->
10.4.56.18
NFS
V3
SETATTR
Reply
(Call
In
254)





1060
 
 
So it is clear that the RTT for CREATE, SETATTR and COMMIT is _very_ much higher on 8.2.7 than on 0.7.25
 
None of the NFS-settings are changed when DRBD 8.2.7 is used compared with when DRBD 0.7.25 is used.
 
The config files for drbd and for NFS are attached.
 
Has anyone seen anything similar ? Any clues ?
 
br Håkan E. 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20101104/a82c778a/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: attachments1.tar.gz
Type: application/x-gzip
Size: 8984 bytes
Desc: not available
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20101104/a82c778a/attachment-0001.bin>


More information about the drbd-user mailing list