Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi drbd-users,
We use DRBD on a system with two fileservers. We have three replicated DRBD-partitions, and on top of that we use ext3 filesystem, and these are exported over NFS,
pretty basic setup. NFS-server is in kernel. Linux kernel is 2.6.27.39, running on a x86_64 system. C-protocol for drbd.
Picture of setup:
Primary server: Secondary server: NFS-client:
|.........................................................................|
| |
NFS-server NFS-cleint
|
ext3-filesystem
|
drbd-partition ................. drbd-partition
| |
/dev/sda? (sas-disk) /dev/sda? (disk identical to the primary)
The problem we have is that after the upgrade to 8.2.7, the performance, when creating many small files from an NFS-client, has decreased drastically.
When using 0.7.25 it takes 1:20 (1 minute 20 seconds) to do "pax -r -f 10000_files_each_5k.tar" on the NFS-client, over NFS and to a drbd-partition.
The time for the same test when using 8.2.7 is 19:22, nothin except drbd is changed. Config for DRBD is somwhat changed when going from 0.7.25 to 8.2.7,
but I cannot see those changes should cause such a performance degradation.
If the same tar-archive is unpacked directly on the Primary server, not using NFS but still on a drbd-partition, it takes 0.3 seconds,
regardless of drbd-version.
When doing the tests above, all drbd-partitions are consistent and up to date.
If the test is done from an NFS-client to a drbd-partition, but the secondary FSB is removed, so no replication has to be performed,
the time is app. 40 seconds regardless of DRBD-version.
If we unpack the same tar-archive over NFS but to a "non-drbd" partition (ordinary ext3 on top of a /dev/sda? partition),
the test takes app. 40 seconds (regardless of DRBD-version, of course, since it is not used)
So the problem is only seen when NFS and drbd are combined, that is, NFS-server/client on top of a drbd-partition.
At first I thought it was some buffer/cache that ran full, and that the problem occurd after that, some queing problem,
but when doing a packet capture of the NFS-traffic on the primary server, it shows that the problem is there already from the
very first file created during the "pax-operation".
This is a snippet of the opacket-capture, with some RTT-statistics added (the right-most column is the time since previous packet in micro seconds):
DRBD 8.2.7
10.4.149.50
->
10.4.145.13
NFS
V3
LOOKUP
Call,
DH:0x1bd94dba/file.47
10.4.145.13
->
10.4.149.50
NFS
V3
LOOKUP
Reply
(Call
In
11358)
Error:NFS3ERR_NOENT
32
10.4.149.50
->
10.4.145.13
NFS
V3
CREATE
Call,
DH:0x1bd94dba/file.47
Mode:EXCLUSIVE
250
10.4.145.13
->
10.4.149.50
NFS
V3
CREATE
Reply
(Call
In
11360)
35572
10.4.149.50
->
10.4.145.13
NFS
V3
SETATTR
Call,
FH:0x43dd169c
349
10.4.145.13
->
10.4.149.50
NFS
V3
SETATTR
Reply
(Call
In
11364)
23581
10.4.149.50
->
10.4.145.13
NFS
V3
GETATTR
Call,
FH:0x43dd169c
279
10.4.145.13
->
10.4.149.50
NFS
V3
GETATTR
Reply
(Call
In
11366)
Regular
File
mode:0644
uid:0
gid:0
30
10.4.149.50
->
10.4.145.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=0)
207
10.4.149.50
->
10.4.145.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=1480)
4
10.4.149.50
->
10.4.145.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=2960)
9
10.4.149.50
->
10.4.145.13
NFS
V3
WRITE
Call,
FH:0x43dd169c
Offset:0
Len:5000
UNSTABLE
12
10.4.145.13
->
10.4.149.50
NFS
V3
WRITE
Reply
(Call
In
11371)
Len:5000
UNSTABLE
65
10.4.149.50
->
10.4.145.13
NFS
V3
COMMIT
Call,
FH:0x43dd169c
219
10.4.145.13
->
10.4.149.50
NFS
V3
COMMIT
Reply
(Call
In
11373)
41264
10.4.149.50
->
10.4.145.13
NFS
V3
ACCESS
Call,
FH:0x1bd94dba
395
10.4.145.13
->
10.4.149.50
NFS
V3
ACCESS
Reply
(Call
In
11375)
21
10.4.149.50
->
10.4.145.13
NFS
V3
SETATTR
Call,
FH:0x43dd169c
174
10.4.145.13
->
10.4.149.50
NFS
V3
SETATTR
Reply
(Call
In
11377)
17469
DRBD 0.7.25
10.4.56.18
->
10.4.49.13
NFS
V3
LOOKUP
Call,
DH:0x6ee6e4ba/file.108
10.4.49.13
->
10.4.56.18
NFS
V3
LOOKUP
Reply
(Call
In
237)
Error:NFS3ERR_NOENT
31
10.4.56.18
->
10.4.49.13
NFS
V3
CREATE
Call,
DH:0x6ee6e4ba/file.108
Mode:EXCLUSIVE
94
10.4.49.13
->
10.4.56.18
NFS
V3
CREATE
Reply
(Call
In
239)
2202
10.4.56.18
->
10.4.49.13
NFS
V3
SETATTR
Call,
FH:0x7919d20a
178
10.4.49.13
->
10.4.56.18
NFS
V3
SETATTR
Reply
(Call
In
241)
1004
10.4.56.18
->
10.4.49.13
NFS
V3
GETATTR
Call,
FH:0x7919d20a
250
10.4.49.13
->
10.4.56.18
NFS
V3
GETATTR
Reply
(Call
In
243)
Regular
File
mode:0644
uid:0
gid:0
22
10.4.56.18
->
10.4.49.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=0)
222
10.4.56.18
->
10.4.49.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=1480)
4
10.4.56.18
->
10.4.49.13
IP
Fragmented
IP
protocol
(proto=UDP
0x11,
off=2960)
2
10.4.56.18
->
10.4.49.13
NFS
V3
WRITE
Call,
FH:0x7919d20a
Offset:0
Len:5000
UNSTABLE
2
10.4.49.13
->
10.4.56.18
NFS
V3
WRITE
Reply
(Call
In
248)
Len:5000
UNSTABLE
59
10.4.56.18
->
10.4.49.13
NFS
V3
COMMIT
Call,
FH:0x7919d20a
104
10.4.49.13
->
10.4.56.18
NFS
V3
COMMIT
Reply
(Call
In
250)
1903
10.4.56.18
->
10.4.49.13
NFS
V3
ACCESS
Call,
FH:0x6ee6e4ba
273
10.4.49.13
->
10.4.56.18
NFS
V3
ACCESS
Reply
(Call
In
252)
30
10.4.56.18
->
10.4.49.13
NFS
V3
SETATTR
Call,
FH:0x7919d20a
210
10.4.49.13
->
10.4.56.18
NFS
V3
SETATTR
Reply
(Call
In
254)
1060
So it is clear that the RTT for CREATE, SETATTR and COMMIT is _very_ much higher on 8.2.7 than on 0.7.25
None of the NFS-settings are changed when DRBD 8.2.7 is used compared with when DRBD 0.7.25 is used.
The config files for drbd and for NFS are attached.
Has anyone seen anything similar ? Any clues ?
br Håkan E.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20101104/a82c778a/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: attachments1.tar.gz
Type: application/x-gzip
Size: 8984 bytes
Desc: not available
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20101104/a82c778a/attachment.bin>