[DRBD-user] sock_sendmsg and pending_cnt <0

matthias zeichmann matthias at volltext.net
Thu Feb 5 11:56:29 CET 2004

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On Thu, 2004-02-05 at 01:49, Lars Ellenberg wrote:
> / 2004-02-04 16:16:06 +0100
> \ matthias zeichmann:
> > i used the opportunity to upgrade drbd to latest cvs and to switch to
> > protocol C.
>           ^^^ 
> > Feb  3 20:49:29 atem kernel: drbd5: Connection established. size=96358
> > KB / blksize=1024 B
>                    ^^^
>  ??

so whats the proper way to switch protocols? i changed the disk-size and
protocol parameters on StandAlone primary and restarted drbd with
/etc/init.d/drbd restart. i thougt that would suffice... 

> So how did you get the "unknown packet type"?
> TCP checksum is ok, DRBD_MAGIC is ok, and there still is
> an unknown packet type?

only during the fullsync after resizing of partitions; though there were
errors on gigabit interface on the receiving side:
-------->8--------------------------------------------------------
atem:root# ifconfig eth2
eth2      Protocol:Ethernet  Hardware Adress 00:02:B3:B7:A4:C2  
          inet adress:192.168.0.2  Bcast:192.168.0.255              
          netmask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:70627144 errors:24 dropped:24 overruns:24 frame:0
          TX packets:37093641 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:1000 
          RX bytes:1579455730 (1.4 GiB)  TX bytes:541666805 (516.5 MiB)
          Interrupt:27 Basisadresse:0x1c00 Speicher:fd020000-fd040000 

atem:root# 
-------->8--------------------------------------------------------

> This does not feel good, but I doubt that drbd is at fault.
> Better transfer huge amounts of data,
> and verify MD5 SHA1 or whatever finger prints!!

you are talking about e.g. 
md5sum /dev/nb0 # with drbd running and both nodes secondary?
or just fingerprinting big files, change roles and compare?


> Don't come running in three month time,
> telling us "drbd" corrupts your data...
> check for it NOW.


> > Feb  3 21:02:31 kaelte kernel: drbd4: [drbd_syncer_4/27961] sock_sendmsg
> > time expired, ko = 4294967295
> 
> No problem, it gets retried until ko hits zero ...
> ko decrements with every ping intervall,
> but is reset with each transfered block.
> 
> > Feb  3 21:18:50 kaelte kernel: drbd2: [drbd_syncer_2/2439] send timed
> > out!!
> 
> again:
> > Feb  3 21:18:50 atem kernel: drbd2: unknown packet type!
> 
> > i also stumbled across
> > http://thread.gmane.org/gmane.comp.linux.drbd/5571
> > and i understand i have to tune the net sections [2] with my drbd
> > devices. maybe sync-nice=0 is too harsh? i'm just not sure how, cos i am
> > not really satisfied with the sync speed i get.
> 
> For the range of the sync-nice value, see "man nice".
> Normal interactive processes run with priority 0.
> 
> > i use intel gigabit nics on 2.4.24 with rx polling support (mtu 1500)
> > for drbd.
> 
> increase mtu to 5000, if you like.
> 
> > the two nodes are connected via crossover cable and sync speed
> > maxes out at approx. 13MB/s while i know that the disks do better [3]
> > and i know that the network interfaces can do better.
> 
> probably network latency. try sndbuf-size = 65534 
> but if that breaks something, don't bug me ;)
> 
> > another note on synching: during fullsync pe was >> 0 on the sending
> > side and ua >> 0 on the receiving side; is this normal? after sync both
> > coloumns went back to zero.
> 
> absolutely.
> "pending" and "unacknowledged" just means there is data on the fly.
> if they are high, it is much data ;)
> 
> > Feb  4 03:00:05 kaelte kernel: drbd2: pending_cnt <0 !!!
> > 
> > didn't see that for quite some time; last time i saw it it came along
> > with a deadlock on primary :-/
> 
> hm...
> ?

that was drbd 0.6.1 or 0.6.3

> > [4] net throughput [...]
> obviously not your bottleneck.
> 
> check the latency of differently sized packets, though.
> anyone wants to recommend a benchmarking tool for this purpose?
> ping?  bing?

regards m
-------->8--------------------------------------------------------
atem:bin# bing -e 10240 192.168.0.2 192.168.0.3
BING    192.168.0.2 (192.168.0.2) and 192.168.0.3 (192.168.0.3)
    44 and 108 data bytes
1024 bits in 0.000ms
1024 bits in 0.007ms: 146285714bps, 0.000007ms per bit
1024 bits in 0.003ms: 341333333bps, 0.000003ms per bit
1024 bits in 0.005ms: 204800000bps, 0.000005ms per bit
1024 bits in 0.007ms: 146285714bps, 0.000007ms per bit
1024 bits in 0.006ms: 170666667bps, 0.000006ms per bit
1024 bits in 0.008ms: 128000000bps, 0.000008ms per bit
1024 bits in 0.007ms: 146285714bps, 0.000007ms per bit
1024 bits in 0.005ms: 204800000bps, 0.000005ms per bit

--- 192.168.0.2 statistics ---
bytes   out    in   dup  loss   rtt (ms): min       avg       max
   44 10240 10240          0%           0.016     0.021     0.174
  108 10240 10240          0%           0.015     0.017     0.100

--- 192.168.0.3 statistics ---
bytes   out    in   dup  loss   rtt (ms): min       avg       max
   44 10240 10240          0%           0.043     0.058     4.108
  108 10240 10240          0%           0.048     0.066     2.834

--- estimated link characteristics ---
warning: rtt big host1 0.015ms < rtt small host2 0.016ms
estimated throughput 204800000bps
minimum delay per packet 0.024ms (4826 bits)

average statistics (experimental) :
packet loss: small 0%, big 0%, total 0%
warning: rtt big host1 0.017ms < rtt small host2 0.021ms
average throughput 128000000bps
average delay per packet 0.034ms (6966 bits)
weighted average throughput 128000000bps

resetting after 10240 samples.
atem:bin# bing -e 10240 -s 288 -b 768 192.168.0.2 192.168.0.3
BING    192.168.0.2 (192.168.0.2) and 192.168.0.3 (192.168.0.3)
    288 and 768 data bytes
7680 bits in 0.000ms
7680 bits in 0.033ms: 232727273bps, 0.000004ms per bit
7680 bits in 0.036ms: 213333333bps, 0.000005ms per bit
7680 bits in 0.038ms: 202105263bps, 0.000005ms per bit
7680 bits in 0.036ms: 213333333bps, 0.000005ms per bit
7680 bits in 0.032ms: 240000000bps, 0.000004ms per bit
7680 bits in 0.031ms: 247741935bps, 0.000004ms per bit
7680 bits in 0.032ms: 240000000bps, 0.000004ms per bit
7680 bits in 0.025ms: 307200000bps, 0.000003ms per bit
7680 bits in 0.026ms: 295384615bps, 0.000003ms per bit
7680 bits in 0.024ms: 320000000bps, 0.000003ms per bit
7680 bits in 0.023ms: 333913043bps, 0.000003ms per bit
7680 bits in 0.022ms: 349090909bps, 0.000003ms per bit
7680 bits in 0.020ms: 384000000bps, 0.000003ms per bit
7680 bits in 0.021ms: 365714286bps, 0.000003ms per bit
7680 bits in 0.020ms: 384000000bps, 0.000003ms per bit

--- 192.168.0.2 statistics ---
bytes   out    in   dup  loss   rtt (ms): min       avg       max
  288 10240 10240          0%           0.017     0.019     0.135
  768 10240 10240          0%           0.017     0.019     0.063

--- 192.168.0.3 statistics ---
bytes   out    in   dup  loss   rtt (ms): min       avg       max
  288 10240 10240          0%           0.057     0.065     0.263
  768 10240 10240          0%           0.077     0.091     0.209

--- estimated link characteristics ---
estimated throughput 384000000bps
minimum delay per packet 0.028ms (10752 bits)

average statistics (experimental) :
packet loss: small 0%, big 0%, total 0%
average throughput 295384615bps
average delay per packet 0.034ms (13114 bits)
weighted average throughput 295384615bps

resetting after 10240 samples.
atem:bin# bing -e 10240 -s 288 -S 768 192.168.0.2 192.168.0.3
BING    192.168.0.2 (192.168.0.2) and 192.168.0.3 (192.168.0.3)
    1024 and 3768 data bytes
43904 bits in 0.076ms: 577684211bps, 0.000002ms per bit
43904 bits in 0.113ms: 388530973bps, 0.000003ms per bit
43904 bits in 0.115ms: 381773913bps, 0.000003ms per bit
43904 bits in 0.105ms: 418133333bps, 0.000002ms per bit
43904 bits in 0.104ms: 422153846bps, 0.000002ms per bit
43904 bits in 0.087ms: 504643678bps, 0.000002ms per bit
43904 bits in 0.086ms: 510511628bps, 0.000002ms per bit
43904 bits in 0.085ms: 516517647bps, 0.000002ms per bit
43904 bits in 0.084ms: 522666667bps, 0.000002ms per bit
43904 bits in 0.085ms: 516517647bps, 0.000002ms per bit
43904 bits in 0.076ms: 577684211bps, 0.000002ms per bit
43904 bits in 0.077ms: 570181818bps, 0.000002ms per bit
43904 bits in 0.078ms: 562871795bps, 0.000002ms per bit
43904 bits in 0.079ms: 555746835bps, 0.000002ms per bit
43904 bits in 0.077ms: 570181818bps, 0.000002ms per bit
43904 bits in 0.078ms: 562871795bps, 0.000002ms per bit
43904 bits in 0.077ms: 570181818bps, 0.000002ms per bit

--- 192.168.0.2 statistics ---
bytes   out    in   dup  loss   rtt (ms): min       avg       max
 1024 10240 10240          0%           0.021     0.024     0.213
 3768 10240 10240          0%           0.029     0.036     0.191

--- 192.168.0.3 statistics ---
bytes   out    in   dup  loss   rtt (ms): min       avg       max
 1024 10240 10240          0%           0.087     0.101     9.744
 3768 10240 10240          0%           0.172     0.200     2.230

--- estimated link characteristics ---
estimated throughput 570181818bps
minimum delay per packet 0.037ms (21248 bits)

average statistics (experimental) :
packet loss: small 0%, big 0%, total 0%
average throughput 504643678bps
average delay per packet 0.048ms (27113 bits)
weighted average throughput 504643678bps

resetting after 10240 samples.
-------->8--------------------------------------------------------
atem:bin# for i in $(seq 0 5); do   ping kaelte_priv -s $[1024 * 2 **
$i] -c 10 | grep -E '^(PING|round-trip|$)|packets transmitted'; echo;
done
PING kaelte_priv (192.168.0.3): 1024 data bytes

10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.0/0.0/0.1 ms

PING kaelte_priv (192.168.0.3): 2048 data bytes

10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.1/0.2 ms

PING kaelte_priv (192.168.0.3): 4096 data bytes

10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.2/0.2/0.3 ms

PING kaelte_priv (192.168.0.3): 8192 data bytes

10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.3/0.3/0.4 ms

PING kaelte_priv (192.168.0.3): 16384 data bytes

10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.6/0.6/0.7 ms

PING kaelte_priv (192.168.0.3): 32768 data bytes

10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 1.1/1.1/1.3 ms

atem:bin# 
-------->8--------------------------------------------------------
atem:src# ttcp -s -r -l 1024 -n10240
ttcp-r: buflen=1024, nbuf=10240, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.13 real seconds = 79996.88 KB/sec +++
ttcp-r: 11155 I/O calls, msec/call = 0.01, calls/sec = 87145.03
ttcp-r: 0.0user 0.1sys 0:00real 91% 0i+0d 0maxrss 0+1pf 0+0csw
atem:src# ttcp -s -r -l 2048 -n 5120
ttcp-r: buflen=2048, nbuf=5120, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.12 real seconds = 85870.74 KB/sec +++
ttcp-r: 5789 I/O calls, msec/call = 0.02, calls/sec = 48545.48
ttcp-r: 0.0user 0.0sys 0:00real 81% 0i+0d 0maxrss 0+1pf 0+0csw
atem:src# ttcp -s -r -l 4096 -n 2560
ttcp-r: buflen=4096, nbuf=2560, align=16384/0, port=5001  tcp
ttcp-r: socket
#ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.11 real seconds = 93931.17 KB/sec +++
ttcp-r: 3144 I/O calls, msec/call = 0.04, calls/sec = 28839.80
ttcp-r: 0.0user 0.1sys 0:00real 110% 0i+0d 0maxrss 0+1pf 0+0csw
atem:src# ttcp -s -r -l  8192 -n 1280
ttcp-r: buflen=8192, nbuf=1280, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.10 real seconds = 98433.14 KB/sec +++
ttcp-r: 2270 I/O calls, msec/call = 0.05, calls/sec = 21820.63
ttcp-r: 0.0user 0.0sys 0:00real 80% 0i+0d 0maxrss 0+2pf 0+0csw
atem:src# ttcp -s -r -l 16384 -n 640 
ttcp-r: buflen=16384, nbuf=640, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.10 real seconds = 106408.40 KB/sec +++
ttcp-r: 1613 I/O calls, msec/call = 0.06, calls/sec = 16761.40
ttcp-r: 0.0user 0.0sys 0:00real 100% 0i+0d 0maxrss 0+4pf 0+0csw
atem:src# ttcp -s -r -l 32768 -n 320
ttcp-r: buflen=32768, nbuf=320, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.11 real seconds = 91464.50 KB/sec +++
ttcp-r: 1895 I/O calls, msec/call = 0.06, calls/sec = 16926.29
ttcp-r: 0.0user 0.0sys 0:00real 72% 0i+0d 0maxrss 0+8pf 0+0csw
atem:src# ttcp -s -r -l 65536 -n 160
ttcp-r: buflen=65536, nbuf=160, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.09 real seconds = 113219.23 KB/sec +++
ttcp-r: 1467 I/O calls, msec/call = 0.06, calls/sec = 16219.98
ttcp-r: 0.0user 0.0sys 0:00real 88% 0i+0d 0maxrss 0+16pf 0+0csw
atem:src# 
-------->8--------------------------------------------------------



-- 
Greetings to you Echelon guys...everybody say cheese...*SNAP*




More information about the drbd-user mailing list