Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Thu, 2004-02-05 at 01:49, Lars Ellenberg wrote:
> / 2004-02-04 16:16:06 +0100
> \ matthias zeichmann:
> > i used the opportunity to upgrade drbd to latest cvs and to switch to
> > protocol C.
> ^^^
> > Feb 3 20:49:29 atem kernel: drbd5: Connection established. size=96358
> > KB / blksize=1024 B
> ^^^
> ??
so whats the proper way to switch protocols? i changed the disk-size and
protocol parameters on StandAlone primary and restarted drbd with
/etc/init.d/drbd restart. i thougt that would suffice...
> So how did you get the "unknown packet type"?
> TCP checksum is ok, DRBD_MAGIC is ok, and there still is
> an unknown packet type?
only during the fullsync after resizing of partitions; though there were
errors on gigabit interface on the receiving side:
-------->8--------------------------------------------------------
atem:root# ifconfig eth2
eth2 Protocol:Ethernet Hardware Adress 00:02:B3:B7:A4:C2
inet adress:192.168.0.2 Bcast:192.168.0.255
netmask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:70627144 errors:24 dropped:24 overruns:24 frame:0
TX packets:37093641 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:1579455730 (1.4 GiB) TX bytes:541666805 (516.5 MiB)
Interrupt:27 Basisadresse:0x1c00 Speicher:fd020000-fd040000
atem:root#
-------->8--------------------------------------------------------
> This does not feel good, but I doubt that drbd is at fault.
> Better transfer huge amounts of data,
> and verify MD5 SHA1 or whatever finger prints!!
you are talking about e.g.
md5sum /dev/nb0 # with drbd running and both nodes secondary?
or just fingerprinting big files, change roles and compare?
> Don't come running in three month time,
> telling us "drbd" corrupts your data...
> check for it NOW.
> > Feb 3 21:02:31 kaelte kernel: drbd4: [drbd_syncer_4/27961] sock_sendmsg
> > time expired, ko = 4294967295
>
> No problem, it gets retried until ko hits zero ...
> ko decrements with every ping intervall,
> but is reset with each transfered block.
>
> > Feb 3 21:18:50 kaelte kernel: drbd2: [drbd_syncer_2/2439] send timed
> > out!!
>
> again:
> > Feb 3 21:18:50 atem kernel: drbd2: unknown packet type!
>
> > i also stumbled across
> > http://thread.gmane.org/gmane.comp.linux.drbd/5571
> > and i understand i have to tune the net sections [2] with my drbd
> > devices. maybe sync-nice=0 is too harsh? i'm just not sure how, cos i am
> > not really satisfied with the sync speed i get.
>
> For the range of the sync-nice value, see "man nice".
> Normal interactive processes run with priority 0.
>
> > i use intel gigabit nics on 2.4.24 with rx polling support (mtu 1500)
> > for drbd.
>
> increase mtu to 5000, if you like.
>
> > the two nodes are connected via crossover cable and sync speed
> > maxes out at approx. 13MB/s while i know that the disks do better [3]
> > and i know that the network interfaces can do better.
>
> probably network latency. try sndbuf-size = 65534
> but if that breaks something, don't bug me ;)
>
> > another note on synching: during fullsync pe was >> 0 on the sending
> > side and ua >> 0 on the receiving side; is this normal? after sync both
> > coloumns went back to zero.
>
> absolutely.
> "pending" and "unacknowledged" just means there is data on the fly.
> if they are high, it is much data ;)
>
> > Feb 4 03:00:05 kaelte kernel: drbd2: pending_cnt <0 !!!
> >
> > didn't see that for quite some time; last time i saw it it came along
> > with a deadlock on primary :-/
>
> hm...
> ?
that was drbd 0.6.1 or 0.6.3
> > [4] net throughput [...]
> obviously not your bottleneck.
>
> check the latency of differently sized packets, though.
> anyone wants to recommend a benchmarking tool for this purpose?
> ping? bing?
regards m
-------->8--------------------------------------------------------
atem:bin# bing -e 10240 192.168.0.2 192.168.0.3
BING 192.168.0.2 (192.168.0.2) and 192.168.0.3 (192.168.0.3)
44 and 108 data bytes
1024 bits in 0.000ms
1024 bits in 0.007ms: 146285714bps, 0.000007ms per bit
1024 bits in 0.003ms: 341333333bps, 0.000003ms per bit
1024 bits in 0.005ms: 204800000bps, 0.000005ms per bit
1024 bits in 0.007ms: 146285714bps, 0.000007ms per bit
1024 bits in 0.006ms: 170666667bps, 0.000006ms per bit
1024 bits in 0.008ms: 128000000bps, 0.000008ms per bit
1024 bits in 0.007ms: 146285714bps, 0.000007ms per bit
1024 bits in 0.005ms: 204800000bps, 0.000005ms per bit
--- 192.168.0.2 statistics ---
bytes out in dup loss rtt (ms): min avg max
44 10240 10240 0% 0.016 0.021 0.174
108 10240 10240 0% 0.015 0.017 0.100
--- 192.168.0.3 statistics ---
bytes out in dup loss rtt (ms): min avg max
44 10240 10240 0% 0.043 0.058 4.108
108 10240 10240 0% 0.048 0.066 2.834
--- estimated link characteristics ---
warning: rtt big host1 0.015ms < rtt small host2 0.016ms
estimated throughput 204800000bps
minimum delay per packet 0.024ms (4826 bits)
average statistics (experimental) :
packet loss: small 0%, big 0%, total 0%
warning: rtt big host1 0.017ms < rtt small host2 0.021ms
average throughput 128000000bps
average delay per packet 0.034ms (6966 bits)
weighted average throughput 128000000bps
resetting after 10240 samples.
atem:bin# bing -e 10240 -s 288 -b 768 192.168.0.2 192.168.0.3
BING 192.168.0.2 (192.168.0.2) and 192.168.0.3 (192.168.0.3)
288 and 768 data bytes
7680 bits in 0.000ms
7680 bits in 0.033ms: 232727273bps, 0.000004ms per bit
7680 bits in 0.036ms: 213333333bps, 0.000005ms per bit
7680 bits in 0.038ms: 202105263bps, 0.000005ms per bit
7680 bits in 0.036ms: 213333333bps, 0.000005ms per bit
7680 bits in 0.032ms: 240000000bps, 0.000004ms per bit
7680 bits in 0.031ms: 247741935bps, 0.000004ms per bit
7680 bits in 0.032ms: 240000000bps, 0.000004ms per bit
7680 bits in 0.025ms: 307200000bps, 0.000003ms per bit
7680 bits in 0.026ms: 295384615bps, 0.000003ms per bit
7680 bits in 0.024ms: 320000000bps, 0.000003ms per bit
7680 bits in 0.023ms: 333913043bps, 0.000003ms per bit
7680 bits in 0.022ms: 349090909bps, 0.000003ms per bit
7680 bits in 0.020ms: 384000000bps, 0.000003ms per bit
7680 bits in 0.021ms: 365714286bps, 0.000003ms per bit
7680 bits in 0.020ms: 384000000bps, 0.000003ms per bit
--- 192.168.0.2 statistics ---
bytes out in dup loss rtt (ms): min avg max
288 10240 10240 0% 0.017 0.019 0.135
768 10240 10240 0% 0.017 0.019 0.063
--- 192.168.0.3 statistics ---
bytes out in dup loss rtt (ms): min avg max
288 10240 10240 0% 0.057 0.065 0.263
768 10240 10240 0% 0.077 0.091 0.209
--- estimated link characteristics ---
estimated throughput 384000000bps
minimum delay per packet 0.028ms (10752 bits)
average statistics (experimental) :
packet loss: small 0%, big 0%, total 0%
average throughput 295384615bps
average delay per packet 0.034ms (13114 bits)
weighted average throughput 295384615bps
resetting after 10240 samples.
atem:bin# bing -e 10240 -s 288 -S 768 192.168.0.2 192.168.0.3
BING 192.168.0.2 (192.168.0.2) and 192.168.0.3 (192.168.0.3)
1024 and 3768 data bytes
43904 bits in 0.076ms: 577684211bps, 0.000002ms per bit
43904 bits in 0.113ms: 388530973bps, 0.000003ms per bit
43904 bits in 0.115ms: 381773913bps, 0.000003ms per bit
43904 bits in 0.105ms: 418133333bps, 0.000002ms per bit
43904 bits in 0.104ms: 422153846bps, 0.000002ms per bit
43904 bits in 0.087ms: 504643678bps, 0.000002ms per bit
43904 bits in 0.086ms: 510511628bps, 0.000002ms per bit
43904 bits in 0.085ms: 516517647bps, 0.000002ms per bit
43904 bits in 0.084ms: 522666667bps, 0.000002ms per bit
43904 bits in 0.085ms: 516517647bps, 0.000002ms per bit
43904 bits in 0.076ms: 577684211bps, 0.000002ms per bit
43904 bits in 0.077ms: 570181818bps, 0.000002ms per bit
43904 bits in 0.078ms: 562871795bps, 0.000002ms per bit
43904 bits in 0.079ms: 555746835bps, 0.000002ms per bit
43904 bits in 0.077ms: 570181818bps, 0.000002ms per bit
43904 bits in 0.078ms: 562871795bps, 0.000002ms per bit
43904 bits in 0.077ms: 570181818bps, 0.000002ms per bit
--- 192.168.0.2 statistics ---
bytes out in dup loss rtt (ms): min avg max
1024 10240 10240 0% 0.021 0.024 0.213
3768 10240 10240 0% 0.029 0.036 0.191
--- 192.168.0.3 statistics ---
bytes out in dup loss rtt (ms): min avg max
1024 10240 10240 0% 0.087 0.101 9.744
3768 10240 10240 0% 0.172 0.200 2.230
--- estimated link characteristics ---
estimated throughput 570181818bps
minimum delay per packet 0.037ms (21248 bits)
average statistics (experimental) :
packet loss: small 0%, big 0%, total 0%
average throughput 504643678bps
average delay per packet 0.048ms (27113 bits)
weighted average throughput 504643678bps
resetting after 10240 samples.
-------->8--------------------------------------------------------
atem:bin# for i in $(seq 0 5); do ping kaelte_priv -s $[1024 * 2 **
$i] -c 10 | grep -E '^(PING|round-trip|$)|packets transmitted'; echo;
done
PING kaelte_priv (192.168.0.3): 1024 data bytes
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.0/0.0/0.1 ms
PING kaelte_priv (192.168.0.3): 2048 data bytes
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.1/0.2 ms
PING kaelte_priv (192.168.0.3): 4096 data bytes
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.2/0.2/0.3 ms
PING kaelte_priv (192.168.0.3): 8192 data bytes
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.3/0.3/0.4 ms
PING kaelte_priv (192.168.0.3): 16384 data bytes
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.6/0.6/0.7 ms
PING kaelte_priv (192.168.0.3): 32768 data bytes
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 1.1/1.1/1.3 ms
atem:bin#
-------->8--------------------------------------------------------
atem:src# ttcp -s -r -l 1024 -n10240
ttcp-r: buflen=1024, nbuf=10240, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.13 real seconds = 79996.88 KB/sec +++
ttcp-r: 11155 I/O calls, msec/call = 0.01, calls/sec = 87145.03
ttcp-r: 0.0user 0.1sys 0:00real 91% 0i+0d 0maxrss 0+1pf 0+0csw
atem:src# ttcp -s -r -l 2048 -n 5120
ttcp-r: buflen=2048, nbuf=5120, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.12 real seconds = 85870.74 KB/sec +++
ttcp-r: 5789 I/O calls, msec/call = 0.02, calls/sec = 48545.48
ttcp-r: 0.0user 0.0sys 0:00real 81% 0i+0d 0maxrss 0+1pf 0+0csw
atem:src# ttcp -s -r -l 4096 -n 2560
ttcp-r: buflen=4096, nbuf=2560, align=16384/0, port=5001 tcp
ttcp-r: socket
#ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.11 real seconds = 93931.17 KB/sec +++
ttcp-r: 3144 I/O calls, msec/call = 0.04, calls/sec = 28839.80
ttcp-r: 0.0user 0.1sys 0:00real 110% 0i+0d 0maxrss 0+1pf 0+0csw
atem:src# ttcp -s -r -l 8192 -n 1280
ttcp-r: buflen=8192, nbuf=1280, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.10 real seconds = 98433.14 KB/sec +++
ttcp-r: 2270 I/O calls, msec/call = 0.05, calls/sec = 21820.63
ttcp-r: 0.0user 0.0sys 0:00real 80% 0i+0d 0maxrss 0+2pf 0+0csw
atem:src# ttcp -s -r -l 16384 -n 640
ttcp-r: buflen=16384, nbuf=640, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.10 real seconds = 106408.40 KB/sec +++
ttcp-r: 1613 I/O calls, msec/call = 0.06, calls/sec = 16761.40
ttcp-r: 0.0user 0.0sys 0:00real 100% 0i+0d 0maxrss 0+4pf 0+0csw
atem:src# ttcp -s -r -l 32768 -n 320
ttcp-r: buflen=32768, nbuf=320, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.11 real seconds = 91464.50 KB/sec +++
ttcp-r: 1895 I/O calls, msec/call = 0.06, calls/sec = 16926.29
ttcp-r: 0.0user 0.0sys 0:00real 72% 0i+0d 0maxrss 0+8pf 0+0csw
atem:src# ttcp -s -r -l 65536 -n 160
ttcp-r: buflen=65536, nbuf=160, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.3
ttcp-r: 10485760 bytes in 0.09 real seconds = 113219.23 KB/sec +++
ttcp-r: 1467 I/O calls, msec/call = 0.06, calls/sec = 16219.98
ttcp-r: 0.0user 0.0sys 0:00real 88% 0i+0d 0maxrss 0+16pf 0+0csw
atem:src#
-------->8--------------------------------------------------------
--
Greetings to you Echelon guys...everybody say cheese...*SNAP*