Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Fri, 2008-12-12 at 11:30 +0100, Lars Ellenberg wrote:
> On Thu, Dec 11, 2008 at 05:56:36PM -0800, Nolan wrote:
> > I've got two nodes running Ubuntu 8.10/64bit using the included DRBD:
> > version: 8.2.6 (api:88/proto:86-88)
> > GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by
> > phil at fat-tyre, 2008-05-30 12:59:17
> >
> > Each node has 4 drives, which are striped together using LVM, and then
> > cut into 24 logical volumes. One DRBD is attached to each of the 24
> > lvs. The two nodes speak over 2x bonded e1000s. All was running well
> > for 40+ days with 24 KVM VMs running.
> >
> > I decided to try out the online verify functionality, and after adding
> > "verify-alg crc32c;" to my config on both hosts, and running adjust, I
> > ran:
> > drbdadm verify VM24
> >
> > All was well, and I watched "/proc/drbd" as the verify progressed. But
> > then it stopped at 14%:
> > 24: cs:VerifyS st:Primary/Secondary ds:UpToDate/UpToDate C r---
> > ns:2035752 nr:160958366 dw:162994118 dr:23997736 al:186 bm:10021
> > lo:0 pe:4624 ua:0 ap:16 oos:0
> > 14% 5847476/39734074
> >
> > The VM using that storage is hung hard. Stracing it shows it blocked in
> > a rather uninformative spot:
> > root at node1:~# strace -p 9878
> > Process 9878 attached - interrupt to quit
> > futex(0xb531a0, 0x80 /* FUTEX_??? */, 2
> >
> > Dmesg on the secondary node has nothing interesting, but dmesg on the
> > primary node has:
> > [3628294.472338] drbd24: state = { cs:Connected st:Primary/Secondary
> > ds:UpToDate/UpToDate r--- }
> > [3628294.481196] drbd24: wanted = { cs:VerifyS st:Primary/Secondary
> > ds:UpToDate/UpToDate r--- }
> > [3628524.571022] drbd24: conn( Connected -> VerifyS )
> > [3628919.655921] drbd24: qemu-system-x86[10223] Concurrent local write
> > detected! [DISCARD L] new: 952311s +3584; pending: 952311s +3584
> > [3628919.668048] drbd24: qemu-system-x86[10223] Concurrent local write
> > detected! [DISCARD L] new: 952318s +512; pending: 952318s +512
> > [3628919.680433] drbd24: qemu-system-x86[10223] Concurrent local write
> > detected! [DISCARD L] new: 799599s +3584; pending: 799599s +3584
> > [3628919.692566] drbd24: qemu-system-x86[10223] Concurrent local write
> > detected! [DISCARD L] new: 799606s +512; pending: 799606s +512
> > [3629004.628073] drbd24: qemu-system-x86[10224] Concurrent local write
> > detected! [DISCARD L] new: 952311s +3584; pending: 952311s +3584
> > [3629004.640192] drbd24: qemu-system-x86[10224] Concurrent local write
> > detected! [DISCARD L] new: 952318s +512; pending: 952318s +512
> > [3629004.652675] drbd24: qemu-system-x86[10224] Concurrent local write
> > detected! [DISCARD L] new: 799599s +3584; pending: 799599s +3584
> > [3629004.664787] drbd24: qemu-system-x86[10224] Concurrent local write
> > detected! [DISCARD L] new: 799606s +512; pending: 799606s +512
> >
> > Any ideas what could be causing this?
> >
> > Google on the "concurrent local write" error only turned up the check-in
> > that added that code to DRBD.
>
> that means something is locally submitting writes to some block where we
> still have pending writes to. that is unexpected behaviour, and would
> lead to non-deterministic results on disk. drbd does not allow that.
When you say locally, do you mean the underlying /dev/mapper/whatever
device that the DRBD device is layered on top of?
> > I can leave the system as it is for a few days, if there is more
> > information I should collect.
>
> great.
>
> please do
> echo 2 > /sys/module/drbd/parameters/proc_details
> and provide the output of
> cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by
phil at fat-tyre, 2008-05-30 12:59:17
1: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:237248042 dw:237248042 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
2: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:673304821 dw:673304821 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
3: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:247877984 dw:247877984 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
4: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:192058892 dw:192058892 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
5: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:164147284 dw:164147284 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
6: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:170181168 dw:170181168 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
7: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:277328856 dw:277328856 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
8: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:348815262 dw:348815262 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
9: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:193074261 dw:193074261 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
10: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:581986372 dw:581986372 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
11: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:168700752 dw:168700752 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
12: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:158936296 dw:158936296 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
13: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:158936296 dw:158936296 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
14: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:158936296 dw:158936296 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
15: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:158936296 dw:158936296 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
16: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:158936296 dw:158936296 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
17: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:158936296 dw:158936296 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
18: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:158936296 dw:158936296 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
19: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:158936296 dw:158936296 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
20: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:325581264 dw:325581264 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
21: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:158936296 dw:158936296 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
22: cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate C r---
ns:0 nr:158936296 dw:158936296 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
23: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:180393314 dw:180393314 dr:0 al:0 bm:9701 lo:0 pe:0 ua:0 ap:0
oos:0
resync: used:0/61 hits:79458447 misses:9701 starving:0 dirty:0
changed:9701
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
nn: lc_number refcnt rs_left
0: FREE
1: FREE
2: FREE
3: FREE
4: FREE
5: FREE
6: FREE
7: FREE
8: FREE
9: FREE
10: FREE
11: FREE
12: FREE
13: FREE
14: FREE
15: FREE
16: FREE
17: FREE
18: FREE
19: FREE
20: FREE
21: FREE
22: FREE
23: FREE
24: FREE
25: FREE
26: FREE
27: FREE
28: FREE
29: FREE
30: FREE
31: FREE
32: FREE
33: FREE
34: FREE
35: FREE
36: FREE
37: FREE
38: FREE
39: FREE
40: FREE
41: FREE
42: FREE
43: FREE
44: FREE
45: FREE
46: FREE
47: FREE
48: FREE
49: FREE
50: FREE
51: FREE
52: FREE
53: FREE
54: FREE
55: FREE
56: FREE
57: FREE
58: FREE
59: FREE
60: FREE
24: cs:VerifyS st:Primary/Secondary ds:UpToDate/UpToDate C r---
ns:2035752 nr:160958366 dw:162994118 dr:23997736 al:186 bm:10021
lo:0 pe:4624 ua:0 ap:16 oos:0
14% 5847476/39734074
resync: used:2/61 hits:85309102 misses:11130 starving:0 dirty:0
changed:11130
act_log: used:3/127 hits:717422 misses:186 starving:0 dirty:0
changed:186
nn: lc_number refcnt rs_left
0: 1399 0 0 --------- ------
1: 1400 0 0 --------- ------
2: 1401 0 0 --------- ------
3: 1402 0 0 --------- ------
4: 1397 0 0 --------- ------
5: 1391 0 0 --------- ------
6: 1398 0 0 --------- ------
7: 1396 0 0 --------- ------
8: 1394 0 0 --------- ------
9: 1395 0 0 --------- ------
10: 1393 0 0 --------- ------
11: 1390 0 0 --------- ------
12: 1389 0 0 --------- ------
13: 1387 0 0 --------- ------
14: 1392 0 0 --------- ------
15: 1386 0 0 --------- ------
16: 1385 0 0 --------- ------
17: 1384 0 0 --------- ------
18: 1388 0 0 --------- ------
19: 1380 0 0 --------- ------
20: 1383 0 0 --------- ------
21: 1375 0 0 --------- ------
22: 1382 0 0 --------- ------
23: 1378 0 0 --------- ------
24: 1371 0 0 --------- ------
25: 1381 0 0 --------- ------
26: 1379 0 0 --------- ------
27: 1368 0 0 --------- ------
28: 1377 0 0 --------- ------
29: 1370 0 0 --------- ------
30: 1374 0 0 --------- ------
31: 1372 0 0 --------- ------
32: 1373 0 0 --------- ------
33: 1428 2996 0 --------- LOCKED
34: 1376 0 0 --------- ------
35: 1427 1612 0 --------- LOCKED
36: 1426 0 0 --------- ------
37: 1369 0 0 --------- ------
38: 1425 0 0 --------- ------
39: 1424 0 0 --------- ------
40: 1422 0 0 --------- ------
41: 1420 0 0 --------- ------
42: 1421 0 0 --------- ------
43: 1423 0 0 --------- ------
44: 1417 0 0 --------- ------
45: 1413 0 0 --------- ------
46: 1415 0 0 --------- ------
47: 1419 0 0 --------- ------
48: 1418 0 0 --------- ------
49: 1416 0 0 --------- ------
50: 1411 0 0 --------- ------
51: 1410 0 0 --------- ------
52: 1408 0 0 --------- ------
53: 1414 0 0 --------- ------
54: 1406 0 0 --------- ------
55: 1407 0 0 --------- ------
56: 1404 0 0 --------- ------
57: 1412 0 0 --------- ------
58: 1403 0 0 --------- ------
59: 1405 0 0 --------- ------
60: 1409 0 0 --------- ------
> ps -eo pid,state,wchan:30,cmd | grep -e drbd -e D
PID S WCHAN CMD
6472 S down_interruptible [drbd1_worker]
6482 S sk_wait_data [drbd1_receiver]
6489 S down_interruptible [drbd2_worker]
6495 S sk_wait_data [drbd2_receiver]
6501 S down_interruptible [drbd3_worker]
6508 S sk_wait_data [drbd3_receiver]
6514 S down_interruptible [drbd4_worker]
6522 S sk_wait_data [drbd4_receiver]
6527 S down_interruptible [drbd5_worker]
6534 S sk_wait_data [drbd5_receiver]
6540 S down_interruptible [drbd6_worker]
6547 S sk_wait_data [drbd6_receiver]
6550 S down_interruptible [drbd7_worker]
6560 S sk_wait_data [drbd7_receiver]
6567 S down_interruptible [drbd8_worker]
6573 S sk_wait_data [drbd8_receiver]
6579 S down_interruptible [drbd9_worker]
6586 S sk_wait_data [drbd9_receiver]
6589 S down_interruptible [drbd10_worker]
6600 S sk_wait_data [drbd10_receiver]
6602 S down_interruptible [drbd11_worker]
6612 S sk_wait_data [drbd11_receiver]
6615 S down_interruptible [drbd12_worker]
6625 S sk_wait_data [drbd12_receiver]
6628 S down_interruptible [drbd13_worker]
6638 S sk_wait_data [drbd13_receiver]
6641 S down_interruptible [drbd14_worker]
6651 S sk_wait_data [drbd14_receiver]
6654 S down_interruptible [drbd15_worker]
6664 S sk_wait_data [drbd15_receiver]
6667 S down_interruptible [drbd16_worker]
6677 S sk_wait_data [drbd16_receiver]
6683 S down_interruptible [drbd17_worker]
6690 S sk_wait_data [drbd17_receiver]
6693 S down_interruptible [drbd18_worker]
6704 S sk_wait_data [drbd18_receiver]
6706 S down_interruptible [drbd19_worker]
6716 S sk_wait_data [drbd19_receiver]
6722 S down_interruptible [drbd20_worker]
6730 S sk_wait_data [drbd20_receiver]
6736 S down_interruptible [drbd21_worker]
6742 S sk_wait_data [drbd21_receiver]
6748 S down_interruptible [drbd22_worker]
6755 S sk_wait_data [drbd22_receiver]
6758 S down_interruptible [drbd23_worker]
6768 S sk_wait_data [drbd23_receiver]
6771 S down_interruptible [drbd24_worker]
6781 S sk_wait_data [drbd24_receiver]
6805 S sk_wait_data [drbd1_asender]
6892 S sk_wait_data [drbd4_asender]
6893 S sk_wait_data [drbd5_asender]
6894 S sk_wait_data [drbd3_asender]
6895 S sk_wait_data [drbd6_asender]
6896 S sk_wait_data [drbd7_asender]
6897 S sk_wait_data [drbd9_asender]
6898 S sk_wait_data [drbd8_asender]
6899 S sk_wait_data [drbd10_asender]
6900 S sk_wait_data [drbd11_asender]
6901 S sk_wait_data [drbd12_asender]
6902 S sk_wait_data [drbd13_asender]
6903 S sk_wait_data [drbd14_asender]
6904 S sk_wait_data [drbd15_asender]
6905 S sk_wait_data [drbd16_asender]
6906 S sk_wait_data [drbd17_asender]
6907 S sk_wait_data [drbd18_asender]
6908 S sk_wait_data [drbd21_asender]
6909 S sk_wait_data [drbd19_asender]
6910 S sk_wait_data [drbd20_asender]
6911 S sk_wait_data [drbd22_asender]
6912 S sk_wait_data [drbd23_asender]
6913 S sk_wait_data [drbd24_asender]
6914 S sk_wait_data [drbd2_asender]
9878 S futex_wait /usr/local/bin/qemu-system-x86_64
-m 512 -drive file=/dev/drbd24,if=scsi,cache=off,boot=on -drive
if=ide,index=2,media=cdrom,boot=off -usbdevice tablet -name root_vm_1
-net nic,macaddr=00:0c:29:12:47:21,model=rtl8139 -net tap -monitor
unix:/tmp/VM_root_vm_1,server,nowait -tdf -daemonize -vnc :0,password
> cat /proc/mounts
rootfs / rootfs rw 0 0
none /sys sysfs rw,nosuid,nodev,noexec 0 0
none /proc proc rw,nosuid,nodev,noexec 0 0
udev /dev tmpfs rw,mode=755 0 0
fusectl /sys/fs/fuse/connections fusectl rw 0 0
/dev/md1 / ext3 rw,relatime,errors=remount-ro,data=ordered 0 0
/dev/md1 /dev/.static/dev ext3 ro,errors=remount-ro,data=ordered 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,mode=755 0 0
varrun /var/run tmpfs rw,nosuid,mode=755 0 0
varlock /var/lock tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,gid=5,mode=620 0 0
/dev/sde1 /boot ext3 rw,noatime,errors=continue,data=ordered 0 0
securityfs /sys/kernel/security securityfs rw 0 0
/dev/md2 /tile ext3 rw,relatime,errors=continue,data=ordered 0 0
> cat /proc/partitions
major minor #blocks name
8 0 976762584 sda
8 1 3156741 sda1
8 2 4200997 sda2
8 3 15735667 sda3
8 4 953666595 sda4
8 16 976762584 sdb
8 17 3156741 sdb1
8 18 4200997 sdb2
8 19 15735667 sdb3
8 20 953666595 sdb4
8 32 976762584 sdc
8 33 3156741 sdc1
8 34 4200997 sdc2
8 35 15735667 sdc3
8 36 953666595 sdc4
8 48 976762584 sdd
8 49 3156741 sdd1
8 50 4200997 sdd2
8 51 15735667 sdd3
8 52 953666595 sdd4
9 0 6313344 md0
9 1 8401792 md1
254 0 158941184 dm-0
254 1 158941184 dm-1
254 2 158941184 dm-2
254 3 158941184 dm-3
254 4 158941184 dm-4
254 5 158941184 dm-5
254 6 158941184 dm-6
254 7 158941184 dm-7
254 8 158941184 dm-8
254 9 158941184 dm-9
254 10 158941184 dm-10
254 11 158941184 dm-11
254 12 158941184 dm-12
254 13 158941184 dm-13
254 14 158941184 dm-14
254 15 158941184 dm-15
254 16 158941184 dm-16
254 17 158941184 dm-17
254 18 158941184 dm-18
254 19 158941184 dm-19
254 20 158941184 dm-20
254 21 158941184 dm-21
254 22 158941184 dm-22
254 23 158941184 dm-23
9 2 31471104 md2
8 64 983040 sde
8 65 979933 sde1
147 1 158936296 drbd1
147 2 158936296 drbd2
147 3 158936296 drbd3
147 4 158936296 drbd4
147 5 158936296 drbd5
147 6 158936296 drbd6
147 7 158936296 drbd7
147 8 158936296 drbd8
147 9 158936296 drbd9
147 10 158936296 drbd10
147 11 158936296 drbd11
147 12 158936296 drbd12
147 13 158936296 drbd13
147 14 158936296 drbd14
147 15 158936296 drbd15
147 16 158936296 drbd16
147 17 158936296 drbd17
147 18 158936296 drbd18
147 19 158936296 drbd19
147 20 158936296 drbd20
147 21 158936296 drbd21
147 22 158936296 drbd22
147 23 158936296 drbd23
147 24 158936296 drbd24
> dmsetup table
VMs-VM24: 0 317882368 striped 4 2048 8:4 1827824000 8:20 1827824000 8:36
1827824000 8:52 1827824000
VMs-VM23: 0 317882368 striped 4 2048 8:4 1748353408 8:20 1748353408 8:36
1748353408 8:52 1748353408
VMs-VM22: 0 317882368 striped 4 2048 8:4 1668882816 8:20 1668882816 8:36
1668882816 8:52 1668882816
VMs-VM9: 0 317882368 striped 4 2048 8:4 635765120 8:20 635765120 8:36
635765120 8:52 635765120
VMs-VM19: 0 317882368 striped 4 2048 8:4 1430471040 8:20 1430471040 8:36
1430471040 8:52 1430471040
VMs-VM21: 0 317882368 striped 4 2048 8:4 1589412224 8:20 1589412224 8:36
1589412224 8:52 1589412224
VMs-VM8: 0 317882368 striped 4 2048 8:4 556294528 8:20 556294528 8:36
556294528 8:52 556294528
VMs-VM18: 0 317882368 striped 4 2048 8:4 1351000448 8:20 1351000448 8:36
1351000448 8:52 1351000448
VMs-VM20: 0 317882368 striped 4 2048 8:4 1509941632 8:20 1509941632 8:36
1509941632 8:52 1509941632
VMs-VM7: 0 317882368 striped 4 2048 8:4 476823936 8:20 476823936 8:36
476823936 8:52 476823936
VMs-VM17: 0 317882368 striped 4 2048 8:4 1271529856 8:20 1271529856 8:36
1271529856 8:52 1271529856
VMs-VM6: 0 317882368 striped 4 2048 8:4 397353344 8:20 397353344 8:36
397353344 8:52 397353344
VMs-VM16: 0 317882368 striped 4 2048 8:4 1192059264 8:20 1192059264 8:36
1192059264 8:52 1192059264
VMs-VM5: 0 317882368 striped 4 2048 8:4 317882752 8:20 317882752 8:36
317882752 8:52 317882752
VMs-VM15: 0 317882368 striped 4 2048 8:4 1112588672 8:20 1112588672 8:36
1112588672 8:52 1112588672
VMs-VM4: 0 317882368 striped 4 2048 8:4 238412160 8:20 238412160 8:36
238412160 8:52 238412160
VMs-VM14: 0 317882368 striped 4 2048 8:4 1033118080 8:20 1033118080 8:36
1033118080 8:52 1033118080
VMs-VM3: 0 317882368 striped 4 2048 8:4 158941568 8:20 158941568 8:36
158941568 8:52 158941568
VMs-VM13: 0 317882368 striped 4 2048 8:4 953647488 8:20 953647488 8:36
953647488 8:52 953647488
VMs-VM2: 0 317882368 striped 4 2048 8:4 79470976 8:20 79470976 8:36
79470976 8:52 79470976
VMs-VM12: 0 317882368 striped 4 2048 8:4 874176896 8:20 874176896 8:36
874176896 8:52 874176896
VMs-VM1: 0 317882368 striped 4 2048 8:4 384 8:20 384 8:36 384 8:52 384
VMs-VM11: 0 317882368 striped 4 2048 8:4 794706304 8:20 794706304 8:36
794706304 8:52 794706304
VMs-VM10: 0 317882368 striped 4 2048 8:4 715235712 8:20 715235712 8:36
715235712 8:52 715235712