Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
So last night my server started doing the same thing! We have a 1.5 TB array (80% utilized) and noticed some users were getting permisioned denied messages accessing certin directories, including the root user. We have the same reiser error in the message log each time someone tries to access one of the corrup folders. I actually disconnected the secondary this weekend, expecting to do some work on it, I don't think thats the problem, but what I am wondering is if I reconnect the servers, if the corruption will sync over. The hardware is a dell scsi storage vault connected to a dell 2650 running 2.6.8-2/debian with drbd 7. Any ideas or suggesttions??? Extended downtime for fsck is my last option :-( Also I checked my partitions and the LVM volume has 1G more space then the reiserfs on top of it (ie drbd has 1GB for meta info) Thanks Dan- Stephan Rattai wrote: >>(unless something else before that went terribly wrong, and some of our >>threads died without that being noticed. or something corrupts memory.) >> >>but I may be wrong, of course... >> >> > >Ok, opening my syslog again and... this wasn't the first oops... I will attach >the two proceding oopses... > > > >------------------------------------------------------------------------ > >Aug 4 11:37:05 test2 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000007 >Aug 4 11:37:05 test2 kernel: printing eip: >Aug 4 11:37:05 test2 kernel: f8c6c259 >Aug 4 11:37:05 test2 kernel: *pde = 00000000 >Aug 4 11:37:05 test2 kernel: Oops: 0000 [#1] >Aug 4 11:37:05 test2 kernel: SMP >Aug 4 11:37:05 test2 kernel: Modules linked in: drbd iptable_filter ehci_hcd uhci_hcd >Aug 4 11:37:05 test2 kernel: CPU: 1 >Aug 4 11:37:05 test2 kernel: EIP: 0060:[pg0+947667545/1068852224] Not tainted VLI >Aug 4 11:37:05 test2 kernel: EIP: 0060:[<f8c6c259>] Not tainted VLI >Aug 4 11:37:05 test2 kernel: EFLAGS: 00010282 (2.6.12.2) >Aug 4 11:37:05 test2 kernel: EIP is at got_NegDReply+0x1a/0xe4 [drbd] >Aug 4 11:37:05 test2 kernel: eax: f3fa32b8 ebx: 00000018 ecx: 00000002 edx: 00000011 >Aug 4 11:37:05 test2 kernel: esi: ffffffff edi: f3fa3000 ebp: f3fa32b8 esp: f5bb7f70 >Aug 4 11:37:05 test2 kernel: ds: 007b es: 007b ss: 0068 >Aug 4 11:37:05 test2 kernel: Process drbd0_asender (pid: 8910, threadinfo=f5bb6000 task=f0f78020) >Aug 4 11:37:05 test2 kernel: Stack: 00000000 f0f78020 00000282 c012390f 00000018 00000020 f3fa3000 f8c6c6ad >Aug 4 11:37:05 test2 kernel: f3fa3000 f3fa32b8 00000018 f5bb7fb0 00000008 00000011 00000020 f3fa32d8 >Aug 4 11:37:05 test2 kernel: 67027483 00000c00 f3fa3450 00000000 f3fa3448 00000000 f8c718dc f3fa3448 >Aug 4 11:37:05 test2 kernel: Call Trace: >Aug 4 11:37:05 test2 kernel: [flush_signals+77/106] flush_signals+0x4d/0x6a >Aug 4 11:37:05 test2 kernel: [<c012390f>] flush_signals+0x4d/0x6a >Aug 4 11:37:05 test2 kernel: [pg0+947668653/1068852224] drbd_asender+0x211/0x471 [drbd] >Aug 4 11:37:05 test2 kernel: [<f8c6c6ad>] drbd_asender+0x211/0x471 [drbd] >Aug 4 11:37:05 test2 kernel: [pg0+947689692/1068852224] drbd_thread_setup+0x81/0xf2 [drbd] >Aug 4 11:37:05 test2 kernel: [<f8c718dc>] drbd_thread_setup+0x81/0xf2 [drbd] >Aug 4 11:37:05 test2 kernel: [pg0+947689563/1068852224] drbd_thread_setup+0x0/0xf2 [drbd] >Aug 4 11:37:05 test2 kernel: [<f8c7185b>] drbd_thread_setup+0x0/0xf2 [drbd] >Aug 4 11:37:05 test2 kernel: [kernel_thread_helper+5/11] kernel_thread_helper+0x5/0xb >Aug 4 11:37:05 test2 kernel: [<c0100f31>] kernel_thread_helper+0x5/0xb >Aug 4 11:37:05 test2 kernel: Code: 80 ce bf 89 44 24 04 e8 83 dc 4a c7 e9 dc fe ff ff 83 ec 1c 89 7c 24 18 89 5c 24 10 89 74 24 14 8b 44 24 24 8b 7c 24 20 8b 70 10 <81> 7e 08 9c 4d c6 f8 74 31 89 f8 c7 44 24 0c 3e 08 00 00 c7 44 >Aug 4 11:37:07 test2 kernel: <6>drbd0: 428268804 KB now marked out-of-sync by on disk bit-map. >Aug 4 11:37:07 test2 kernel: drbd0: drbd_start_resync: (!drbd_md_test_flag(mdev,MDF_Consistent)) in /root/src/drbd-0.7.11/drbd/drbd_worker.c:851 > > >------------------------------------------------------------------------ > >Aug 4 11:37:12 test2 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000504 >Aug 4 11:37:12 test2 kernel: printing eip: >Aug 4 11:37:12 test2 kernel: c035d683 >Aug 4 11:37:12 test2 kernel: *pde = 00000000 >Aug 4 11:37:12 test2 kernel: Oops: 0002 [#2] >Aug 4 11:37:12 test2 kernel: drbd0: worker terminated >Aug 4 11:37:12 test2 kernel: SMP >Aug 4 11:37:12 test2 kernel: Modules linked in: drbd iptable_filter ehci_hcd uhci_hcd >Aug 4 11:37:12 test2 kernel: CPU: 1 >Aug 4 11:37:13 test2 kernel: EIP: 0060:[_spin_lock_irqsave+5/29] Not tainted VLI >Aug 4 11:37:13 test2 kernel: EIP: 0060:[<c035d683>] Not tainted VLI >Aug 4 11:37:13 test2 kernel: EFLAGS: 00010002 (2.6.12.2) >Aug 4 11:37:13 test2 kernel: EIP is at _spin_lock_irqsave+0x5/0x1d >Aug 4 11:37:13 test2 kernel: eax: 00000202 ebx: 00000001 ecx: 000008fc edx: 00000504 >Aug 4 11:37:13 test2 kernel: esi: f0f78020 edi: f3fa3000 ebp: 00000001 esp: f5b51ee4 >Aug 4 11:37:13 test2 kernel: ds: 007b es: 007b ss: 0068 >Aug 4 11:37:13 test2 kernel: Process drbd0_receiver (pid: 8900, threadinfo=f5b50000 task=f6894a20) >Aug 4 11:37:13 test2 kernel: Stack: c0124548 00000001 00000001 f0d7f020 00000002 f3fa3448 f3fa3000 c0124e5e >Aug 4 11:37:13 test2 kernel: 00000001 00000001 f0f78020 f8c71b57 00000001 f0f78020 c0485d40 00000008 >Aug 4 11:37:13 test2 kernel: 00000004 f3fa3400 f3fa3000 f3fa33d0 f3fa33d0 f8c6b355 f3fa3448 00000000 >Aug 4 11:37:13 test2 kernel: Call Trace: >Aug 4 11:37:13 test2 kernel: [force_sig_info+39/165] force_sig_info+0x27/0xa5 >Aug 4 11:37:13 test2 kernel: [<c0124548>] force_sig_info+0x27/0xa5 >Aug 4 11:37:13 test2 kernel: [force_sig+31/35] force_sig+0x1f/0x23 >Aug 4 11:37:13 test2 kernel: [<c0124e5e>] force_sig+0x1f/0x23 >Aug 4 11:37:13 test2 kernel: [pg0+947690327/1068852224] _drbd_thread_stop+0x7d/0x1e9 [drbd] >Aug 4 11:37:13 test2 kernel: [<f8c71b57>] _drbd_thread_stop+0x7d/0x1e9 [drbd] >Aug 4 11:37:13 test2 kernel: [pg0+947663701/1068852224] drbd_disconnect+0xb5/0x727 [drbd] >Aug 4 11:37:13 test2 kernel: [<f8c6b355>] drbd_disconnect+0xb5/0x727 [drbd] >Aug 4 11:37:13 test2 kernel: [pg0+947657597/1068852224] receive_DataRequest+0x587/0x714 [drbd] >Aug 4 11:37:13 test2 kernel: [<f8c69b7d>] receive_DataRequest+0x587/0x714 [drbd] >Aug 4 11:37:13 test2 kernel: [pg0+947650435/1068852224] drbd_recv_header+0x2b/0xf2 [drbd] >Aug 4 11:37:13 test2 kernel: [<f8c67f83>] drbd_recv_header+0x2b/0xf2 [drbd] >Aug 4 11:37:13 test2 kernel: [pg0+947656182/1068852224] receive_DataRequest+0x0/0x714 [drbd] >Aug 4 11:37:13 test2 kernel: [<f8c695f6>] receive_DataRequest+0x0/0x714 [drbd] >Aug 4 11:37:13 test2 kernel: [printk+23/27] printk+0x17/0x1b >Aug 4 11:37:13 test2 kernel: [<c0119ed4>] printk+0x17/0x1b >Aug 4 11:37:13 test2 kernel: [pg0+947663344/1068852224] drbdd+0x91/0x141 [drbd] >Aug 4 11:37:13 test2 kernel: [<f8c6b1f0>] drbdd+0x91/0x141 [drbd] >Aug 4 11:37:13 test2 kernel: [pg0+947666264/1068852224] drbdd_init+0xc1/0x16d [drbd] >Aug 4 11:37:13 test2 kernel: [<f8c6bd58>] drbdd_init+0xc1/0x16d [drbd] >Aug 4 11:37:13 test2 kernel: [pg0+947689692/1068852224] drbd_thread_setup+0x81/0xf2 [drbd] >Aug 4 11:37:13 test2 kernel: [<f8c718dc>] drbd_thread_setup+0x81/0xf2 [drbd] >Aug 4 11:37:13 test2 kernel: [pg0+947689563/1068852224] drbd_thread_setup+0x0/0xf2 [drbd] >Aug 4 11:37:13 test2 kernel: [<f8c7185b>] drbd_thread_setup+0x0/0xf2 [drbd] >Aug 4 11:37:13 test2 kernel: [kernel_thread_helper+5/11] kernel_thread_helper+0x5/0xb >Aug 4 11:37:13 test2 kernel: [<c0100f31>] kernel_thread_helper+0x5/0xb >Aug 4 11:37:13 test2 kernel: Code: 00 01 0f 94 c0 84 c0 b9 01 00 00 00 75 09 f0 81 02 00 00 00 01 30 c9 89 c8 c3 f0 83 28 01 79 05 e8 43 e5 ff ff c3 89 c2 9c 58 fa <f0> fe 0a 79 12 a9 00 02 00 00 74 01 fb f3 90 80 3a 00 7e f9 fa > > > >------------------------------------------------------------------------ > >_______________________________________________ >drbd-user mailing list >drbd-user at lists.linbit.com >http://lists.linbit.com/mailman/listinfo/drbd-user > >