Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi all,
I had a problem this night with a DRBD Primary/Slave.
The slave experienced a hardware issue (LSI controller freezed).
It seems the master hold I/O waiting for the slave to respond until timeout.
This caused all targets exported trough infiniband to be disconnected
from the master.
So, practically, the master stop responding due to a failure on the slave.
I had to hard reboot (power cycle) the slave because UDEV wasn't
responding and did not allow normal reboot.
After slave reboot, drdb did reconnect. It was in status pri/sec
uptodate/uptodate.
But the LSI controller immediatly timeout causing the same issue a
second time.
How can we prevent issue on the slave to impact the master ?
Thank you.
Matthieu Lejeune
drbd8-utils : 2:8.3.13-2 amd64
RAID 1 over tcp/ip for Linux utilities
Debian :
root at ifprdstor8a:~/trunk# cat /proc/version
Linux version 3.2.0-4-amd64 (debian-kernel at lists.debian.org) (gcc
version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.51-1
root at ifprdstor8a:~/trunk#
We are using the scst/srpt with the Trunk version of the 7 January 2014
I give you the config :
*drbd global : **
*
root at ifprdstor8a:/etc/drbd.d# cat global_common.conf
global {
usage-count yes;
# minor-count dialog-refresh disable-ip-verification
}
common {
protocol C;
handlers {
# The following 3 handlers were disabled due to #576511.
# Please check the DRBD manual and enable them, if they make
sense in your setup.
# pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";
# pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";
# local-io-error "/usr/lib/drbd/notify-io-error.sh;
/usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger
; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target
"/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target
/usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
}
disk {
# on-io-error fencing use-bmbv no-disk-barrier no-disk-flushes
# no-disk-drain no-md-flushes max-bio-bvecs
}
net {
# sndbuf-size rcvbuf-size timeout connect-int ping-int
ping-timeout max-buffers
# max-epoch-size ko-count allow-two-primaries cram-hmac-alg
shared-secret
# after-sb-0pri after-sb-1pri after-sb-2pri data-integrity-alg
no-tcp-cork
}
syncer {
# rate after al-extents use-rle cpu-mask verify-alg csums-alg
}
}
*Ressource Configuration : *
root at ifprdstor8a:/etc/drbd.d# cat DSA801.res
resource DSA801 {
protocol C;
startup {
wfc-timeout 0;
}
disk {
on-io-error detach;
}
syncer {
rate 400M;
verify-alg md5;
}
on ifprdstor8a {
device /dev/drbd1;
disk /dev/sda;
address 10.13.1.5:7788;
meta-disk internal;
}
on ifprdstor8b {
device /dev/drbd1;
disk /dev/sda;
address 10.13.1.6:7788;
meta-disk internal;
}
}
*
**This is the scst.conf on the master SAN : **
*
root at ifprdstor8a:/etc/drbd.d# cat /etc/scst.conf
HANDLER vdisk_blockio {
DEVICE DSA801{
filename /dev/drbd1
}
DEVICE DSA802{
filename /dev/drbd2
}
DEVICE DSA803{
filename /dev/drbd3
}
DEVICE DSA804{
filename /dev/drbd4
}
DEVICE DSA805{
filename /dev/drbd5
}
DEVICE DSA806{
filename /dev/drbd6
}
DEVICE DSA807{
filename /dev/drbd7
}
}
TARGET_DRIVER ib_srpt {
TARGET ib_srpt_target_0 {
rel_tgt_id 1
enabled 1
LUN 0 DSA801
LUN 1 DSA802
LUN 2 DSA803
LUN 3 DSA804
LUN 4 DSA805
LUN 5 DSA806
LUN 6 DSA807
}
}
*I give you the log files : *
*On the slave : *
Mar 5 04:00:01 ifprdstor8b /USR/SBIN/CRON[32179]: (root) CMD
(/usr/local/sbin/vivaldi.sh 2>&1 > /dev/null)
Mar 5 04:00:58 ifprdstor8b kernel: [4641211.561608] megasas: Found FW
in FAULT state, will reset adapter.
Mar 5 04:00:58 ifprdstor8b kernel: [4641211.561613] megaraid_sas:
resetting fusion adapter.
Mar 5 04:02:28 ifprdstor8b kernel: [4641300.910051] megasas: waitingfor
controller reset to finish
Mar 5 04:02:33 ifprdstor8b kernel: [4641305.929495] megasas: waitingfor
controller reset to finish
Mar 5 04:02:38 ifprdstor8b kernel: [4641310.949471] megasas: waitingfor
controller reset to finish
Mar 5 04:02:43 ifprdstor8b kernel: [4641315.969469] megasas: waitingfor
controller reset to finish
Mar 5 04:02:46 ifprdstor8b kernel: [4641318.673468] megaraid_sas: Diag
reset adapter never cleared!
Mar 5 04:02:48 ifprdstor8b kernel: [4641320.989482] megasas: waitingfor
controller reset to finish
Mar 5 04:02:53 ifprdstor8b kernel: [4641326.009490] megasas: waitingfor
controller reset to finish
Mar 5 04:02:58 ifprdstor8b kernel: [4641331.029489] megasas: waitingfor
controller reset to finish
Mar 5 04:03:03 ifprdstor8b kernel: [4641336.049462] megasas: waitingfor
controller reset to finish
Mar 5 04:03:08 ifprdstor8b kernel: [4641341.069459] megasas: waitingfor
controller reset to finish
Mar 5 04:03:13 ifprdstor8b kernel: [4641346.089467] megasas: waitingfor
controller reset to finish
Mar 5 04:03:18 ifprdstor8b kernel: [4641351.109483] megasas: waitingfor
controller reset to finish
Mar 5 04:03:23 ifprdstor8b kernel: [4641356.129476] megasas: waitingfor
controller reset to finish
Mar 5 04:03:28 ifprdstor8b kernel: [4641361.149457] megasas: waitingfor
controller reset to finish
Mar 5 04:03:33 ifprdstor8b kernel: [4641366.169459] megasas: waitingfor
controller reset to finish
Mar 5 04:03:38 ifprdstor8b kernel: [4641371.189471] megasas: waitingfor
controller reset to finish
Mar 5 04:03:43 ifprdstor8b kernel: [4641376.209451] megasas: waitingfor
controller reset to finish
Mar 5 04:03:48 ifprdstor8b kernel: [4641381.229452] megasas: waitingfor
controller reset to finish
Mar 5 04:03:53 ifprdstor8b kernel: [4641386.249475] megasas: waitingfor
controller reset to finish
Mar 5 04:03:58 ifprdstor8b kernel: [4641391.269473] megasas: waitingfor
controller reset to finish
Mar 5 04:04:03 ifprdstor8b kernel: [4641396.289470] megasas: waitingfor
controller reset to finish
Mar 5 04:04:08 ifprdstor8b kernel: [4641401.309444] megasas: waitingfor
controller reset to finish
Mar 5 04:04:13 ifprdstor8b kernel: [4641406.329443] megasas: waitingfor
controller reset to finish
Mar 5 04:04:18 ifprdstor8b kernel: [4641411.349461] megasas: waitingfor
controller reset to finish
Mar 5 04:04:23 ifprdstor8b kernel: [4641416.369458] megasas: waitingfor
controller reset to finish
Mar 5 04:04:28 ifprdstor8b kernel: [4641421.389436] megasas: waitingfor
controller reset to finish
Mar 5 04:04:31 ifprdstor8b kernel: [4641423.587272] megasas: waitingfor
controller reset to finish
Mar 5 04:04:33 ifprdstor8b kernel: [4641425.785435] megaraid_sas: Diag
reset adapter never cleared!
Mar 5 04:04:33 ifprdstor8b kernel: [4641426.409443] megasas: waitingfor
controller reset to finish
Mar 5 04:04:36 ifprdstor8b kernel: [4641428.605457] megasas: waitingfor
controller reset to finish
Mar 5 04:04:38 ifprdstor8b kernel: [4641431.429457] megasas: waitingfor
controller reset to finish
Mar 5 04:04:41 ifprdstor8b kernel: [4641433.625454] megasas: waitingfor
controller reset to finish
Mar 5 04:04:43 ifprdstor8b kernel: [4641436.449386] megasas: waitingfor
controller reset to finish
Mar 5 04:04:46 ifprdstor8b kernel: [4641438.645431] megasas: waitingfor
controller reset to finish
Mar 5 04:04:48 ifprdstor8b kernel: [4641441.469451] megasas: waitingfor
controller reset to finish
Mar 5 04:04:51 ifprdstor8b kernel: [4641443.665432] megasas: waitingfor
controller reset to finish
Mar 5 04:04:53 ifprdstor8b kernel: [4641446.489429] megasas: waitingfor
controller reset to finish
Mar 5 04:04:56 ifprdstor8b kernel: [4641448.685424] megasas: waitingfor
controller reset to finish
Mar 5 04:04:58 ifprdstor8b kernel: [4641451.509431] megasas: waitingfor
controller reset to finish
Mar 5 04:05:01 ifprdstor8b kernel: [4641453.705431] megasas: waitingfor
controller reset to finish
Mar 5 04:05:01 ifprdstor8b /USR/SBIN/CRON[32250]: (root) CMD (command
-v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 5 04:05:01 ifprdstor8b /USR/SBIN/CRON[32251]: (root) CMD
(/usr/local/sbin/vivaldi.sh 2>&1 > /dev/null)
Mar 5 04:05:03 ifprdstor8b kernel: [4641456.529455] megasas: waitingfor
controller reset to finish
Mar 5 04:05:06 ifprdstor8b kernel: [4641458.725447] megasas: waitingfor
controller reset to finish
Mar 5 04:05:08 ifprdstor8b kernel: [4641461.549518] megasas: waitingfor
controller reset to finish
Mar 5 04:05:11 ifprdstor8b kernel: [4641463.745448] megasas: waitingfor
controller reset to finish
Mar 5 04:05:14 ifprdstor8b kernel: [4641466.569498] megasas: waitingfor
controller reset to finish
Mar 5 04:05:16 ifprdstor8b kernel: [4641468.765421] megasas: waitingfor
controller reset to finish
Mar 5 04:05:19 ifprdstor8b kernel: [4641471.589419] megasas: waitingfor
controller reset to finish
Mar 5 04:05:21 ifprdstor8b kernel: [4641473.785424] megasas: waitingfor
controller reset to finish
Mar 5 04:05:24 ifprdstor8b kernel: [4641476.609440] megasas: waitingfor
controller reset to finish
Mar 5 04:05:26 ifprdstor8b kernel: [4641478.805441] megasas: waitingfor
controller reset to finish
Mar 5 04:05:29 ifprdstor8b kernel: [4641481.629443] megaraid_sas: timed
out whilewaiting for HBA to recover
Mar 5 04:05:31 ifprdstor8b kernel: [4641483.825416] megasas: waitingfor
controller reset to finish
Mar 5 04:05:36 ifprdstor8b kernel: [4641488.845418] megasas: waitingfor
controller reset to finish
Mar 5 04:05:41 ifprdstor8b kernel: [4641493.865416] megasas: waitingfor
controller reset to finish
Mar 5 04:05:46 ifprdstor8b kernel: [4641498.885491] megasas: waitingfor
controller reset to finish
Mar 5 04:05:51 ifprdstor8b kernel: [4641503.905431] megasas: waitingfor
controller reset to finish
Mar 5 04:05:56 ifprdstor8b kernel: [4641508.925407] megasas: waitingfor
controller reset to finish
Mar 5 04:06:01 ifprdstor8b kernel: [4641513.945410] megasas: waitingfor
controller reset to finish
Mar 5 04:06:06 ifprdstor8b kernel: [4641518.965427] megasas: waitingfor
controller reset to finish
Mar 5 04:06:11 ifprdstor8b kernel: [4641523.985425] megasas: waitingfor
controller reset to finish
Mar 5 04:06:16 ifprdstor8b kernel: [4641529.005358] megasas: waitingfor
controller reset to finish
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893406] megaraid_sas: Diag
reset adapter never cleared!
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893410] megaraid_sas: Reset
failed, killing adapter.
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893416] sd 0:2:1:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893420] sd 0:2:1:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893424] sd 0:2:1:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893431] sd 0:2:1:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893435] sd 0:2:1:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893439] sd 0:2:1:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893442] sd 0:2:1:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893446] sd 0:2:1:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893450] sd 0:2:1:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893454] sd 0:2:3:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893458] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893462] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893465] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893469] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893473] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893477] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893480] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893484] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893488] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893491] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893495] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893499] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893503] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893506] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893510] sd 0:2:2:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893514] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893518] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893522] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893525] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893529] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893532] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893536] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893540] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893543] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893547] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893551] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893555] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893558] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.893562] sd 0:2:0:0: Device
offlined - not ready after error recovery
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897752] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897773] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897778] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897794] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897797] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897813] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897816] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897831] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897835] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897850] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897853] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897868] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897872] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897887] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897890] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897906] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897909] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897924] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897928] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897943] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897946] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897961] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897965] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897994] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.897997] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898026] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898030] sd 0:2:0:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898059] sd 0:2:0:0: [sda]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898066] sd 0:2:1:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898096] sd 0:2:1:0: [sdb]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898099] sd 0:2:1:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898128] sd 0:2:1:0: [sdb]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898132] sd 0:2:1:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898161] sd 0:2:1:0: [sdb]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898164] sd 0:2:1:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898193] sd 0:2:1:0: [sdb]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898197] sd 0:2:1:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898226] sd 0:2:1:0: [sdb]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898229] sd 0:2:1:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898258] sd 0:2:1:0: [sdb]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898262] sd 0:2:1:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898291] sd 0:2:1:0: [sdb]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898294] sd 0:2:1:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898323] sd 0:2:1:0: [sdb]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898327] sd 0:2:1:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898356] sd 0:2:1:0: [sdb]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898360] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898389] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898393] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898422] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898425] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898454] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898458] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898487] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898490] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898519] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.898523] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903711] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903714] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903743] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903747] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903776] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903779] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903809] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903812] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903841] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903844] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903874] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903877] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903906] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903909] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903939] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903942] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903971] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.903974] sd 0:2:2:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904004] sd 0:2:2:0: [sdc]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904079] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904084] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904099] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7b c7 00 00 38 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904132] end_request: I/O
error, dev sda, sector 891976647
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904170] block drbd1: write:
error=-5 s=891976647s
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904183] block drbd1: disk(
UpToDate -> Failed )
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904195] block drbd1: Local
IO failed in drbd_endio_write_sec_final. Detaching...
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904262] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904269] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904280] sd 0:2:0:0: [sda] CDB:
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904289] block drbd1:
Notified peer that I am detaching my disk
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904300] Write(10): 2a 00 35
2a 7b ff 00 00 01 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904325] end_request: I/O
error, dev sda, sector 891976703
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904361] block drbd1: write:
error=-5 s=891976703s
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904374] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904381] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904392] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7c 00 00 00 07 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904419] end_request: I/O
error, dev sda, sector 891976704
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904455] block drbd1: write:
error=-5 s=891976704s
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904467] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904474] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904485] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7c 07 00 00 18 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904513] end_request: I/O
error, dev sda, sector 891976711
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904549] block drbd1: write:
error=-5 s=891976711s
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904560] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904563] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904569] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7c 1f 00 00 58 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904588] end_request: I/O
error, dev sda, sector 891976735
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904627] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904633] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904645] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7c 77 00 00 58 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904672] end_request: I/O
error, dev sda, sector 891976823
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904711] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904714] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904719] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7c cf 00 00 30 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904730] end_request: I/O
error, dev sda, sector 891976911
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904765] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904768] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904773] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7c ff 00 00 01 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904784] end_request: I/O
error, dev sda, sector 891976959
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904819] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904822] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904827] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7d 00 00 00 07 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904839] end_request: I/O
error, dev sda, sector 891976960
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904874] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904877] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904882] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7d 07 00 00 20 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904893] end_request: I/O
error, dev sda, sector 891976967
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904927] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904930] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904935] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7d 27 00 00 58 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904946] end_request: I/O
error, dev sda, sector 891976999
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904980] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904983] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904988] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 35 2a 7d 7f 00 00 20 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.904999] end_request: I/O
error, dev sda, sector 891977087
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905034] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905037] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905042] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 00 60 37 e7 00 00 08 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905053] end_request: I/O
error, dev sda, sector 6305767
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905087] sd 0:2:0:0: [sda]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905090] sd 0:2:0:0: [sda]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905096] sd 0:2:0:0: [sda]
CDB: Write(10): 2a 00 01 10 03 47 00 00 08 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905106] end_request: I/O
error, dev sda, sector 17826631
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905143] sd 0:2:1:0: [sdb]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905146] sd 0:2:1:0: [sdb]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905151] sd 0:2:1:0: [sdb]
CDB: Write(10): 2a 00 00 c0 d5 60 00 00 58 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905162] end_request: I/O
error, dev sdb, sector 12637536
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905175] block drbd1: disk(
Failed -> Diskless )
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905190] block drbd1:
Notified peer that I'm now diskless.
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905203] block drbd2: disk(
UpToDate -> Failed )
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905212] block drbd2: Local
IO failed in drbd_endio_write_sec_final. Detaching...
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905255] block drbd1:
receiver updated UUIDs to effective data uuid: D222B0557049B6DD
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905272] block drbd2:
Notified peer that I am detaching my disk
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905288] sd 0:2:1:0: [sdb]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905293] sd 0:2:1:0: [sdb]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905309] sd 0:2:1:0: [sdb]
CDB: Write(10): 2a 00 00 c0 d5 b8 00 00 28 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905328] end_request: I/O
error, dev sdb, sector 12637624
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905365] sd 0:2:1:0: [sdb]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905371] sd 0:2:1:0: [sdb]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905380] sd 0:2:1:0: [sdb]
CDB: Write(10): 2a 00 06 40 14 ff 00 00 01 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905391] end_request: I/O
error, dev sdb, sector 104862975
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905432] sd 0:2:1:0: [sdb]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905438] sd 0:2:1:0: [sdb]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905448] sd 0:2:1:0: [sdb]
CDB: Write(10): 2a 00 08 f4 7e 60 00 00 01 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905474] end_request: I/O
error, dev sdb, sector 150240864
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905512] sd 0:2:1:0: [sdb]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905515] sd 0:2:1:0: [sdb]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905523] sd 0:2:1:0: [sdb]
CDB: Write(10): 2a 00 01 d3 7e c8 00 00 01 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905550] end_request: I/O
error, dev sdb, sector 30637768
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905587] sd 0:2:1:0: [sdb]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905590] sd 0:2:1:0: [sdb]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905595] sd 0:2:1:0: [sdb]
CDB: Write(10): 2a 00 08 f4 7e 61 00 00 01 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905606] end_request: I/O
error, dev sdb, sector 150240865
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905650] sd 0:2:1:0: [sdb]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905656] sd 0:2:1:0: [sdb]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905665] sd 0:2:1:0: [sdb]
CDB: Write(10): 2a 00 00 0c 13 f8 00 00 01 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905692] end_request: I/O
error, dev sdb, sector 791544
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905730] sd 0:2:1:0: [sdb]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905733] sd 0:2:1:0: [sdb]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905738] sd 0:2:1:0: [sdb]
CDB: Write(10): 2a 00 01 fc 6e 00 00 00 01 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905749] end_request: I/O
error, dev sdb, sector 33320448
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905785] sd 0:2:1:0: [sdb]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905791] sd 0:2:1:0: [sdb]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905801] sd 0:2:1:0: [sdb]
CDB: Write(10): 2a 00 01 4f 50 58 00 00 02 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905827] end_request: I/O
error, dev sdb, sector 21975128
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905899] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905902] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905907] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905915] block drbd2: disk(
Failed -> Diskless )
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905922] 04
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905928] block drbd2:
receiver updated UUIDs to effective data uuid: 557B4AD2F3623003
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905938] e3 97 58 00 00 58 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905950] end_request: I/O
error, dev sdc, sector 82024280
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905961] block drbd3: disk(
UpToDate -> Failed )
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905966] block drbd3: Local
IO failed in drbd_endio_write_sec_final. Detaching...
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905975] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905979] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905987] sd 0:2:2:0: [sdc] CDB:
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.905994] block drbd3:
Notified peer that I am detaching my disk
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906002] Write(10): 2a 00 04
e3 97 b0 00 00 20 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906015] end_request: I/O
error, dev sdc, sector 82024368
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906024] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906029] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906037] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 97 d0 00 00 30 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906056] end_request: I/O
error, dev sdc, sector 82024400
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906066] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906070] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906082] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 98 00 00 00 28 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906097] end_request: I/O
error, dev sdc, sector 82024448
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906107] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906114] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906122] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 98 28 00 00 20 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906133] end_request: I/O
error, dev sdc, sector 82024488
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906140] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906146] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906151] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 98 48 00 00 58 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906159] end_request: I/O
error, dev sdc, sector 82024520
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906168] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906173] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906182] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 98 a0 00 00 20 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906197] end_request: I/O
error, dev sdc, sector 82024608
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906207] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906210] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906219] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 98 c0 00 00 40 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906234] end_request: I/O
error, dev sdc, sector 82024640
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906245] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906248] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906255] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 99 00 00 00 18 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906270] end_request: I/O
error, dev sdc, sector 82024704
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906283] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906288] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906296] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 99 18 00 00 20 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906313] end_request: I/O
error, dev sdc, sector 82024728
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906326] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906329] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906335] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 99 38 00 00 58 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906350] end_request: I/O
error, dev sdc, sector 82024760
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906362] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906367] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906375] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 99 90 00 00 20 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906392] end_request: I/O
error, dev sdc, sector 82024848
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906402] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906405] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906410] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 99 b0 00 00 50 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906426] end_request: I/O
error, dev sdc, sector 82024880
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906438] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906443] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906449] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 9a 00 00 00 08 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906460] end_request: I/O
error, dev sdc, sector 82024960
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906472] sd 0:2:2:0: [sdc]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906476] sd 0:2:2:0: [sdc]
Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906480] sd 0:2:2:0: [sdc]
CDB: Write(10): 2a 00 04 e3 9a 08 00 00 20 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906495] end_request: I/O
error, dev sdc, sector 82024968
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906513] sd 0:2:3:0:
rejecting I/O to offline device
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906517] sd 0:2:3:0: [sdd]
killing request
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906522] block drbd3: disk(
Failed -> Diskless )
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906527] sd 0:2:3:0: [sdd]
Unhandled error code
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906530] sd 0:2:3:0: [sdd]
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906535] block drbd3:
Notified peer that I'm now diskless.
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906541] Result:
hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906545] sd 0:2:3:0: [sdd]
CDB: Write(10): 2a 00 01 2b 7e 9a 00 00 0d 00
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906552] end_request: I/O
error, dev sdd, sector 19627674
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906558] block drbd4: disk(
UpToDate -> Failed )
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906563] block drbd4: Local
IO failed in drbd_endio_write_sec_final. Detaching...
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906579] block drbd4:
Notified peer that I am detaching my disk
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906604] block drbd4: disk(
Failed -> Diskless )
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906612] block drbd4:
Notified peer that I'm now diskless.
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906635] block drbd3:
receiver updated UUIDs to effective data uuid: 04CAF441EA88B615
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.906659] block drbd2:
Notified peer that I'm now diskless.
Mar 5 04:06:20 ifprdstor8b kernel: [4641532.907162] block drbd4:
receiver updated UUIDs to effective data uuid: 21FF04C86F9BC2B1
Mar 5 04:06:21 ifprdstor8b kernel: [4641534.025426] megasas: waitingfor
controller reset to finish
Mar 5 04:06:26 ifprdstor8b kernel: [4641539.045407] megasas: waitingfor
controller reset to finish
Mar 5 04:06:31 ifprdstor8b kernel: [4641543.891817] megasas: waitingfor
controller reset to finish
Mar 5 04:06:31 ifprdstor8b kernel: [4641544.065398] megasas: waitingfor
controller reset to finish
Mar 5 04:06:36 ifprdstor8b kernel: [4641548.909431] megasas: waitingfor
controller reset to finish
Mar 5 04:06:36 ifprdstor8b kernel: [4641549.085422] megasas: waitingfor
controller reset to finish
Mar 5 04:06:41 ifprdstor8b kernel: [4641553.929393] megasas: waitingfor
controller reset to finish
Mar 5 04:06:41 ifprdstor8b kernel: [4641554.105420] megasas: waitingfor
controller reset to finish
Mar 5 04:06:46 ifprdstor8b kernel: [4641558.949393] megasas: waitingfor
controller reset to finish
Mar 5 04:06:46 ifprdstor8b kernel: [4641559.125392] megasas: waitingfor
controller reset to finish
Mar 5 04:06:51 ifprdstor8b kernel: [4641563.969422] megasas: waitingfor
controller reset to finish
Mar 5 04:06:51 ifprdstor8b kernel: [4641564.145395] megasas: waitingfor
controller reset to finish
Mar 5 04:06:56 ifprdstor8b kernel: [4641568.989411] megasas: waitingfor
controller reset to finish
Mar 5 04:06:56 ifprdstor8b kernel: [4641569.165416] megasas: waitingfor
controller reset to finish
Mar 5 04:07:01 ifprdstor8b kernel: [4641574.009392] megasas: waitingfor
controller reset to finish
Mar 5 04:07:01 ifprdstor8b kernel: [4641574.185415] megasas: waitingfor
controller reset to finish
Mar 5 04:07:06 ifprdstor8b kernel: [4641579.029397] megasas: waitingfor
controller reset to finish
Mar 5 04:07:06 ifprdstor8b kernel: [4641579.205385] megasas: waitingfor
controller reset to finish
Mar 5 04:07:11 ifprdstor8b kernel: [4641584.049416] megasas: waitingfor
controller reset to finish
Mar 5 04:07:11 ifprdstor8b kernel: [4641584.225390] megasas: waitingfor
controller reset to finish
Mar 5 04:07:16 ifprdstor8b kernel: [4641589.069408] megasas: waitingfor
controller reset to finish
Mar 5 04:07:16 ifprdstor8b kernel: [4641589.245313] megasas: waitingfor
controller reset to finish
Mar 5 04:07:21 ifprdstor8b kernel: [4641594.089386] megasas: waitingfor
controller reset to finish
Mar 5 04:07:21 ifprdstor8b kernel: [4641594.265403] megasas: waitingfor
controller reset to finish
Mar 5 04:07:26 ifprdstor8b kernel: [4641599.109385] megasas: waitingfor
controller reset to finish
Mar 5 04:07:26 ifprdstor8b kernel: [4641599.285405] megasas: waitingfor
controller reset to finish
Mar 5 04:07:31 ifprdstor8b kernel: [4641604.129379] megasas: waitingfor
controller reset to finish
Mar 5 04:07:31 ifprdstor8b kernel: [4641604.305380] megaraid_sas: timed
out whilewaiting for HBA to recover
Mar 5 04:07:36 ifprdstor8b kernel: [4641609.149403] megasas: waitingfor
controller reset to finish
Mar 5 04:07:41 ifprdstor8b kernel: [4641614.169399] megasas: waitingfor
controller reset to finish
Mar 5 04:07:46 ifprdstor8b kernel: [4641619.189374] megasas: waitingfor
controller reset to finish
Mar 5 04:07:51 ifprdstor8b kernel: [4641624.209377] megasas: waitingfor
controller reset to finish
Mar 5 04:07:56 ifprdstor8b kernel: [4641629.229301] megasas: waitingfor
controller reset to finish
Mar 5 04:08:01 ifprdstor8b kernel: [4641634.249388] megasas: waitingfor
controller reset to finish
Mar 5 04:08:06 ifprdstor8b kernel: [4641639.269311] megasas: waitingfor
controller reset to finish
Mar 5 04:08:11 ifprdstor8b kernel: [4641644.289370] megasas: waitingfor
controller reset to finish
Mar 5 04:08:16 ifprdstor8b kernel: [4641649.309352] megasas: waitingfor
controller reset to finish
Mar 5 04:08:21 ifprdstor8b kernel: [4641654.329368] megasas: waitingfor
controller reset to finish
Mar 5 04:08:26 ifprdstor8b kernel: [4641659.349383] megasas: waitingfor
controller reset to finish
Mar 5 04:08:31 ifprdstor8b kernel: [4641664.369386] megasas: waitingfor
controller reset to finish
Mar 5 04:08:36 ifprdstor8b kernel: [4641669.389359] megasas: waitingfor
controller reset to finish
Mar 5 04:08:41 ifprdstor8b kernel: [4641674.409360] megasas: waitingfor
controller reset to finish
Mar 5 04:08:46 ifprdstor8b kernel: [4641679.429294] megasas: waitingfor
controller reset to finish
Mar 5 04:08:51 ifprdstor8b kernel: [4641684.449376] megasas: waitingfor
controller reset to finish
Mar 5 04:08:56 ifprdstor8b kernel: [4641689.469378] megasas: waitingfor
controller reset to finish
Mar 5 04:09:01 ifprdstor8b kernel: [4641694.489355] megasas: waitingfor
controller reset to finish
Mar 5 04:09:06 ifprdstor8b kernel: [4641699.509351] megasas: waitingfor
controller reset to finish
Mar 5 04:09:11 ifprdstor8b kernel: [4641704.529352] megasas: waitingfor
controller reset to finish
Mar 5 04:09:16 ifprdstor8b kernel: [4641709.549332] megasas: waitingfor
controller reset to finish
Mar 5 04:09:22 ifprdstor8b kernel: [4641714.569275] megasas: waitingfor
controller reset to finish
Mar 5 04:09:27 ifprdstor8b kernel: [4641719.589342] megasas: waitingfor
controller reset to finish
Mar 5 04:09:32 ifprdstor8b kernel: [4641724.609361] megaraid_sas: timed
out whilewaiting for HBA to recover
Mar 5 04:10:01 ifprdstor8b /USR/SBIN/CRON[32314]: (root) CMD
(/usr/local/sbin/vivaldi.sh 2>&1 > /dev/null)
Mar 5 04:15:01 ifprdstor8b /USR/SBIN/CRON[32403]: (root) CMD
(/usr/local/sbin/vivaldi.sh 2>&1 > /dev/null)
Mar 5 04:15:01 ifprdstor8b /USR/SBIN/CRON[32404]: (root) CMD (command
-v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 5 04:17:01 ifprdstor8b /USR/SBIN/CRON[32503]: (root) CMD ( cd /
&& run-parts --report /etc/cron.hourly)
Mar 5 04:17:09 ifprdstor8b kernel: [4642181.733805] megasas: waitingfor
controller reset to finish
Mar 5 04:17:14 ifprdstor8b kernel: [4642186.753216] megasas: waitingfor
controller reset to finish
Mar 5 04:17:19 ifprdstor8b kernel: [4642191.773193] megasas: waitingfor
controller reset to finish
Mar 5 04:17:24 ifprdstor8b kernel: [4642196.793191] megasas: waitingfor
controller reset to finish
Mar 5 04:17:29 ifprdstor8b kernel: [4642201.813215] megasas: waitingfor
controller reset to finish
Mar 5 04:17:34 ifprdstor8b kernel: [4642206.833209] megasas: waitingfor
controller reset to finish
Mar 5 04:17:39 ifprdstor8b kernel: [4642211.853190] megasas: waitingfor
controller reset to finish
Mar 5 04:17:44 ifprdstor8b kernel: [4642216.873206] megasas: waitingfor
controller reset to finish
Mar 5 04:17:49 ifprdstor8b kernel: [4642221.893210] megasas: waitingfor
controller reset to finish
Mar 5 04:17:54 ifprdstor8b kernel: [4642226.913180] megasas: waitingfor
controller reset to finish
Mar 5 04:17:59 ifprdstor8b kernel: [4642231.933195] megasas: waitingfor
controller reset to finish
Mar 5 04:18:04 ifprdstor8b kernel: [4642236.953208] megasas: waitingfor
controller reset to finish
Mar 5 04:18:09 ifprdstor8b kernel: [4642241.973199] megasas: waitingfor
controller reset to finish
Mar 5 04:18:14 ifprdstor8b kernel: [4642246.993174] megasas: waitingfor
controller reset to finish
Mar 5 04:18:19 ifprdstor8b kernel: [4642252.013190] megasas: waitingfor
controller reset to finish
Mar 5 04:18:24 ifprdstor8b kernel: [4642257.033192] megasas: waitingfor
controller reset to finish
Mar 5 04:18:29 ifprdstor8b kernel: [4642262.053198] megasas: waitingfor
controller reset to finish
Mar 5 04:18:34 ifprdstor8b kernel: [4642267.073168] megasas: waitingfor
controller reset to finish
Mar 5 04:18:39 ifprdstor8b kernel: [4642272.093173] megasas: waitingfor
controller reset to finish
Mar 5 04:18:42 ifprdstor8b shutdown[32581]: shutting down for system reboot
*On the master we have only log of timeout of scst (SORRY this san have
2 ours more than the slave :*
Mar 5 05:59:00 ifprdstor8a kernel: [4804404.774885] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff88021f5f40b8, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:00 ifprdstor8a kernel: [4804404.774896] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 32, lun_set
0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff8801d5515c00
Mar 5 05:59:00 ifprdstor8a kernel: [4804404.774904] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff88021f5f40b8 to active
mgmt cmd list
Mar 5 05:59:00 ifprdstor8a kernel: [4804404.775010] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff88021f5f40b8 from active cmd list
Mar 5 05:59:00 ifprdstor8a kernel: [4804404.775022] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff88022a75f868, sn -20, set 1,
tag 32, queue_type 1)
Mar 5 05:59:00 ifprdstor8a kernel: [4804404.775029] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff88022a75f868 (tag 32, op 2a)
Mar 5 05:59:00 ifprdstor8a kernel: [4804404.775037] [3269]:
scst_abort_cmd:5038:cmd ffff88022a75f868 (tag 32) is being executed
Mar 5 05:59:00 ifprdstor8a kernel: [4804404.775048] [3269]: scst:
scst_abort_cmd:5069:cmd ffff88022a75f868 (tag 32, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 60 sec., timeout 30 sec.),
deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff88021f5f40b8))
Mar 5 05:59:00 ifprdstor8a kernel: [4804404.775061] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:05 ifprdstor8a kernel: [4804409.774499] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff88021f5f4128, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:05 ifprdstor8a kernel: [4804409.774510] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 34, lun_set
0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff8801d6184000
Mar 5 05:59:05 ifprdstor8a kernel: [4804409.774518] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff88021f5f4128 to active
mgmt cmd list
Mar 5 05:59:05 ifprdstor8a kernel: [4804409.774622] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff88021f5f4128 from active cmd list
Mar 5 05:59:05 ifprdstor8a kernel: [4804409.774632] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff8801d6db1068, sn -20, set 1,
tag 34, queue_type 1)
Mar 5 05:59:05 ifprdstor8a kernel: [4804409.774640] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff8801d6db1068 (tag 34, op 2a)
Mar 5 05:59:05 ifprdstor8a kernel: [4804409.774647] [3269]:
scst_abort_cmd:5038:cmd ffff8801d6db1068 (tag 34) is being executed
Mar 5 05:59:05 ifprdstor8a kernel: [4804409.774658] [3269]: scst:
scst_abort_cmd:5069:cmd ffff8801d6db1068 (tag 34, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 65 sec., timeout 30 sec.),
deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff88021f5f4128))
Mar 5 05:59:05 ifprdstor8a kernel: [4804409.774669] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:10 ifprdstor8a kernel: [4804414.774095] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff88021f5f4198, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:10 ifprdstor8a kernel: [4804414.774106] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 1, lun_set 0,
lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff8801e1449400
Mar 5 05:59:10 ifprdstor8a kernel: [4804414.774114] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff88021f5f4198 to active
mgmt cmd list
Mar 5 05:59:10 ifprdstor8a kernel: [4804414.774216] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff88021f5f4198 from active cmd list
Mar 5 05:59:10 ifprdstor8a kernel: [4804414.774228] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff8801d6db1468, sn -20, set 1,
tag 1, queue_type 1)
Mar 5 05:59:10 ifprdstor8a kernel: [4804414.774235] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff8801d6db1468 (tag 1, op 2a)
Mar 5 05:59:10 ifprdstor8a kernel: [4804414.774243] [3269]:
scst_abort_cmd:5038:cmd ffff8801d6db1468 (tag 1) is being executed
Mar 5 05:59:10 ifprdstor8a kernel: [4804414.774253] [3269]: scst:
scst_abort_cmd:5069:cmd ffff8801d6db1468 (tag 1, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 70 sec., timeout 30 sec.),
deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff88021f5f4198))
Mar 5 05:59:10 ifprdstor8a kernel: [4804414.774266] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:15 ifprdstor8a kernel: [4804419.773696] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff88021f5f4208, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:15 ifprdstor8a kernel: [4804419.773707] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 54, lun_set
0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff880235dcb800
Mar 5 05:59:15 ifprdstor8a kernel: [4804419.773715] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff88021f5f4208 to active
mgmt cmd list
Mar 5 05:59:15 ifprdstor8a kernel: [4804419.773818] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff88021f5f4208 from active cmd list
Mar 5 05:59:15 ifprdstor8a kernel: [4804419.773829] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff880229d31c68, sn -20, set 1,
tag 54, queue_type 1)
Mar 5 05:59:15 ifprdstor8a kernel: [4804419.773836] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff880229d31c68 (tag 54, op 2a)
Mar 5 05:59:15 ifprdstor8a kernel: [4804419.773844] [3269]:
scst_abort_cmd:5038:cmd ffff880229d31c68 (tag 54) is being executed
Mar 5 05:59:15 ifprdstor8a kernel: [4804419.773855] [3269]: scst:
scst_abort_cmd:5069:cmd ffff880229d31c68 (tag 54, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 75 sec., timeout 30 sec.),
deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff88021f5f4208))
Mar 5 05:59:15 ifprdstor8a kernel: [4804419.773867] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:20 ifprdstor8a kernel: [4804424.773331] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff8801d5e76f28, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:20 ifprdstor8a kernel: [4804424.773342] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 55, lun_set
0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff8801e17fc400
Mar 5 05:59:20 ifprdstor8a kernel: [4804424.773350] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff8801d5e76f28 to active
mgmt cmd list
Mar 5 05:59:20 ifprdstor8a kernel: [4804424.773451] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff8801d5e76f28 from active cmd list
Mar 5 05:59:20 ifprdstor8a kernel: [4804424.773475] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff880233995c68, sn -20, set 1,
tag 55, queue_type 1)
Mar 5 05:59:20 ifprdstor8a kernel: [4804424.773490] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff880233995c68 (tag 55, op 2a)
Mar 5 05:59:20 ifprdstor8a kernel: [4804424.773503] [3269]:
scst_abort_cmd:5038:cmd ffff880233995c68 (tag 55) is being executed
Mar 5 05:59:20 ifprdstor8a kernel: [4804424.773520] [3269]: scst:
scst_abort_cmd:5069:cmd ffff880233995c68 (tag 55, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 80 sec., timeout 30 sec.),
deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff8801d5e76f28))
Mar 5 05:59:20 ifprdstor8a kernel: [4804424.773550] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:25 ifprdstor8a kernel: [4804429.772914] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff8801d5e76eb8, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:25 ifprdstor8a kernel: [4804429.772925] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 3, lun_set 0,
lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff8801e1719800
Mar 5 05:59:25 ifprdstor8a kernel: [4804429.772933] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff8801d5e76eb8 to active
mgmt cmd list
Mar 5 05:59:25 ifprdstor8a kernel: [4804429.773038] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff8801d5e76eb8 from active cmd list
Mar 5 05:59:25 ifprdstor8a kernel: [4804429.773049] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff8801d682c068, sn -20, set 1,
tag 3, queue_type 1)
Mar 5 05:59:25 ifprdstor8a kernel: [4804429.773056] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff8801d682c068 (tag 3, op 2a)
Mar 5 05:59:25 ifprdstor8a kernel: [4804429.773063] [3269]:
scst_abort_cmd:5038:cmd ffff8801d682c068 (tag 3) is being executed
Mar 5 05:59:25 ifprdstor8a kernel: [4804429.773074] [3269]: scst:
scst_abort_cmd:5069:cmd ffff8801d682c068 (tag 3, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 85 sec., timeout 30 sec.),
deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff8801d5e76eb8))
Mar 5 05:59:25 ifprdstor8a kernel: [4804429.773085] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:30 ifprdstor8a kernel: [4804434.772475] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff8801d5e76e48, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:30 ifprdstor8a kernel: [4804434.772486] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 46, lun_set
0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff880233bf7800
Mar 5 05:59:30 ifprdstor8a kernel: [4804434.772493] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff8801d5e76e48 to active
mgmt cmd list
Mar 5 05:59:30 ifprdstor8a kernel: [4804434.772598] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff8801d5e76e48 from active cmd list
Mar 5 05:59:30 ifprdstor8a kernel: [4804434.772608] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff880235a29468, sn -20, set 1,
tag 46, queue_type 1)
Mar 5 05:59:30 ifprdstor8a kernel: [4804434.772616] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff880235a29468 (tag 46, op 2a)
Mar 5 05:59:30 ifprdstor8a kernel: [4804434.772623] [3269]:
scst_abort_cmd:5038:cmd ffff880235a29468 (tag 46) is being executed
Mar 5 05:59:30 ifprdstor8a kernel: [4804434.772633] [3269]: scst:
scst_abort_cmd:5069:cmd ffff880235a29468 (tag 46, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 90 sec., timeout 30 sec.),
deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff8801d5e76e48))
Mar 5 05:59:30 ifprdstor8a kernel: [4804434.772645] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:35 ifprdstor8a kernel: [4804439.772125] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff8801d5e76dd8, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:35 ifprdstor8a kernel: [4804439.772136] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 50, lun_set
0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff8801e1555000
Mar 5 05:59:35 ifprdstor8a kernel: [4804439.772143] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff8801d5e76dd8 to active
mgmt cmd list
Mar 5 05:59:35 ifprdstor8a kernel: [4804439.772247] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff8801d5e76dd8 from active cmd list
Mar 5 05:59:35 ifprdstor8a kernel: [4804439.772258] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff8801e14f2c68, sn -20, set 1,
tag 50, queue_type 1)
Mar 5 05:59:35 ifprdstor8a kernel: [4804439.772265] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff8801e14f2c68 (tag 50, op 2a)
Mar 5 05:59:35 ifprdstor8a kernel: [4804439.772273] [3269]:
scst_abort_cmd:5038:cmd ffff8801e14f2c68 (tag 50) is being executed
Mar 5 05:59:35 ifprdstor8a kernel: [4804439.772284] [3269]: scst:
scst_abort_cmd:5069:cmd ffff8801e14f2c68 (tag 50, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 95 sec., timeout 30 sec.),
deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff8801d5e76dd8))
Mar 5 05:59:35 ifprdstor8a kernel: [4804439.772295] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:40 ifprdstor8a kernel: [4804444.771717] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff8801d5e76d68, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:40 ifprdstor8a kernel: [4804444.771728] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 21, lun_set
0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff8801d6859000
Mar 5 05:59:40 ifprdstor8a kernel: [4804444.771735] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff8801d5e76d68 to active
mgmt cmd list
Mar 5 05:59:40 ifprdstor8a kernel: [4804444.771839] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff8801d5e76d68 from active cmd list
Mar 5 05:59:40 ifprdstor8a kernel: [4804444.771849] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff8801d4f0d068, sn -20, set 1,
tag 21, queue_type 1)
Mar 5 05:59:40 ifprdstor8a kernel: [4804444.771857] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff8801d4f0d068 (tag 21, op 2a)
Mar 5 05:59:40 ifprdstor8a kernel: [4804444.771864] [3269]:
scst_abort_cmd:5038:cmd ffff8801d4f0d068 (tag 21) is being executed
Mar 5 05:59:40 ifprdstor8a kernel: [4804444.771875] [3269]: scst:
scst_abort_cmd:5069:cmd ffff8801d4f0d068 (tag 21, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 100 sec., timeout 30
sec.), deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff8801d5e76d68))
Mar 5 05:59:40 ifprdstor8a kernel: [4804444.771886] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:45 ifprdstor8a kernel: [4804449.771282] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff8801d5e76cf8, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:45 ifprdstor8a kernel: [4804449.771293] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 12, lun_set
0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff880229d52000
Mar 5 05:59:45 ifprdstor8a kernel: [4804449.771301] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff8801d5e76cf8 to active
mgmt cmd list
Mar 5 05:59:45 ifprdstor8a kernel: [4804449.771404] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff8801d5e76cf8 from active cmd list
Mar 5 05:59:45 ifprdstor8a kernel: [4804449.771415] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff8801d6dc0068, sn -20, set 1,
tag 12, queue_type 1)
Mar 5 05:59:45 ifprdstor8a kernel: [4804449.771422] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff8801d6dc0068 (tag 12, op 2a)
Mar 5 05:59:45 ifprdstor8a kernel: [4804449.771429] [3269]:
scst_abort_cmd:5038:cmd ffff8801d6dc0068 (tag 12) is being executed
Mar 5 05:59:45 ifprdstor8a kernel: [4804449.771440] [3269]: scst:
scst_abort_cmd:5069:cmd ffff8801d6dc0068 (tag 12, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 104 sec., timeout 30
sec.), deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff8801d5e76cf8))
Mar 5 05:59:45 ifprdstor8a kernel: [4804449.771451] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:50 ifprdstor8a kernel: [4804454.770900] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff8801d5e765f8, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:50 ifprdstor8a kernel: [4804454.770911] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 6, lun_set 0,
lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff880228eed800
Mar 5 05:59:50 ifprdstor8a kernel: [4804454.770918] [11385]:
scst_post_rx_mgmt_cmd:6291:Adding mgmt cmd ffff8801d5e765f8 to active
mgmt cmd list
Mar 5 05:59:50 ifprdstor8a kernel: [4804454.771023] [3269]:
scst_tm_thread:6176:Deleting mgmt cmd ffff8801d5e765f8 from active cmd list
Mar 5 05:59:50 ifprdstor8a kernel: [4804454.771034] [3269]:
scst_abort_task:5794:Aborting task (cmd ffff8802337cd068, sn -20, set 1,
tag 6, queue_type 1)
Mar 5 05:59:50 ifprdstor8a kernel: [4804454.771041] [3269]: scst:
scst_abort_cmd:4956:Aborting cmd ffff8802337cd068 (tag 6, op 2a)
Mar 5 05:59:50 ifprdstor8a kernel: [4804454.771049] [3269]:
scst_abort_cmd:5038:cmd ffff8802337cd068 (tag 6) is being executed
Mar 5 05:59:50 ifprdstor8a kernel: [4804454.771060] [3269]: scst:
scst_abort_cmd:5069:cmd ffff8802337cd068 (tag 6, sn 4294967276) being
executed/xmitted (state 120, op 2a, proc time 109 sec., timeout 30
sec.), deferring ABORT (cmd_done_wait_count 1, cmd_finish_wait_count 1,
internal 0, mcmd fn 0 (mcmd ffff8801d5e765f8))
Mar 5 05:59:50 ifprdstor8a kernel: [4804454.771071] [3269]: scst:
scst_set_mcmd_next_state:5110:cmd_done_wait_count(1) not 0, preparing to
wait
Mar 5 05:59:55 ifprdstor8a kernel: [4804459.770475] [11385]: scst:
scst_rx_mgmt_fn:6363:TM fn 0 (mcmd ffff8801d5e76668, initiator
0x0000000000000000003048ffff9df745, target ib_srpt_target_0)
Mar 5 05:59:55 ifprdstor8a kernel: [4804459.770486] [11385]:
scst_rx_mgmt_fn:6375:sess=ffff88022a4ab180, tag_set 1, tag 24, lun_set
0, lun=0, cmd_sn_set 0, cmd_sn 0, priv ffff880228f0ec00
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20140305/658dc552/attachment.htm>