[DRBD-user] drbd_panic() in drbd_receiver.c

[Graham, Simon]

I too have been looking into this -- I agree with Damian and think it's
very important that DRBD never panic in cases like this if it is to be
used in an HA system -- I think the final approach has to be one of
fixing up underlying disk errors where possible and returning an error
to the caller where it is not possible to fix up.

In this specific case (NegDReply), it seems that it would be OK to
simply remove the panic() and complete the original request with an EIO
error or somesuch - this does mean adding a call to
drbd_bio_endio(bio,0) in addition to removing the panic() though.

Even if this is acceptable, there are a bunch of other places where
panic is currently done that, I think, also need to be changed,
including:

1. In drbd_set_state if the node is now Primary and does not have access
to good data; I think this can simply be removed
   since drbd_fail_request_early already returns a failure to the caller
in this case.

2. Failure to write bitmap to disk; not sure what the right answer is
here - any suggestions? (perhaps force the disk to be
   inconsistent in some manner that will require a complete resync?)

3. Failure to write meta data to disk; ditto above only harder -- if you
cant write to the meta-data area, you cant store data
   that indicates the contents are bad...

4. Received NegRSDReply -- during resync, SyncTarget gets error from
SyncSource; In this specific case, it seems to me that 
   a possible solution is to leave the block in question set in the
bitmap, ensure that the state is never set consistent 
   on the current SyncTarget and ensure that no matter what happens, the
current SyncSource remains the best source of data.
   A potential issue with this is that the SyncTarget will continue to
attempt to synchronize the block in question - since 
   it's still set in the bitmap it will eventually be found again when
the syncer wraps round - maybe that's OK though (so 
   long as there is some sort of delay between attempts)?

I am planning on implementing these, assuming there isn't any huge
disagreement on the approach and assuming it isn't already in
progress...

Perhaps we should take this discussion to drvd-dev?
Simon

PS: Once the panics are gone, there is a second phase required which is
to fix up underlying errors where possible -- for example, if the volume
is consistent on both sides and a read on the primary fails, not only
should the read be retried to the secondary but also the returned data
should be rewritten on the primary -- for a class of errors, this will
actually fix the problem as the disk will remap a bad block when the
write is done; is anyone working on this?

-----Original Message-----
From: drbd-user-bounces at linbit.com [mailto:drbd-user-bounces at linbit.com]
On Behalf Of Damian Pietras
Sent: Monday, July 03, 2006 7:11 AM
To: drbd-user at linbit.com
Subject: [DRBD-user] drbd_panic() in drbd_receiver.c

I found that in case when data are inconsistent and the peer says that
it has I/O errors, so that synchronization can't be finished, DRBD does
drbd_panic() which means kernel panic.

This is the code from got_NegDReply():

drbd_panic("Got NegDReply. WE ARE LOST. We lost our up-to-date
disk.\n");

        // THINK do we have other options, but panic?
        //       what about bio_endio, in case we don't panic ??


As a test I removed panic() and setting drbd_did_panic from drbd_panic()
to see what will happen. After diskonnecting the disk from SyncSource
peer, I/O errors occured, the secondary peer reported that (the above
message) and everything is running well. I've restarted the primary node
with the disk connected, the synchronization continues.

I'm doing those test because I don't like the idea that the whole
machine panics because of I/O errors on one DRBD device, I'd like to
have the box running with other DRBD devices available. From the
comment in the above code I can see that just commenting out the panic()
code isn't a good solution :)

Can you explain why panic() is used in this case instead of just
behaving like the connection has been lost?

-- 
Damian Pietras
_______________________________________________
drbd-user mailing list
drbd-user at lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user