Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
I too have been looking into this -- I agree with Damian and think it's very important that DRBD never panic in cases like this if it is to be used in an HA system -- I think the final approach has to be one of fixing up underlying disk errors where possible and returning an error to the caller where it is not possible to fix up. In this specific case (NegDReply), it seems that it would be OK to simply remove the panic() and complete the original request with an EIO error or somesuch - this does mean adding a call to drbd_bio_endio(bio,0) in addition to removing the panic() though. Even if this is acceptable, there are a bunch of other places where panic is currently done that, I think, also need to be changed, including: 1. In drbd_set_state if the node is now Primary and does not have access to good data; I think this can simply be removed since drbd_fail_request_early already returns a failure to the caller in this case. 2. Failure to write bitmap to disk; not sure what the right answer is here - any suggestions? (perhaps force the disk to be inconsistent in some manner that will require a complete resync?) 3. Failure to write meta data to disk; ditto above only harder -- if you cant write to the meta-data area, you cant store data that indicates the contents are bad... 4. Received NegRSDReply -- during resync, SyncTarget gets error from SyncSource; In this specific case, it seems to me that a possible solution is to leave the block in question set in the bitmap, ensure that the state is never set consistent on the current SyncTarget and ensure that no matter what happens, the current SyncSource remains the best source of data. A potential issue with this is that the SyncTarget will continue to attempt to synchronize the block in question - since it's still set in the bitmap it will eventually be found again when the syncer wraps round - maybe that's OK though (so long as there is some sort of delay between attempts)? I am planning on implementing these, assuming there isn't any huge disagreement on the approach and assuming it isn't already in progress... Perhaps we should take this discussion to drvd-dev? Simon PS: Once the panics are gone, there is a second phase required which is to fix up underlying errors where possible -- for example, if the volume is consistent on both sides and a read on the primary fails, not only should the read be retried to the secondary but also the returned data should be rewritten on the primary -- for a class of errors, this will actually fix the problem as the disk will remap a bad block when the write is done; is anyone working on this? -----Original Message----- From: drbd-user-bounces at linbit.com [mailto:drbd-user-bounces at linbit.com] On Behalf Of Damian Pietras Sent: Monday, July 03, 2006 7:11 AM To: drbd-user at linbit.com Subject: [DRBD-user] drbd_panic() in drbd_receiver.c I found that in case when data are inconsistent and the peer says that it has I/O errors, so that synchronization can't be finished, DRBD does drbd_panic() which means kernel panic. This is the code from got_NegDReply(): drbd_panic("Got NegDReply. WE ARE LOST. We lost our up-to-date disk.\n"); // THINK do we have other options, but panic? // what about bio_endio, in case we don't panic ?? As a test I removed panic() and setting drbd_did_panic from drbd_panic() to see what will happen. After diskonnecting the disk from SyncSource peer, I/O errors occured, the secondary peer reported that (the above message) and everything is running well. I've restarted the primary node with the disk connected, the synchronization continues. I'm doing those test because I don't like the idea that the whole machine panics because of I/O errors on one DRBD device, I'd like to have the box running with other DRBD devices available. From the comment in the above code I can see that just commenting out the panic() code isn't a good solution :) Can you explain why panic() is used in this case instead of just behaving like the connection has been lost? -- Damian Pietras _______________________________________________ drbd-user mailing list drbd-user at lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user