[DRBD-user] can't remove directory with a few million files

J. Ryan Earl oss at jryanearl.us
Sat Jan 29 00:48:08 CET 2011

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Fri, Jan 28, 2011 at 1:44 PM, Joseph Hauptmann <joseph at digiconcept.net>wrote:

>  Yes, I did try that. Doesn't make much of a (speed) difference.
> It seems, that the problem is less that rm gets stuck for good, but that it
> takes really long breaks (about 20 sec.) while deleting - during those
> breaks the whole partition is stuck and iostat reports 100% utilization
> compared to ~95% while actually deleting files.

What does I/O state report on the request queue?  What's the average length?
 What's the avereage I/O request latency?  I suspect they are high.

> Could the "hang-time" be DRBD writing meta-information (internal in my
> case) and blocking every other access as long the meta-data isn't written to
> the disk? Of course there is also the ext3-journal that has to be written,
> but still I don't see why it should take that long: I'm currently timing how
> long it takes to delete a subdir with 285868 block-sized files in it
> (already more than 30 min).


> The filesystem on resource 0 is ext3  with a block size of 4096 and lies on
> a SW-RAID5 (far from ideal - I know).

Far from ideal is an understatement.  You're actually using the worst
possible RAID configuration: RAID stripe parity without a write cache.  The
performance you see will quickly become asymptotically bound by the
performance a single spindle in your RAID group.  You're running internal
metadata (like a journal) on DRBD, and you're getting double small FUA
writes from the filesystem journal in addition to DRBD barriers.

Turn off ext3 journalling.  Turn off DRBD barriers.  Run your maintenance to
remove the files.  Turn them both back on.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20110128/0010e99c/attachment.htm>

More information about the drbd-user mailing list