[DRBD-user] can't remove directory with a few million files

Marcelo Roccasalva roccas at gmail.com
Mon Jan 31 16:25:27 CET 2011

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Sat, Jan 29, 2011 at 12:48, Joseph Hauptmann
<joseph.hauptmann at digiconcept.net> wrote:
> Am 29.01.2011 00:48, schrieb J. Ryan Earl:
>> On Fri, Jan 28, 2011 at 1:44 PM, Joseph
>> Hauptmann<joseph at digiconcept.net>wrote:
>>>  Yes, I did try that. Doesn't make much of a (speed) difference.
>>> It seems, that the problem is less that rm gets stuck for good, but that
>>> it
>>> takes really long breaks (about 20 sec.) while deleting - during those
>>> breaks the whole partition is stuck and iostat reports 100% utilization
>>> compared to ~95% while actually deleting files.
>> What does I/O state report on the request queue?  What's the average
>> length?
>>  What's the avereage I/O request latency?  I suspect they are high.
> not really, even when the DRBD device blocks I/O-access completley for a few
> seconds.
>>> The filesystem on resource 0 is ext3  with a block size of 4096 and lies
>>> on
>>> a SW-RAID5 (far from ideal - I know).
>> Far from ideal is an understatement.  You're actually using the worst
>> possible RAID configuration: RAID stripe parity without a write cache.
>>  The
>> performance you see will quickly become asymptotically bound by the
>> performance a single spindle in your RAID group.  You're running internal
>> metadata (like a journal) on DRBD, and you're getting double small FUA
>> writes from the filesystem journal in addition to DRBD barriers.
>> Turn off ext3 journalling.  Turn off DRBD barriers.  Run your maintenance
>> to
>> remove the files.  Turn them both back on.
>> -JR
> Disconnecting the peer was the first thing I did. What really bugs me, is
> that deleting the same kind of files on an ext3 lvm that lies next to the
> drbd resource on the same md device takes about a minute for 100k files -
> without blocking access.

The fastest solution is to format the filesystem, but this is not
always possible. If you can remove al files in the directory, the
fastest I/O friendly command could be:

ionice -c 3 rm -rf /directory


"¿No será acaso que ésta vida moderna está teniendo más de moderna que
de vida?" (Mafalda)

More information about the drbd-user mailing list