On Fri, Jan 28, 2011 at 1:44 PM, Joseph Hauptmann <span dir="ltr"><<a href="mailto:joseph@digiconcept.net">joseph@digiconcept.net</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div text="#000000" bgcolor="#ffffff">
Yes, I did try that. Doesn't make much of a (speed) difference.<br>
<br>
It seems, that the problem is less that rm gets stuck for good, but
that it takes really long breaks (about 20 sec.) while deleting -
during those breaks the whole partition is stuck and iostat reports
100% utilization compared to ~95% while actually deleting files.</div></blockquote><div><br></div><div>What does I/O state report on the request queue? What's the average length? What's the avereage I/O request latency? I suspect they are high.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div text="#000000" bgcolor="#ffffff">
Could the "hang-time" be DRBD writing meta-information (internal in
my case) and blocking every other access as long the meta-data isn't
written to the disk? Of course there is also the ext3-journal that
has to be written, but still I don't see why it should take that
long: I'm currently timing how long it takes to delete a subdir with
285868 block-sized files in it (already more than 30 min).<br></div></blockquote><div><br></div><div><snip> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div text="#000000" bgcolor="#ffffff">The filesystem on resource 0 is ext3 with a block size of 4096 and lies on a SW-RAID5 (far from ideal - I know).</div></blockquote><div><br></div><div>Far from ideal is an understatement. You're actually using the worst possible RAID configuration: RAID stripe parity without a write cache. The performance you see will quickly become asymptotically bound by the performance a single spindle in your RAID group. You're running internal metadata (like a journal) on DRBD, and you're getting double small FUA writes from the filesystem journal in addition to DRBD barriers.</div>
<div><br></div><div>Turn off ext3 journalling. Turn off DRBD barriers. Run your maintenance to remove the files. Turn them both back on.</div><div><br></div><div>-JR</div></div>