Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi,
as of yesterday we have a strange problem occuring on our drbd-device.
drbd-config, dumpe2fs can be found here:
http://pastebin.com/8NZprdLz
The disk-setup is as follow:
/dev/drbd0 -> LVM -> LV -> ext3 (394GB) -> /data/nfs
On top of our DRBD-device we have LVM with two groups. One logical
Volume is formatted as ext3 and mounted to /data/nfs. This directory is
then exported via nfs to a couple of machines.
Yesterday during normal daily operation (meaning nothing special like
more or less I/O than usual), i was not able to create files in
/data/nfs anymore, even though i had and have about 130GB available.
$ dd if=/dev/zero of="file.foo.${RANDOM}" count=1024 bs=2048
"file.foo.26653: no space left on device"
After deleting a few files, i can create a new file without any errors.
This procedure can be repeated with the same result. If the above error
occurs, the file is still created but not fully written. Just like if
there was not enough space.
After that i deleted a couple thousand files to run more tests.
I created a new directory and played around in it with dd. Right after
creating the 15066th file, the error occured. Deleting a few files and
recreating them showed, that the error could be reproduced everytime the
15066th file was created. But that number changed going up and down
(10-20+-) while repeating the test. I dont think, that the number
actually means something. It just shows, that we seem to be hitting some
kind of boundary or limitation we dont know about. Since this is an
nfs-share with 10 machines connected, numerous files might be
changed/created/deleted in parallel thus explaining the movement in the
limitation described.
I checked various things i could think of:
- plenty of inodes available
- plenty of blocks available
- plenty of inodes/blocks is also true for the remote
- this also happens when the remote is not connected
- reserved blocks for root is only 5%, leaving plenty available
- this is not related to the ext3-subdirectory-limit (31999), i verified
that by creating 31999 directory in my test-folder. that yields "too
many links" as an error
Dont get me wrong, i'm not saying this is DRBDs fault. But since it is
involved and everything above DRBD seems(!) to be ok, i think its ok to
ask here.
Thanks for reading :-)
- volker