Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
For anyone else who has this problem, we have reduced the time required to trim a 1.3TB volume from 3 days to 1.5 minutes. Initially, we used mdraid rto build a raid0 array with a 32K chunk size. We initialized it as a drbd disk, synced it, built an lvm logical volume on it, and created an ext4 filesystem on the volume. Creating the filesystem and tripping it took 3 days (each time, every time, across multiple tests). When running lsblk -D, we noticed that the DISC-MAX value for the array was only 32K, compared to 4GB for the SSD drive itself. We also noticed that the number matched the chunk size. We theorized that the small DISC-MAX value was responsible for the slow trim rate across the DRBD link. We deleted the array and built a new one with a 4MB chunk size. The DISC-MAX value changed 4MB, which is the max selectable chunk size (but still way below the other DISC-MAX values shown in lsblk -D). We realized that, when using mdadm, the DISK-MAX value ends up matching the array chunk size. Instead of using mdadm to build the array, we used LVM to create a striped logical volume and made that the backing device for drbd. Then lsblk -D showed a DISC-MAX size of 128MB. Creating an ext4 filesystem on it and trimming only took 1.5 minutes (across multiple tests). Somebody knowledgeable may be able to explain how DISC-MAX affects the trim speed, and why the DISC-MAX value is different when creating the array with mdadm versus lvm. -- Eric Robinson From: drbd-user-bounces at lists.linbit.com [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Eric Robinson Sent: Tuesday, August 01, 2017 3:28 PM To: drbd-user at lists.linbit.com Subject: [DRBD-user] DRBD and TRIM -- Slow! I have 6 x Samsung SSD 840 pro drives in an RAID 0 configuration (mdraid). When I write to a non-DRBD partition on the drive, bypassing caches, I get 400 MB/sec. When I trim a filesystem mounted on a non-DRBD partition, it finishes fast. When I write to a DRBD replicated volume, I get 80MB/sec, which is about when I would expect using protocol C over a gigabit network. When I trim a filesystem that is mounted on a drbd device it is extremely slow. It takes three days to trim a 1.2TB volume. Running iperf between nodes shows 900Mbits/sec bandwidth, <1 ms latency, no packet loss. Why is trimming a DRBD volume so slow? This makes the servers unusable. -- Eric Robinson -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20170803/10e3247b/attachment.htm>