[DRBD-user] umount /drbdpart takes >50 seconds
Harald Dunkel
harald.dunkel at aixigo.de
Fri Dec 14 09:32:14 CET 2018
Hi folks,
On 12/13/18 11:49 PM, Igor Cicimov wrote:
> On Fri, Dec 14, 2018 at 2:57 AM Lars Ellenberg <lars.ellenberg at linbit.com <mailto:lars.ellenberg at linbit.com>> wrote:
>
>
> Unlikely to have anything to do with DRBD.
>
> since you apparently can reproduce, monitor
> grep -e Dirty -e Writeback /proc/meminfo
> and slabtop before/during/after umount.
>
> Also check sysctl settings
> sysctl vm | grep dirty
>
Attached. Hope this helps.
>
> Good point, people running servers with huge amount of ram should understand there is also a huge amount of cache that needs to get flushed to the device before it gets removed.
>
I agree that the problem might be unrelated to drbd, but isn't sync
supposed to flush page cache to the block device? The sample test I
ran took 47 secs *after* the sync; see output.txt. sync itself took
just a few millisecs.
Is sync broken for drbd?
Regards
Harri
-------------- next part --------------
# mount | grep /data1
/dev/drbd1 on /data1 type ext4 (rw,noatime,stripe=256)
# grep -e Dirty -e Writeback /proc/meminfo
Dirty: 7488 kB
Writeback: 0 kB
WritebackTmp: 0 kB
# sysctl vm | grep dirty
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
# slabtop -o
Active / Total Objects (% used) : 72522435 / 72526157 (100.0%)
Active / Total Slabs (% used) : 1884685 / 1884685 (100.0%)
Active / Total Caches (% used) : 97 / 147 (66.0%)
Active / Total Size (% used) : 28241385.36K / 28242758.19K (100.0%)
Minimum / Average / Maximum Object : 0.01K / 0.39K / 23.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
20200656 20200648 0% 0.19K 480968 42 3847744K dentry
20089200 20089140 0% 1.06K 669640 30 21428480K ext4_inode_cache
13065156 13065156 100% 0.10K 335004 39 1340016K buffer_head
10687866 10687826 0% 0.09K 254473 42 1017892K kmalloc-96
6182976 6182913 0% 0.06K 96609 64 386436K kmalloc-64
829260 828753 0% 0.04K 8130 102 32520K ext4_extent_status
362816 362816 100% 0.12K 11338 32 45352K kmalloc-128
287140 287055 0% 0.57K 10255 28 164080K radix_tree_node
125776 125776 100% 0.07K 2246 56 8984K Acpi-Operand
90972 90972 100% 0.14K 3249 28 12996K ext4_groupinfo_4k
75310 75310 100% 0.05K 886 85 3544K ftrace_event_field
72450 72450 100% 0.13K 2415 30 9660K kernfs_node_cache
38912 38912 100% 0.06K 608 64 2432K pid
33920 33802 0% 0.03K 265 128 1060K kmalloc-32
28160 28160 100% 0.01K 55 512 220K kmalloc-8
27948 27948 100% 0.04K 274 102 1096K Acpi-Namespace
26082 26082 100% 0.09K 567 46 2268K anon_vma
25545 25545 100% 0.20K 655 39 5240K vm_area_struct
21504 21504 100% 0.02K 84 256 336K kmalloc-16
18285 18285 100% 0.59K 345 53 11040K inode_cache
18250 17520 0% 0.05K 250 73 1000K mbcache
16290 16290 100% 0.13K 543 30 2172K kvm_async_pf
16002 16002 100% 0.19K 381 42 3048K kmalloc-192
15680 15247 0% 0.25K 490 32 3920K filp
12624 12624 100% 0.66K 263 48 8416K proc_inode_cache
11648 11616 0% 0.50K 364 32 5824K kmalloc-512
11556 11360 0% 0.22K 321 36 2568K drbd_req
10336 10058 0% 0.25K 323 32 2584K pool_workqueue
10038 10038 100% 0.19K 239 42 1912K cred_jar
7888 7752 0% 0.12K 232 34 928K jbd2_journal_head
5920 5920 100% 0.12K 185 32 740K scsi_sense_cache
4992 4352 0% 0.06K 78 64 312K ext4_io_end
4896 4896 100% 0.04K 48 102 192K pde_opener
4448 4321 0% 1.00K 139 32 4448K kmalloc-1024
4224 4224 100% 0.03K 33 128 132K fscrypt_info
4080 4080 100% 0.05K 48 85 192K fscrypt_ctx
4048 4048 100% 0.70K 88 46 2816K shmem_inode_cache
3680 3680 100% 0.25K 115 32 920K skbuff_head_cache
3431 3431 100% 0.05K 47 73 188K Acpi-Parse
2890 2890 100% 0.02K 17 170 68K numa_policy
2688 2624 0% 0.50K 84 32 1344K skbuff_fclone_cache
2688 2688 100% 0.07K 48 56 192K eventpoll_pwq
2624 2624 100% 0.06K 41 64 164K dmaengine-unmap-2
2528 2528 100% 0.25K 79 32 632K proc_dir_entry
2499 2499 100% 0.08K 49 51 196K Acpi-State
2448 2448 100% 0.16K 48 51 384K sigqueue
2432 2432 100% 1.00K 76 32 2432K signal_cache
2397 2397 100% 0.62K 47 51 1504K task_group
2346 2346 100% 0.69K 51 46 1632K files_cache
2346 2346 100% 0.69K 51 46 1632K sock_inode_cache
2208 2208 100% 2.00K 138 16 4416K kmalloc-2048
2176 2176 100% 0.25K 68 32 544K kmalloc-256
2142 2142 100% 0.19K 51 42 408K dmaengine-unmap-16
2080 2080 100% 0.12K 65 32 260K seq_file
1886 1886 100% 0.09K 41 46 164K trace_event_file
1880 1880 100% 0.20K 47 40 376K file_lock_cache
1872 1872 100% 0.10K 48 39 192K blkdev_ioc
1728 1728 100% 0.12K 54 32 216K secpath_cache
1590 1590 100% 1.06K 53 30 1696K mm_struct
1536 1536 100% 0.25K 48 32 384K dquot
1536 1536 100% 0.12K 48 32 192K ext4_allocation_context
# time sync
real 0m0.014s
user 0m0.002s
sys 0m0.000s
# grep -e Dirty -e Writeback /proc/meminfo
Dirty: 820 kB
Writeback: 0 kB
WritebackTmp: 0 kB
# sysctl vm | grep dirty
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
# slabtop -o
Active / Total Objects (% used) : 72522200 / 72525840 (100.0%)
Active / Total Slabs (% used) : 1884676 / 1884676 (100.0%)
Active / Total Caches (% used) : 97 / 147 (66.0%)
Active / Total Size (% used) : 28241349.16K / 28242684.23K (100.0%)
Minimum / Average / Maximum Object : 0.01K / 0.39K / 23.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
20200656 20200648 0% 0.19K 480968 42 3847744K dentry
20089200 20089140 0% 1.06K 669640 30 21428480K ext4_inode_cache
13065156 13065156 100% 0.10K 335004 39 1340016K buffer_head
10687866 10687826 0% 0.09K 254473 42 1017892K kmalloc-96
6182976 6182913 0% 0.06K 96609 64 386436K kmalloc-64
829260 828753 0% 0.04K 8130 102 32520K ext4_extent_status
362816 362816 100% 0.12K 11338 32 45352K kmalloc-128
287140 287055 0% 0.57K 10255 28 164080K radix_tree_node
125776 125776 100% 0.07K 2246 56 8984K Acpi-Operand
90972 90972 100% 0.14K 3249 28 12996K ext4_groupinfo_4k
75310 75310 100% 0.05K 886 85 3544K ftrace_event_field
72450 72450 100% 0.13K 2415 30 9660K kernfs_node_cache
38976 38976 100% 0.06K 609 64 2436K pid
33920 33802 0% 0.03K 265 128 1060K kmalloc-32
28160 28160 100% 0.01K 55 512 220K kmalloc-8
27948 27948 100% 0.04K 274 102 1096K Acpi-Namespace
26082 26082 100% 0.09K 567 46 2268K anon_vma
25584 25584 100% 0.20K 656 39 5248K vm_area_struct
21504 21504 100% 0.02K 84 256 336K kmalloc-16
18285 18285 100% 0.59K 345 53 11040K inode_cache
18250 17520 0% 0.05K 250 73 1000K mbcache
16290 16290 100% 0.13K 543 30 2172K kvm_async_pf
16002 16002 100% 0.19K 381 42 3048K kmalloc-192
15680 15247 0% 0.25K 490 32 3920K filp
12624 12624 100% 0.66K 263 48 8416K proc_inode_cache
11648 11616 0% 0.50K 364 32 5824K kmalloc-512
10944 10640 0% 0.22K 304 36 2432K drbd_req
10528 10372 0% 0.25K 329 32 2632K pool_workqueue
10038 10038 100% 0.19K 239 42 1912K cred_jar
7888 7752 0% 0.12K 232 34 928K jbd2_journal_head
5920 5920 100% 0.12K 185 32 740K scsi_sense_cache
4992 4416 0% 0.06K 78 64 312K ext4_io_end
4896 4896 100% 0.04K 48 102 192K pde_opener
4448 4321 0% 1.00K 139 32 4448K kmalloc-1024
4224 4224 100% 0.03K 33 128 132K fscrypt_info
4080 4080 100% 0.05K 48 85 192K fscrypt_ctx
4048 4048 100% 0.70K 88 46 2816K shmem_inode_cache
3680 3680 100% 0.25K 115 32 920K skbuff_head_cache
3431 3431 100% 0.05K 47 73 188K Acpi-Parse
2890 2890 100% 0.02K 17 170 68K numa_policy
2688 2624 0% 0.50K 84 32 1344K skbuff_fclone_cache
2688 2688 100% 0.07K 48 56 192K eventpoll_pwq
2624 2624 100% 0.06K 41 64 164K dmaengine-unmap-2
2528 2528 100% 0.25K 79 32 632K proc_dir_entry
2499 2499 100% 0.08K 49 51 196K Acpi-State
2448 2448 100% 0.16K 48 51 384K sigqueue
2432 2432 100% 1.00K 76 32 2432K signal_cache
2397 2397 100% 0.62K 47 51 1504K task_group
2346 2346 100% 0.69K 51 46 1632K files_cache
2346 2346 100% 0.69K 51 46 1632K sock_inode_cache
2208 2208 100% 2.00K 138 16 4416K kmalloc-2048
2176 2176 100% 0.25K 68 32 544K kmalloc-256
2142 2142 100% 0.19K 51 42 408K dmaengine-unmap-16
2080 2080 100% 0.12K 65 32 260K seq_file
1886 1886 100% 0.09K 41 46 164K trace_event_file
1880 1880 100% 0.20K 47 40 376K file_lock_cache
1872 1872 100% 0.10K 48 39 192K blkdev_ioc
1728 1728 100% 0.12K 54 32 216K secpath_cache
1590 1590 100% 1.06K 53 30 1696K mm_struct
1536 1536 100% 0.25K 48 32 384K dquot
1536 1536 100% 0.12K 48 32 192K ext4_allocation_context
# time umount /data1
real 0m47.821s
user 0m0.000s
sys 0m47.793s
# grep -e Dirty -e Writeback /proc/meminfo
Dirty: 12 kB
Writeback: 0 kB
WritebackTmp: 0 kB
# sysctl vm | grep dirty
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
# slabtop -o
Active / Total Objects (% used) : 1153666 / 1203726 (95.8%)
Active / Total Slabs (% used) : 26750 / 26750 (100.0%)
Active / Total Caches (% used) : 97 / 147 (66.0%)
Active / Total Size (% used) : 292668.49K / 304711.85K (96.0%)
Minimum / Average / Maximum Object : 0.01K / 0.25K / 23.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
133014 120791 0% 0.19K 3167 42 25336K dentry
125776 125776 100% 0.07K 2246 56 8984K Acpi-Operand
106692 98365 0% 0.04K 1046 102 4184K ext4_extent_status
87360 85658 0% 1.06K 2912 30 93184K ext4_inode_cache
72450 72450 100% 0.13K 2415 30 9660K kernfs_node_cache
61035 56132 0% 0.10K 1565 39 6260K buffer_head
57664 53163 0% 0.06K 901 64 3604K kmalloc-64
56532 47839 0% 0.57K 2019 28 32304K radix_tree_node
39872 39872 100% 0.06K 623 64 2492K pid
33920 33800 0% 0.03K 265 128 1060K kmalloc-32
28160 28160 100% 0.01K 55 512 220K kmalloc-8
26542 26542 100% 0.09K 577 46 2308K anon_vma
26208 26208 100% 0.20K 672 39 5376K vm_area_struct
25398 24971 0% 0.04K 249 102 996K Acpi-Namespace
21504 21504 100% 0.02K 84 256 336K kmalloc-16
19295 17659 0% 0.05K 227 85 908K ftrace_event_field
18285 18285 100% 0.59K 345 53 11040K inode_cache
18250 17520 0% 0.05K 250 73 1000K mbcache
17094 13482 0% 0.09K 407 42 1628K kmalloc-96
16290 16290 100% 0.13K 543 30 2172K kvm_async_pf
15680 15247 0% 0.25K 490 32 3920K filp
14910 14813 0% 0.19K 355 42 2840K kmalloc-192
12624 12624 100% 0.66K 263 48 8416K proc_inode_cache
10944 10640 0% 0.22K 304 36 2432K drbd_req
10688 10399 0% 0.50K 334 32 5344K kmalloc-512
10528 9893 0% 0.12K 329 32 1316K kmalloc-128
10528 10372 0% 0.25K 329 32 2632K pool_workqueue
10038 10038 100% 0.19K 239 42 1912K cred_jar
7888 7752 0% 0.12K 232 34 928K jbd2_journal_head
5920 5920 100% 0.12K 185 32 740K scsi_sense_cache
4992 4416 0% 0.06K 78 64 312K ext4_io_end
4896 4896 100% 0.04K 48 102 192K pde_opener
4448 4321 0% 1.00K 139 32 4448K kmalloc-1024
4224 4224 100% 0.03K 33 128 132K fscrypt_info
4080 4080 100% 0.05K 48 85 192K fscrypt_ctx
4048 4048 100% 0.70K 88 46 2816K shmem_inode_cache
3680 3680 100% 0.25K 115 32 920K skbuff_head_cache
3431 3431 100% 0.05K 47 73 188K Acpi-Parse
2890 2890 100% 0.02K 17 170 68K numa_policy
2688 2624 0% 0.50K 84 32 1344K skbuff_fclone_cache
2688 2688 100% 0.07K 48 56 192K eventpoll_pwq
2624 2624 100% 0.06K 41 64 164K dmaengine-unmap-2
2604 2437 0% 0.14K 93 28 372K ext4_groupinfo_4k
2528 2528 100% 0.25K 79 32 632K proc_dir_entry
2499 2499 100% 0.08K 49 51 196K Acpi-State
2448 2448 100% 0.16K 48 51 384K sigqueue
2432 2432 100% 1.00K 76 32 2432K signal_cache
2397 2397 100% 0.62K 47 51 1504K task_group
2346 2346 100% 0.69K 51 46 1632K files_cache
2346 2346 100% 0.69K 51 46 1632K sock_inode_cache
2208 2208 100% 2.00K 138 16 4416K kmalloc-2048
2184 2184 100% 0.19K 52 42 416K dmaengine-unmap-16
2176 2176 100% 0.25K 68 32 544K kmalloc-256
2080 2080 100% 0.12K 65 32 260K seq_file
1886 1886 100% 0.09K 41 46 164K trace_event_file
1880 1880 100% 0.20K 47 40 376K file_lock_cache
1872 1872 100% 0.10K 48 39 192K blkdev_ioc
1728 1728 100% 0.12K 54 32 216K secpath_cache
1590 1590 100% 1.06K 53 30 1696K mm_struct
1536 1536 100% 0.25K 48 32 384K dquot
1536 1536 100% 0.12K 48 32 192K ext4_allocation_context
More information about the drbd-user
mailing list