https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13601 Lustre <p>A customer mounts several lustre fs , 10th fs is failed to mount due to a memory allocation failure.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[7335638.554981] mount.lustre: page allocation failure: order:4, mode:0xc050 [7335638.554990] CPU: 0 PID: 303506 Comm: mount.lustre Kdump: loaded Tainted: G OE ------------ T 3.10.0-957.41.1.el7.x86 _64 #1 [7335638.554992] Hardware name: Penguin Computing XE2142e-OEM/S2600BPS, BIOS SE5C620.86B.00.01.0016.020120190930 02/01/2019 [7335638.554994] Call Trace: [7335638.555012] [&lt;ffffffffae565ac0&gt;] dump_stack+0x19/0x1b [7335638.555022] [&lt;ffffffffadfbe200&gt;] warn_alloc_failed+0x110/0x180 [7335638.555025] [&lt;ffffffffadfc2cbf&gt;] __alloc_pages_nodemask+0x9df/0xbe0 [7335638.555029] [&lt;ffffffffae00fce8&gt;] alloc_pages_current+0x98/0x110 [7335638.555035] [&lt;ffffffffadfddb68&gt;] kmalloc_order+0x18/0x40 [7335638.555042] [&lt;ffffffffae01b066&gt;] kmalloc_order_trace+0x26/0xa0 [7335638.555095] [&lt;ffffffffc0f94eec&gt;] ll_init_sbi+0x4c/0x660 [lustre] [7335638.555154] [&lt;ffffffffc0ae7be2&gt;] ? lustre_start_mgc+0x4d2/0x2b00 [obdclass] [7335638.555157] [&lt;ffffffffae01df36&gt;] ? kmem_cache_alloc_trace+0x1d6/0x200 [7335638.555169] [&lt;ffffffffc0fa145a&gt;] ? ll_fill_super+0x7a/0x14e0 [lustre] [7335638.555179] [&lt;ffffffffc0fa14b5&gt;] ll_fill_super+0xd5/0x14e0 [lustre] [7335638.555196] [&lt;ffffffffc0aeac04&gt;] lustre_fill_super+0x264/0xb70 [obdclass] [7335638.555210] [&lt;ffffffffc0aea9a0&gt;] ? lustre_common_put_super+0x270/0x270 [obdclass] [7335638.555216] [&lt;ffffffffae046e3f&gt;] mount_nodev+0x4f/0xb0 [7335638.555230] [&lt;ffffffffc0ae07c8&gt;] lustre_mount+0x38/0x60 [obdclass] [7335638.555232] [&lt;ffffffffae0479be&gt;] mount_fs+0x3e/0x1b0 [7335638.555237] [&lt;ffffffffae065607&gt;] vfs_kern_mount+0x67/0x110 [7335638.555238] [&lt;ffffffffae067c2f&gt;] do_mount+0x1ef/0xce0 [7335638.555242] [&lt;ffffffffae03fe1a&gt;] ? __check_object_size+0x1ca/0x250 [7335638.555244] [&lt;ffffffffae01dd9c&gt;] ? kmem_cache_alloc_trace+0x3c/0x200 [7335638.555246] [&lt;ffffffffae068a63&gt;] SyS_mount+0x83/0xd0 [7335638.555249] [&lt;ffffffffae578ddb&gt;] system_call_fastpath+0x22/0x27 [7335638.555251] Mem-Info: [7335638.555264] active_anon:85113660 inactive_anon:4989547 isolated_anon:0 active_file:1793179 inactive_file:2080266 isolated_file:0 unevictable:0 dirty:26440 writeback:34 unstable:0 slab_reclaimable:836114 slab_unreclaimable:1111382 mapped:810319 shmem:25487119 pagetables:149686 bounce:0 free:243878 free_pcp:287 free_cma:0 </pre> </div></div> LU-13601

page allocation failure during mount

Bug Major Resolved Fixed Andreas Dilger Alexander Zarochentsev Mon, 25 May 2020 19:31:22 +0000 Fri, 3 Dec 2021 21:38:32 +0000 Fri, 3 Dec 2021 21:38:32 +0000 Upstream Lustre 2.12.4 Lustre 2.15.0 0 5 <p>Allocation of struct ll_sb_info fails in ll_init_sbi() b/c memory is fragmented and ll_sb_info is more than 50kB in size, requires order 4 allocation.<br/> A quick trace of page allocation during mount/unmount and a simple file copy test shows no other page allocation with order &gt; 3 not protected by OBD_ALLOC_LARGE except<br/> this ll_init_sbi()</p> <p>Alexander Zarochentsev (alexander.zarochentsev@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/38713" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38713</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13601" title="page allocation failure during mount " class="issue-link" data-issue-key="LU-13601"><del>LU-13601</del></a> llite: OBD_ALLOC_LARGE for ll_sb_info<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 541cc5cf186d5de2a4fb624219acad7f35a19f1b</p> <p>It looks like the root cause of this very large allocation is because of <tt>ll_rw_extents_info</tt> and <tt>ll_rw_process_info</tt> stats that are almost never used:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ pahole lustre/llite/lustre.ko | grep -A 80 '^struct ll_sb_info' struct ll_sb_info { spinlock_t ll_lock; /* 0 4 */ spinlock_t ll_pp_extent_lock; /* 4 4 */ spinlock_t ll_process_lock; /* 8 4 */ struct obd_uuid ll_sb_uuid; /* 12 40 */ /* XXX 4 bytes hole, try to pack */ struct obd_export * ll_md_exp; /* 56 8 */ /* --- cacheline 1 boundary (64 bytes) --- */ struct obd_export * ll_dt_exp; /* 64 8 */ struct obd_device * ll_md_obd; /* 72 8 */ struct obd_device * ll_dt_obd; /* 80 8 */ struct dentry * ll_debugfs_entry; /* 88 8 */ struct lu_fid ll_root_fid; /* 96 16 */ int ll_flags; /* 112 4 */ unsigned int ll_umounting:1; /* 116:31 4 */ unsigned int ll_xattr_cache_enabled:1; /* 116:30 4 */ unsigned int ll_xattr_cache_set:1; /* 116:29 4 */ unsigned int ll_client_common_fill_super_succeeded:1; /* 116:28 4 */ unsigned int ll_checksum_set:1; /* 116:27 4 */ /* XXX 27 bits hole, try to pack */ struct lustre_client_ocd ll_lco; /* 120 56 */ /* --- cacheline 2 boundary (128 bytes) was 48 bytes ago --- */ struct lprocfs_stats * ll_stats; /* 176 8 */ struct cl_client_cache * ll_cache; /* 184 8 */ /* --- cacheline 3 boundary (192 bytes) --- */ struct lprocfs_stats * ll_ra_stats; /* 192 8 */ struct ll_ra_info ll_ra_info; /* 200 32 */ unsigned int ll_namelen; /* 232 4 */ /* XXX 4 bytes hole, try to pack */ struct file_operations * ll_fop; /* 240 8 */ struct lu_site * ll_site; /* 248 8 */ /* --- cacheline 4 boundary (256 bytes) --- */ struct cl_device * ll_cl; /* 256 8 */ struct ll_rw_extents_info ll_rw_extents_info; /* 264 5896 */ /* --- cacheline 96 boundary (6144 bytes) was 16 bytes ago --- */ int ll_extent_process_count; /* 6160 4 */ /* XXX 4 bytes hole, try to pack */ struct ll_rw_process_info ll_rw_process_info[10]; /* 6168 640 */ /* --- cacheline 106 boundary (6784 bytes) was 24 bytes ago --- */ unsigned int ll_offset_process_count; /* 6808 4 */ enum stats_track_type ll_stats_track_type; /* 13224 4 */ int ll_rw_stats_on; /* 13228 4 */ unsigned int ll_sa_running_max; /* 13232 4 */ unsigned int ll_sa_max; /* 13236 4 */ atomic_t ll_sa_total; /* 13240 4 */ atomic_t ll_sa_wrong; /* 13244 4 */ /* --- cacheline 207 boundary (13248 bytes) --- */ atomic_t ll_sa_running; /* 13248 4 */ atomic_t ll_agl_total; /* 13252 4 */ dev_t ll_sdev_orig; /* 13256 4 */ /* XXX 4 bytes hole, try to pack */ struct root_squash_info ll_squash; /* 13264 56 */ /* --- cacheline 208 boundary (13312 bytes) was 8 bytes ago --- */ struct path ll_mnt; /* 13320 16 */ unsigned int ll_stat_blksize; /* 13336 4 */ unsigned int ll_statfs_max_age; /* 13340 4 */ struct kset ll_kset; /* 13344 96 */ /* --- cacheline 210 boundary (13440 bytes) --- */ struct completion ll_kobj_unregister; /* 13440 32 */ /* size: 13472, cachelines: 211, members: 47 */ /* sum members: 13452, holes: 5, sum holes: 20 */ /* bit holes: 1, sum bit holes: 27 bits */ /* last cacheline: 32 bytes */ }; </pre> </div></div> <p>What about dynamically allocating those stats structures when they are first used, and freeing them at unmount time? That would bring the <tt>ll_sb_info</tt> allocation size down to 1176 bytes for almost all uses, and it would never fail allocation at that size.</p> <p>Andreas, we have <a href="https://review.whamcloud.com/#/c/31236/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31236/</a> landed to our branch and it increases size of ll_sb_info up to 52kB, while master's version has only </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(gdb) p sizeof(struct ll_sb_info) $1 = 13568 (gdb) </pre> </div></div> <p>I am going to abandon the patch.</p> <p>I think especially in this case it would make sense to allocate the stats structs in the superblock dynamically when these stats are first enabled. </p> <p>So should this ticket be closed as Will Not Fix?</p> <p>I'd rather fix the problem than close the ticket. Allocating <tt>ll_rw_extents_info</tt> (5896 bytes), <tt>ll_rw_offset_info</tt> (6400 bytes), and <tt>ll_rw_process_info</tt> (640 bytes) only when these stats are enabled via <tt>ll_rw_extents_stats_pp_seq_write()</tt>, <tt>ll_rw_extents_stats_seq_write()</tt>, or <tt>ll_rw_offset_stats_seq_write()</tt> is straight forward to implement, and should really be done before patch <a href="https://review.whamcloud.com/31236" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31236</a> lands.</p> <p>The <tt>struct obd_histogram</tt> in <tt>struct client_obd</tt> and <tt>struct lmv_obd</tt> are also very large. I'll push a prototype patch which allocates the histograms on demand.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40901" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40901</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13601" title="page allocation failure during mount " class="issue-link" data-issue-key="LU-13601"><del>LU-13601</del></a> llite: avoid needless large allocations<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f1cf5c17ddd59a32bd6b0f0de1bb4e65f1009424</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/40901/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40901/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13601" title="page allocation failure during mount " class="issue-link" data-issue-key="LU-13601"><del>LU-13601</del></a> llite: avoid needless large stats alloc<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9490fd9bb84dc277bd103bf16286fc26882e5b5e</p> Related LU-4533 LU-14055 Development Rank 1|i01173: Rank (Obsolete) 9223372036854775807 Severity