https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14045 Lustre <p>Because of patch <a href="https://review.whamcloud.com/38967" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38967</a>, we can end up in a situation where osc_release_bounce_pages() mistakenly consider pages as fscrypt bounce pages, and tries to free them, as shown in the stack below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2020-10-18 15:26:49 [ 4462.081809][T14012] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 15:26:49 (1603049209) 2020-10-18 15:26:52 [ 4464.514691][T30281] BUG: kernel NULL pointer dereference, address: 0000000000000048 2020-10-18 15:26:52 [ 4464.524282][T30281] #PF: supervisor read access in kernel mode 2020-10-18 15:26:52 [ 4464.532011][T30281] #PF: error_code(0x0000) - not-present page 2020-10-18 15:26:52 [ 4464.539709][T30281] PGD 80000007edcce067 P4D 80000007edcce067 PUD 7f1306067 PMD 0 2020-10-18 15:26:52 [ 4464.549144][T30281] Oops: 0000 [#1] PREEMPT SMP PTI 2020-10-18 15:26:52 [ 4464.555851][T30281] CPU: 0 PID: 30281 Comm: ptlrpcd_00_04 Tainted: G W 5.7.0-rc7+ #1 2020-10-18 15:26:52 [ 4464.566720][T30281] Hardware name: Supermicro Super Server/To be filled by O.E.M., BIOS 2.0b 08/12/2016 2020-10-18 15:26:52 [ 4464.577932][T30281] RIP: 0010:mempool_free+0x12/0x80 2020-10-18 15:26:52 [ 4464.584690][T30281] Code: 60 e8 ff cc cc cc cc cc 0f 1f 44 00 00 e9 86 a3 08 00 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 85 ff 48 89 fd 53 74 1a 48 89 f3 &lt;8b&gt; 46 48 39 46 4c 7c 12 48 8b 73 58 48 8b 43 68 48 89 ef 5b 5d ff 2020-10-18 15:26:52 [ 4464.607734][T30281] RSP: 0018:ffffc9002414fcc0 EFLAGS: 00010282 2020-10-18 15:26:52 [ 4464.615423][T30281] RAX: ffff8887d44fb5e0 RBX: 0000000000000000 RCX: 0000000000000000 2020-10-18 15:26:52 [ 4464.625013][T30281] RDX: ffff888845abb780 RSI: 0000000000000000 RDI: ffffea001f553340 2020-10-18 15:26:52 [ 4464.634577][T30281] RBP: ffffea001f553340 R08: 0000000000000000 R09: 0000000000000000 2020-10-18 15:26:52 [ 4464.644109][T30281] R10: 0000000000000000 R11: 000000000000000f R12: 0000000000000000 2020-10-18 15:26:52 [ 4464.653614][T30281] R13: ffff8887d736c9f0 R14: 0000000000000010 R15: ffff888845abb780 2020-10-18 15:26:52 [ 4464.663095][T30281] FS: 0000000000000000(0000) GS:ffff88885e600000(0000) knlGS:0000000000000000 2020-10-18 15:26:52 [ 4464.673521][T30281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2020-10-18 15:26:52 [ 4464.681579][T30281] CR2: 0000000000000048 CR3: 00000007cf9fa004 CR4: 00000000001606f0 2020-10-18 15:26:52 [ 4464.691015][T30281] Call Trace: 2020-10-18 15:26:52 [ 4464.695751][T30281] brw_interpret+0xac/0xa60 [osc] 2020-10-18 15:26:52 [ 4464.702190][T30281] ? _raw_spin_unlock+0x29/0x50 2020-10-18 15:26:52 [ 4464.708490][T30281] ptlrpc_check_set+0x329/0x1790 [ptlrpc] 2020-10-18 15:26:52 [ 4464.715599][T30281] ptlrpcd_check+0x411/0x460 [ptlrpc] 2020-10-18 15:26:52 [ 4464.722318][T30281] ptlrpcd+0x278/0x300 [ptlrpc] 2020-10-18 15:26:52 [ 4464.728463][T30281] ? remove_wait_queue+0x60/0x60 2020-10-18 15:26:52 [ 4464.734667][T30281] kthread+0x12a/0x170 2020-10-18 15:26:52 [ 4464.739993][T30281] ? ptlrpcd_check+0x460/0x460 [ptlrpc] 2020-10-18 15:26:52 [ 4464.746745][T30281] ? kthread_bind+0x10/0x10 2020-10-18 15:26:52 [ 4464.752431][T30281] ret_from_fork+0x24/0x30 </pre> </div></div> LU-14045

Fix O_DIRECT and encrypted files

Bug Minor Resolved Fixed Sebastien Buisson Sebastien Buisson patch sec Mon, 19 Oct 2020 08:56:10 +0000 Thu, 7 Jan 2021 15:11:49 +0000 Sat, 7 Nov 2020 14:35:54 +0000 Lustre 2.14.0 0 6 <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/40295" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40295</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14045" title="Fix O_DIRECT and encrypted files" class="issue-link" data-issue-key="LU-14045"><del>LU-14045</del></a> sec: fix O_DIRECT and encrypted files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 15384abdb2cb0236bcd60e7f754dbc59e2a16f57</p> <p>Here is an occurence of the crash hit without fix <a href="https://review.whamcloud.com/40295" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40295</a></p> <p>Test:<br/> <a href="https://testing.whamcloud.com/test_sets/8383343d-2440-49c5-992b-861551c6b407" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8383343d-2440-49c5-992b-861551c6b407</a><br/> Crash dump:<br/> <a href="https://testing.whamcloud.com/test_logs/1bf73796-3256-48a3-bab2-75f6d96c1180/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_logs/1bf73796-3256-48a3-bab2-75f6d96c1180/show_text</a></p> <p><a href="https://testing.whamcloud.com/test_sessions/8cfbafe5-ac90-4a05-831f-e9f636a229a5" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/8cfbafe5-ac90-4a05-831f-e9f636a229a5</a></p> <p>+2 with recent master at <a href="https://testing.whamcloud.com/test_sets/24317b7d-ea90-4b01-ae0a-e01b5284c227" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/24317b7d-ea90-4b01-ae0a-e01b5284c227</a> and <a href="https://testing.whamcloud.com/test_sets/fb2d522c-391e-4979-a709-c6c4d8a967a0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/fb2d522c-391e-4979-a709-c6c4d8a967a0</a></p> <p>I may be conflating two issues, but AFAICS, sanity test_56w has only crashed a couple of times in the past 4 weeks:<br/> <a href="https://testing.whamcloud.com/test_sets/5850ea8a-7bc0-40a1-b88b-5aabd945fe10" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5850ea8a-7bc0-40a1-b88b-5aabd945fe10</a><br/> <a href="https://testing.whamcloud.com/test_sets/6dadd2cd-29c9-4965-9ecd-433452337956" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6dadd2cd-29c9-4965-9ecd-433452337956</a></p> <p>and those were both on 2020-10-10 when testing patch <a href="https://review.whamcloud.com/38883" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38883</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11621" title="Add copy_file_range() API and use it for lfs migrate and mirror resync" class="issue-link" data-issue-key="LU-11621">LU-11621</a> utils: optimize migrate_copy_data() with copy_file_range()</tt>".</p> <p>The only other crash started on <del>aarch64</del> kernels 4.18+ on 2020-10-19, but the patch <a href="https://review.whamcloud.com/38967" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38967</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12275" title="Client-side file data encryption" class="issue-link" data-issue-key="LU-12275"><del>LU-12275</del></a> sec: O_DIRECT for encrypted file</tt>" was landed on master 6 weeks ago. This is failing 100% in sanity test_426 since patch <a href="https://review.whamcloud.com/39695" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39695</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13745" title="tasks hang with copy_file_range: ll_file_splice_read()" class="issue-link" data-issue-key="LU-13745">LU-13745</a> test: add splice test for lustre</tt>" landed, since that patch was submitted with "<tt>Test-Parameters: trivial</tt>" which only tests x86_64 on ldiskfs, but the test is crashing continuously on aarch64 and el8.2, both of which are using 4.18 kernels.</p> <p>If this is related to crypto, it appears the source of the funky pages is the splice IO from "splice". The two failed sanity test_56w are testing <tt>copy_file_range()</tt> that is also using in-kernel data copying, similar to splice. Since the pages are generated in a source filesystem and sent to the target, it isn't whether we can play games with the mapping or not, so it might be better to use a page flag (e.g. <tt>PageChecked</tt>, maybe with a better wrapper like <tt>PageCrypto</tt> for Lustre)?</p> <p>Stack trace from sanity.sh test_426:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[15000.400779] Lustre: DEBUG MARKER: == sanity test 426: splice test on Lustre ==== 20:58:26 (1603227506) [15001.080742] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000004 [15001.102937] user pgtable: 64k pages, 48-bit VAs, pgdp = 000000009f14b2d0 [15001.111120] Internal error: Oops: 96000005 [#1] SMP [15001.149680] CPU: 1 PID: 11273 Comm: ptlrpcd_01_01 4.18.0-147.8.1.el8_1.aarch64 #1 [15001.164523] pc : mempool_free+0x24/0xe0 [15001.167022] lr : llcrypt_free_bounce_page.part.1+0x38/0x48 [libcfs] [15001.223444] Process ptlrpcd_01_01 (pid: 11273, stack limit = 0x00000000f9135a93) [15001.228185] Call trace: [15001.229806] mempool_free+0x24/0xe0 [15001.232143] llcrypt_free_bounce_page.part.1+0x38/0x48 [libcfs] [15001.236007] llcrypt_free_bounce_page+0x24/0x30 [libcfs] [15001.239541] brw_interpret+0x124/0x10c8 [osc] [15001.242729] ptlrpc_check_set+0x688/0x3318 [ptlrpc] [15001.246031] ptlrpcd_check+0x470/0x820 [ptlrpc] [15001.249060] ptlrpcd+0x3d4/0x5c8 [ptlrpc] [15001.251673] kthread+0x130/0x138 </pre> </div></div> <p>I've pushed patch <a href="https://review.whamcloud.com/40326" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40326</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13745" title="tasks hang with copy_file_range: ll_file_splice_read()" class="issue-link" data-issue-key="LU-13745">LU-13745</a> tests: skip sanity test_426 for 4.18+</tt>" to skip this test until the issue is resolved.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40295/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40295/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14045" title="Fix O_DIRECT and encrypted files" class="issue-link" data-issue-key="LU-14045"><del>LU-14045</del></a> sec: fix O_DIRECT and encrypted files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e07d0516dcde4b23375881077875b4cf96c90cd5</p> <p>Landed for 2.14</p> Related LU-13745 LU-12275 LU-14306 Development Rank 1|i01cm7: Rank (Obsolete) 9223372036854775807 Severity