https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16634 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 279.518552] kasan: GPF could be caused by NULL-ptr deref or user memory access [ 279.520881] general protection fault: 0000 1 SMP KASAN NOPTI [ 279.523366] CPU: 1 PID: 555 Comm: test Tainted: G O 5.4.148+ #7 [ 279.527232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 279.530776] RIP: 0010:lustre_set_wire_obdo+0x7e/0x570 [obdclass] [ 279.556223] Call Trace: [ 279.556666] osc_getattr+0x1eb/0x950 [osc] [ 279.558046] osc_iocontrol+0x4f1/0xe80 [osc] [ 279.559241] lov_iocontrol+0x4ba/0x5de0 [lov] [ 279.567254] ll_dir_ioctl+0x2834/0x17cc0 [lustre] [ 279.590353] do_vfs_ioctl+0x405/0x660 [ 279.590784] ksys_ioctl+0x5e/0x90 </pre> </div></div> Lustre commit: 9ddcdee2c8b9ec14986b93cf3180d946cd4869f7 <br/> <br/> <br/> Three server nodes and one client. Kernel version: Ubuntu-5.4.0-90.101 <br/> <br/> # MGS <br/> mkfs.lustre --fsname=lustre --mgs /dev/vda <br/> mount -t lustre /dev/vda /root/lustre-server <br/> <br/> # MDS <br/> mkfs.lustre --fsname=lustre --index=0 --mgsnode=$<a href='mailto:start_ip@tcp0'>start_ip@tcp0</a> --mdt /dev/vda <br/> mount -t lustre /dev/vda /root/lustre-server <br/> <br/> # OSS <br/> mkfs.lustre --ost --fsname=lustre --index=1 --reformat --mgsnode=$<a href='mailto:start_ip@tcp0'>start_ip@tcp0</a> /dev/vda <br/> mount -t lustre /dev/vda /root/lustre-server <br/> <br/> # Client <br/> mount -t lustre $<a href='mailto:start_ip@tcp0'>start_ip@tcp0</a>:/lustre /root/lustre-client LU-16634

Null pointer dereference in lustre_set_wire_obdo

Bug Critical Resolved Fixed Andreas Dilger Tao Lyu Sun, 12 Mar 2023 20:54:44 +0000 Wed, 31 Jan 2024 19:50:13 +0000 Wed, 31 May 2023 20:24:00 +0000 Lustre 2.16.0 0 7 <p>Hello <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=tao.lyu" class="user-hover" rel="tao.lyu">tao.lyu</a>, this bug needs to have more information in it before there is any chance someone could identify the issue.</p> <p>At a minimum, a stack trace and any console logs from before the crash. Preferably also "<tt>gdb&gt; list *(lustre_set_wire_obdo+NNN)</tt>" that shows which line in that function the crash happened. </p> <p>Hi, sorry for the incomplete information.</p> <p>This crashed location is lustre/obdclass/obdo.c:182</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@dfs:~/lustre-cserver# gdb /lib/modules/5.4.148+/updates/fs/obdclass.ko Reading symbols from /lib/modules/5.4.148+/updates/fs/obdclass.ko... (gdb) list *(lustre_set_wire_obdo+0x7e) 0xaa7ee is in lustre_set_wire_obdo (/root/lustre-cserver/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/obdclass/obdo.c:182). 177 */ 178 void lustre_set_wire_obdo(const struct obd_connect_data *ocd, 179 struct obdo *wobdo, 180 const struct obdo *lobdo) 181 { 182 *wobdo = *lobdo; 183 if (ocd == NULL) 184 return; 185 186 if (!(wobdo-&gt;o_valid &amp; OBD_MD_FLUID)) </pre> </div></div> <p>The stack trace is shown below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 279.516647] kasan: CONFIG_KASAN_INLINE enabled [ 279.518552] kasan: GPF could be caused by NULL-ptr deref or user memory access [ 279.520881] general protection fault: 0000 [#1] SMP KASAN NOPTI [ 279.523366] CPU: 1 PID: 555 Comm: test Tainted: G O 5.4.148+ #7 [ 279.527232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 279.530776] RIP: 0010:lustre_set_wire_obdo+0x7e/0x570 [obdclass] [ 279.533624] Code: 40 84 c6 0f 85 78 04 00 00 84 c9 0f 95 c2 0f 9e c0 84 c2 0f 85 68 04 00 00 48 ba 00 00 00 00 00 fc ff df 4c 89 e0 48 c1 e8 03 &lt;0f&gt; b6 0c 10 49 8d 84 24 cf 00 00 00 48 89 c6 83 e0 07 48 c1 ee 03 [ 279.540387] RSP: 0018:ff110002456ff678 EFLAGS: 00010206 [ 279.541933] RAX: 0000000000000003 RBX: ff1100024a8462a0 RCX: 0000000000000000 [ 279.543988] RDX: dffffc0000000000 RSI: 1fe22000495ac501 RDI: ff1100024a8462a0 [ 279.545662] RBP: ff1100024ad628e8 R08: ff11000245455ac0 R09: ffe21c0049508c52 [ 279.547314] R10: ffe21c0049508c51 R11: ff1100024a84628b R12: 0000000000000018 [ 279.548570] R13: ff1100024964b000 R14: 0000000000000018 R15: ff110002509536d8 [ 279.549797] FS: 00007ffff7fc4540(0000) GS:ff11000257280000(0000) knlGS:0000000000000000 [ 279.551165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 279.552129] CR2: 00007ffff7f4fffc CR3: 0000000247498001 CR4: 0000000000761ee0 [ 279.553346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 279.554550] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 279.555759] PKRU: 55555554 [ 279.556223] Call Trace: [ 279.556666] osc_getattr+0x1eb/0x950 [osc] [ 279.558046] osc_iocontrol+0x4f1/0xe80 [osc] [ 279.559241] lov_iocontrol+0x4ba/0x5de0 [lov] [ 279.567254] ll_dir_ioctl+0x2834/0x17cc0 [lustre] [ 279.590353] do_vfs_ioctl+0x405/0x660 [ 279.590784] ksys_ioctl+0x5e/0x90 [ 279.591175] __x64_sys_ioctl+0x16/0x20 [ 279.591626] do_syscall_64+0x48/0x140 [ 279.592038] entry_SYSCALL_64_after_hwframe+0x44/0xa9 </pre> </div></div> <p>After setup the file system, executing the binary compiled from attached poc.c triggers this bug.</p> <p>Tao, thanks for filing your report.</p> <blockquote> <p>After setup the file system, executing the binary compiled from attached poc.c triggers this bug.</p></blockquote> <p>Sorry, I don't see any <tt>poc.c</tt> attachment on this ticket. Alternately, if you consider this to be a security issue, you can email me the <tt>poc.c</tt> program at adilger@whamcloud.com and I can make it available internally for review until after we release a fix.</p> <p>Looking at the stack trace and command being run it looks like you are running some kind of ioctl/syscall fuzzer for a research project? It would be helpful to disclose this kind of information in the initial bug report.</p> <p>It will be interesting to see what the fuzzer is doing. </p> <p>Hi Andreas,</p> <p>Oh, I'm sorry. I forgot to upload the POC. I've sent it to you.</p> <p>Yes, we are testing lustre through a distributed fuzzing framework. It generally tests lustre by feeding lots of mutated syscall sequences.</p> <p>By the way, does lustre support to run servers and clients in the same node? <br/> We tried to compile and install server and client modules in the same node. Unfortunately, it reports the server and client modules are in conflict. <br/> Is this its nature?</p> <blockquote> <p>By the way, does lustre support to run servers and clients in the same node?<br/> We tried to compile and install server and client modules in the same node. Unfortunately, it reports the server and client modules are in conflict. </p></blockquote> <p>This is just a misunderstanding of how the packaging is done. If you build and install the "server" package it can also mount the exported filesystem as a client. The "client" package can only mount the exported filesystem and cannot be a server. The client packages have reduced functionality to make them more easily portable to different kernels/distros.</p> <p>Okay, got it. Thanks a lot for your explaination.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50314" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50314</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> misc: standardize handling of ioctl params<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 496b2b5190703ae6cec88568a6837f0347185fd9</p> <p>Hello <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=tao.lyu" class="user-hover" rel="tao.lyu">tao.lyu</a>, I've created a patch that should address the class of issues that you uncovered. The Gerrit link is above, and I will also attach the current version of the patch against the master branch to this ticket.</p> <p>There is one cosmetic defect in the current version that is causing regression test failures (returning <tt>-EINVAL</tt> instead of <tt>-ENOTTY</tt> in one test case), but it should be ok for your testing. </p> <p>Please let me know if this resolved the issue you hit. Once it has been reviewed and passes regression testing the patch will be landed to master and backpirted to the maintenance branches. </p> <p> Please file a separate Jira ticket you uncover other unrelated issues. </p> <p>PS: I will add "<tt>Reported-by: Tao Lyu &lt;tao.lyu@epfl.ch&gt;</tt>" in the updated version of the patch.</p> <p>Thanks, I'll test it later and reply you.</p> <p>Could you please take a look at another two issues I reported:<br/> <a href="https://jira.whamcloud.com/browse/LU-16617" title="A null-pointer dereference in osc_request.c:3358:function osc_iocontro" class="issue-link" data-issue-key="LU-16617"><del>LU-16617</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-16616" title="crash in osc_brw_prep_request() ASSERTION( page_count == 1 || (ergo(i == 0, poff + pg-&gt;count == PAGE_SIZE) ..." class="issue-link" data-issue-key="LU-16616">LU-16616</a></p> <p>Thanks!</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50331" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50331</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> build: improve checkpatch warnings<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b0b4408491d1414848db289c594524a58cffddba</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50333" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50333</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> misc: remove unnecessary ioctl typecasts<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2dfbbce062dfcfa7e08d1fe6f7745ba2ced0a6b0</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50334" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50334</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> obdclass: improve iocontrol error messages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 08517e8ffb81f2a3092057d4f905afd8e224bc6f</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50335" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50335</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> llite: move common ioctl code to ll_iocontrol()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0609508786a8192e6fa691f632433ee71b209eee</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50390" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50390</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> misc: replace obsolete ioctl numbers<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 882314ade0a2b51f3b119ac71d5c1841e48571f1</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50331/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50331/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> build: improve checkpatch warnings<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fe13a1b01c61801065571b4049e8c6bd544742b7</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50333/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50333/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> misc: remove unnecessary ioctl typecasts<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4a1465577e1310ce09834103f7136bef0e1a1eeb</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50334/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50334/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> obdclass: improve iocontrol error messages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1f4825eff026321a8e769739868d4d482026c3ee</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50335/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50335/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> llite: move common ioctl code to ll_iocontrol()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3be425883918528ef981af697b34de187c3937b3</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=tao.lyu" class="user-hover" rel="tao.lyu">tao.lyu</a>, have you been able to validate that the provided patch (or the latest patch <a href="https://review.whamcloud.com/50314" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50314</a> on master) fixes the issue reported here and in <a href="https://jira.whamcloud.com/browse/LU-16617" title="A null-pointer dereference in osc_request.c:3358:function osc_iocontro" class="issue-link" data-issue-key="LU-16617"><del>LU-16617</del></a>?</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50314/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50314/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> misc: standardize iocontrol param handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5eae8514f5f1730fe93d55d348400f5ecf681078</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50390/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50390/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> misc: replace obsolete ioctl numbers<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 594b780062db7e015837faefa75f7352293f54ae</p> <p>All patches seem to have landed for 2.16</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51596" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51596</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> build: remove obsolete checkpatch strings<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4566b6d452f2ccc6b89c23b1865f3a5b5d0b2038</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51596/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51596/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16634" title="Null pointer dereference in lustre_set_wire_obdo" class="issue-link" data-issue-key="LU-16634"><del>LU-16634</del></a> build: remove obsolete checkpatch strings<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cf50755cce8b19c9cbfb20c12137b0eb3c3b66da</p> Related LU-12613 LU-13107 LU-16616 LU-16617 LU-16688 LU-17489 Development Rank 1|i03g3r: Rank (Obsolete) 9223372036854775807 Severity