https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16616 Lustre <p>An assertion in the Lustre client is triggered by the following calls, which leads to a kernel crash during executing the write syscall shown below:<br/>  <br/>  // Strace<br/> open("./file1", O_RDWR|O_CREAT|O_EXCL|O_TRUNC|O_DIRECT|O_LARGEFILE|O_NOFOLLOW, 000) = 3<br/> write(3, "#! ./file1\n}!F\270\v\21k;{\310T+z\311-\311@:\312\27S"..., 3595</p> Three server nodes and one client. Kernel version: Ubuntu-5.4.0-90.101 <br/> <br/> # MGS <br/> mkfs.lustre --fsname=lustre --mgs /dev/vda <br/> mount -t lustre /dev/vda /root/lustre-server <br/> <br/> # MDS <br/> mkfs.lustre --fsname=lustre --index=0 --mgsnode=$<a href='mailto:start_ip@tcp0'>start_ip@tcp0</a> --mdt /dev/vda <br/> mount -t lustre /dev/vda /root/lustre-server <br/> <br/> # OSS <br/> mkfs.lustre --ost --fsname=lustre --index=1 --reformat --mgsnode=$<a href='mailto:start_ip@tcp0'>start_ip@tcp0</a> /dev/vda <br/> mount -t lustre /dev/vda /root/lustre-server <br/> <br/> # Client <br/> mount -t lustre $<a href='mailto:start_ip@tcp0'>start_ip@tcp0</a>:/lustre /root/lustre-client LU-16616

crash in osc_brw_prep_request() ASSERTION( page_count == 1 || (ergo(i == 0, poff + pg->count == PAGE_SIZE) ...

Bug Critical Open Unresolved WC Triage Tao Lyu client Fri, 3 Mar 2023 13:13:35 +0000 Sat, 15 Apr 2023 00:12:44 +0000 0 3 <p>Tao,</p> <p>Can you please share the crash messages as well?  Specifically the stack trace and LBUG.  Also, what Lustre version are you running?  (The poc is appreciated, but we need some more general info.)</p> <p>Sure.</p> <p>Lustre commit: 9ddcdee2c8b9ec14986b93cf3180d946cd4869f7 </p> <p>The stack trace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@dfs:~# [ 154.265547] LustreError: 298:0:(osc_request.c:1819:osc_brw_prep_request()) ASSERTION( page_count == 1 || (ergo(i == 0, poff + pg-&gt;count == PAGE_SIZE) &amp;&amp; ergo(i &gt; 0 &amp;&amp; i &lt; page_count - 1, poff == 0 &amp;&amp; pg-&gt;count == PAGE_SIZE) &amp;&amp; ergo(i == page_count - 1, poff == 0)) ) failed: i: 0/2 pg: 000000005a02f487 off: 0, count: 3595 [ 154.268801] LustreError: 298:0:(osc_request.c:1819:osc_brw_prep_request()) LBUG [ 154.269714] Kernel panic - not syncing: LBUG [ 154.270224] CPU: 3 PID: 298 Comm: ptlrpcd_00_00 Tainted: G O 5.4.148+ #7 [ 154.271135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014 [ 154.272152] Call Trace: [ 154.272455] dump_stack+0x50/0x63 [ 154.272875] panic+0xfb/0x2bc [ 154.274264] lbug_with_loc.cold+0x2c/0x2c [libcfs] [ 154.275445] osc_brw_prep_request+0x5214/0x6d20 [osc] [ 154.280285] osc_build_rpc+0x1487/0x3770 [osc] [ 154.281284] osc_io_unplug0+0x2f0d/0x5110 [osc] [ 154.286077] brw_queue_work+0xbe/0x220 [osc] [ 154.287007] work_interpreter+0xb3/0x340 [ptlrpc] [ 154.287904] ptlrpc_check_set+0x1244/0x7a90 [ptlrpc] [ 154.291430] ptlrpcd+0x1296/0x23c0 [ptlrpc] [ 154.298351] kthread+0xfb/0x130 [ 154.299703] ret_from_fork+0x1f/0x40 [ 154.300279] Kernel Offset: 0xa400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 154.301513] ---[ end Kernel panic - not syncing: LBUG ]--- </pre> </div></div> <p>Tao,</p> <p>Why are you running based on 9ddcdee2c8b9ec14986b93cf3180d946cd4869f7 ?  Are you intending to test an unreleased version?  That's a recent-ish pull of our development branch.  We appreciate if people want to test it, but we don't recommend it for production.  Our current public maintenance release is b2_15.</p> <p>Tao,</p> <p>With the poc - it looks like maybe you converted that directly from the strace using some tool?  It's very odd to see "syscall(__NR_mmap, ... )" rather than mmap().  Are you able to replace any part of the oic with text and/or symbolic representations instead of all those hex values?  For example, it creates a file called 'tmpfile', but that string appears nowhere in poc.c, so I assume it must be encoded in there.</p> <p>Hi, Patrick,</p> <p>We are developing a bug-finding tool for distributed systems. In order to detect the newest bugs, we run the latest developing version.</p> <p>Yes, this is generated by our tool. It's for directly call syscalls instead of going to libc.<br/> Here is the strace results (I also posted in above description part):<br/> // Strace<br/> open("./file1", O_RDWR|O_CREAT|O_EXCL|O_TRUNC|O_DIRECT|O_LARGEFILE|O_NOFOLLOW, 000) = 3<br/> write(3, "#! ./file1\n}!F\270\v\21k;{\310T+z\311-\311@:\312\27S"..., 3595</p> <p>Tao,</p> <p>Ahh, thank you for explaining.  OK - I'll have to set up an Ubuntu VM on my end, the poc works fine on my RHEL kernel based system.  It is probably a difference in kernel versions.</p> <p>Is it practical for you to collect debug logs from the client if I give you instructions?  It's not particularly difficult.</p> <p>Sure, glad to help collect the debug logs.</p> <p>Hi Patrick,</p> <p>Would you mind debugging and fixing this bug?<br/> Thanks!</p> <p>Best,<br/> Tao</p> Related LU-16617 LU-16634 Development Rank 1|i03flz: Rank (Obsolete) 9223372036854775807 Severity