https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2170 Lustre <p>Hit this assertion running racer in a loop in a single node setup:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 8788.630974] LustreError: 4352:0:(vvp_io.c:1038:vvp_io_commit_write()) Skipped 833 previous similar messages [ 8834.780249] LustreError: 22605:0:(osc_cache.c:506:osc_extent_merge()) ASSERTION( cur-&gt;oe_osclock == victim-&gt;oe_osclock ) failed: [ 8834.780830] LustreError: 22605:0:(osc_cache.c:506:osc_extent_merge()) LBUG [ 8834.781139] Pid: 22605, comm: cat [ 8834.781364] [ 8834.781364] Call Trace: [ 8834.781773] [&lt;ffffffffa0c9e915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [ 8834.782075] [&lt;ffffffffa0c9ef27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [ 8834.782366] [&lt;ffffffffa04ce847&gt;] osc_extent_merge+0x697/0x6a0 [osc] [ 8834.782657] [&lt;ffffffff81112df1&gt;] ? generic_file_buffered_write+0x1f1/0x300 [ 8834.782947] [&lt;ffffffff8106fea7&gt;] ? current_fs_time+0x27/0x30 [ 8834.783209] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [ 8834.783474] [&lt;ffffffffa04d601c&gt;] osc_extent_release+0xfc/0x3e0 [osc] [ 8834.783782] [&lt;ffffffffa059a610&gt;] ? lov_io_end_wrapper+0x0/0x100 [lov] [ 8834.784070] [&lt;ffffffffa04c7ddf&gt;] osc_io_end+0x1f/0x30 [osc] [ 8834.784371] [&lt;ffffffffa11dd7a0&gt;] cl_io_end+0x60/0x150 [obdclass] [ 8834.784661] [&lt;ffffffffa059a701&gt;] lov_io_end_wrapper+0xf1/0x100 [lov] [ 8834.785159] [&lt;ffffffffa059a1ae&gt;] lov_io_call+0x8e/0x130 [lov] [ 8834.785465] [&lt;ffffffffa059bdec&gt;] lov_io_end+0x4c/0x110 [lov] [ 8834.785758] [&lt;ffffffffa11dd7a0&gt;] cl_io_end+0x60/0x150 [obdclass] [ 8834.786013] [&lt;ffffffffa11e28d2&gt;] cl_io_loop+0xc2/0x1b0 [obdclass] [ 8834.786353] [&lt;ffffffffa0a7822b&gt;] ll_file_io_generic+0x42b/0x550 [lustre] [ 8834.786644] [&lt;ffffffffa0a7911c&gt;] ll_file_aio_write+0x13c/0x2c0 [lustre] [ 8834.811295] Call Trace: [ 8834.811521] [&lt;ffffffff814f75e4&gt;] ? panic+0xa0/0x168 [ 8834.811788] [&lt;ffffffffa0c9ef7b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [ 8834.812079] [&lt;ffffffffa04ce847&gt;] ? osc_extent_merge+0x697/0x6a0 [osc] [ 8834.812364] [&lt;ffffffff81112df1&gt;] ? generic_file_buffered_write+0x1f1/0x300 [ 8834.812605] [&lt;ffffffff8106fea7&gt;] ? current_fs_time+0x27/0x30 [ 8834.812928] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [ 8834.813301] [&lt;ffffffffa04d601c&gt;] ? osc_extent_release+0xfc/0x3e0 [osc] [ 8834.813615] [&lt;ffffffffa059a610&gt;] ? lov_io_end_wrapper+0x0/0x100 [lov] [ 8834.813903] [&lt;ffffffffa04c7ddf&gt;] ? osc_io_end+0x1f/0x30 [osc] [ 8834.814164] [&lt;ffffffffa11dd7a0&gt;] ? cl_io_end+0x60/0x150 [obdclass] [ 8834.816218] [&lt;ffffffffa059a701&gt;] ? lov_io_end_wrapper+0xf1/0x100 [lov] [ 8834.816513] [&lt;ffffffffa059a1ae&gt;] ? lov_io_call+0x8e/0x130 [lov] [ 8834.816814] [&lt;ffffffffa059bdec&gt;] ? lov_io_end+0x4c/0x110 [lov] [ 8834.817184] [&lt;ffffffffa11dd7a0&gt;] ? cl_io_end+0x60/0x150 [obdclass] [ 8834.817462] [&lt;ffffffffa11e28d2&gt;] ? cl_io_loop+0xc2/0x1b0 [obdclass] [ 8834.817769] [&lt;ffffffffa0a7822b&gt;] ? ll_file_io_generic+0x42b/0x550 [lustre] [ 8834.818078] [&lt;ffffffffa0a7911c&gt;] ? ll_file_aio_write+0x13c/0x2c0 [lustre] [ 8834.818385] [&lt;ffffffffa0a79409&gt;] ? ll_file_write+0x169/0x2a0 [lustre] [ 8834.818676] [&lt;ffffffff8117b2e8&gt;] ? vfs_write+0xb8/0x1a0 [ 8834.818903] [&lt;ffffffff810d5192&gt;] ? audit_syscall_entry+0x272/0x2a0 [ 8834.819185] [&lt;ffffffff8117bbb1&gt;] ? sys_write+0x51/0x90 [ 8834.819425] [&lt;ffffffff8100b0f2&gt;] ? system_call_fastpath+0x16/0x1b </pre> </div></div> <p>I have a crashdump for this one</p> LU-2170

osc_extent_merge()) ASSERTION( cur->oe_osclock == victim->oe_osclock) while running racer

Bug Blocker Resolved Fixed Jinshan Xiong Oleg Drokin Sat, 13 Oct 2012 18:20:09 +0000 Mon, 17 Jul 2017 18:48:40 +0000 Tue, 13 Nov 2012 13:53:56 +0000 Lustre 2.4.0 Lustre 2.4.0 Lustre 2.1.4 0 7 <p>patch is at: <a href="http://review.whamcloud.com/4316" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4316</a></p> <p>The patch has been effective at eliminating our reproducer.</p> <p>BTW, here are the details of the reproducer from our test engineer:</p> <p>fsx_mpi - fsx(file system exerciser) This test is open sourced and part of LTP.</p> <p>iozone - file system benchmark, freely available</p> <p>growfiles_mpi - growfiles extends and truncates a file or files. <br/> growfiles can do data validation. We have several<br/> versions of growfiles(pre/post the Cary/SGI split), but I <br/> think the one used here was the LTP version with the <br/> addition of MPI to use unique file names if desired. </p> <p>doio_mpi - doio_mpi uses the following system calls to do the I/O: <br/> read, write, readv, writev, mmread, mmwrite, fsync2,<br/> fdatasync</p> <p> The program locks the regions of the file that are being<br/> operated on, to prevent overlapping requests from <br/> clobbering each other. This test does data checking. <br/> doio_mpi was based on iogen/doio which are part of LTP, <br/> but it was a large re-write to add the MPI functionality. </p> <p>mmstress - This is a test program that performs general stress with<br/> memory race conditions. It contains seven testcases that<br/> will test race conditions between simultaneous read fault,<br/> write fault, copy on write (COW) fault e.t.c. system and<br/> can be easily ported to work on other operating systems <br/> as well.<br/> Part of LTP. </p> <p>For the commands I was using, 'ubrun' is a tool that can do a little <br/> setup/cleanup, and identifies which binaries to use when running on<br/> different systems. 'aptun' is our application launching tool, it<br/> determines the batch scheduler used if any, creates a job input<br/> file(if needed), then launches the job.</p> <p>I used a tool, FSLOAD, that randomly picks a command from the pool <br/> of commands, and runs them on 5%-10% of the PEs on the system. I<br/> typically had 5-10 jobs running at any time until the problem hit.</p> <p>The commands I was using: <br/> ubrun -t -A "rm ./fsx*" -e LTPROOT_CL -s "All operations completed -OK!" aptrun -n 4 LTPROOT_CL=testcases/bin/fsx-linux_mpi -q -WR -N 30000 ./fsx1.$$</p> <p>ubrun -t -e APPS_CL -s "iozone test complete" aptrun -n 1 -M 1 APPS_CL=iozone/src/current/RUN/iozone -A2 -f ./ioz1.$$</p> <p>ubrun -t -e OSTEST_CL -s "Hit time value" aptrun -n 4 OSTEST_CL=bin/growfiles_mpi -b -e 1 -i 0 -u -L 300 -d .</p> <p>ubrun -t -e LTPROOT_CL -o aptrun -n 4 LTPROOT_CL=testcases/bin/doio_mpi -i 50000 10000b:./do1</p> <p>ubrun -s "Hit time value of" -x -e OSTEST_CL -o -T gf1 -t -D aptrun -n 8 OSTEST_CL=bin/growfiles_mpi -b -i 0 -L 300 -g 4090 -T 100 -t 408990 -C 10 -c 1000 -d dir1 -S 200</p> <p>ubrun <del>s "Hit time value of" -x -e OSTEST_CL -o -T gf5 -t -D aptrun -n 8 OSTEST_CL=bin/growfiles_mpi -b -r 1-5000 -R 0</del>-1 -i 0 -L 300 -C 1 -S 2</p> <p>ubrun -t -D -o -T CL_LTPmtest05 -e LTPROOT_CL -p. -B "cp $LTPROOT_CL/testcases/bin/dummy ." aptrun -n 1 LTPROOT_CL=testcases/bin/mmstress</p> <p>ubrun -t -D -o -T CL_LTPmtest05 -e LTPROOT_CL -p. -B "cp $LTPROOT_CL/testcases/bin/dummy ." aptrun -n 1 LTPROOT_CL=testcases/bin/mmstress</p> <p>Cory, thank you so much about the reproduce info.</p> <p>patch has been landed to master</p> Related Development Rank 1|hzvaaf: Rank (Obsolete) 5202 Severity