https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3392 Lustre <p>We have experienced an LBUG today on one of our OSS servers which I have not seen before.</p> <p>LustreError: 16066:0:(filter_io_26.c:344:filter_do_bio()) ASSERTION(rw == OBD_BRW_READ) failed<br/> LustreError: 16066:0:(filter_io_26.c:344:filter_do_bio()) LBUG<br/> Pid: 16066, comm: ll_ost_io_126</p> <p>Now after rebooting that OSS same LBUG is triggered as soon as OSTs finish recovery and start servicing their data. Has anyone seen this before ?</p> <p>Our environment:<br/> servers: RHEL6 2.6.32-220.17.1.el6_lustre.x86_64 Lustre-2.1.2<br/> clients: RHEL6 2.6.32-358.6.2.el6.x86_64 Lustre-2.1.5 patchless</p> servers: RHEL6 2.6.32-220.17.1.el6_lustre.x86_64 Lustre-2.1.2<br/> clients: RHEL6 2.6.32-358.6.2.el6.x86_64 Lustre-2.1.5 patchless LU-3392

filter_do_bio()) ASSERTION(rw == OBD_BRW_READ)

Bug Major Resolved Incomplete WC Triage Wojciech Turek Fri, 24 May 2013 10:39:39 +0000 Fri, 3 Feb 2017 05:59:58 +0000 Mon, 28 Sep 2015 17:29:53 +0000 Lustre 2.1.2 1 6 <p>The problem seem to be caused by one particular OST. I identify that OST by unmounting OSTs and I have run fsck on that OST and it found some errors. After fixing them I was able to mount OST and and LBUG was not reoccurring. I can not explain how the corruption of the inode size has crept in in the first place which is concerning. </p> <p>fsck from util-linux-ng 2.17.2<br/> e2fsck 1.42.7.wc1 (12-Apr-2013)<br/> lustre1-OST0003: recovering journal<br/> Pass 1: Checking inodes, blocks, and sizes<br/> Inode 9289789, i_size is 17592186044416, should be 17592184995840. Fix&lt;y&gt;? yes<br/> Pass 2: Checking directory structure<br/> Pass 3: Checking directory connectivity<br/> Pass 4: Checking reference counts<br/> Pass 5: Checking group summary information<br/> Free blocks count wrong (447184151, counted=447797885).<br/> Fix&lt;y&gt;? yes<br/> Free inodes count wrong (16879337, counted=16879523).<br/> Fix&lt;y&gt;? yes</p> <p>lustre1-OST0003: ***** FILE SYSTEM WAS MODIFIED *****</p> <p> 6008797 inodes used (26.25%, out of 22888320)<br/> 455104 non-contiguous files (0.4%)<br/> 32 non-contiguous directories (0.0%)</p> <ol> <li>of inodes with ind/dind/tind blocks: 0/0/0<br/> Extent depth histogram: 5919318/89344/127<br/> 5411611248 blocks used (92.36%, out of 5859409133)<br/> 0 bad blocks<br/> 1726 large files</li> </ol> <p> 6008751 regular files<br/> 37 directories<br/> 0 character device files<br/> 0 block device files<br/> 0 fifos<br/> 0 links<br/> 0 symbolic links (0 fast symbolic links)<br/> 0 sockets<br/> ------------<br/> 6008788 files</p> <p>This LBUG has hit us again, and I found that fsck alone does not actually fix it but aborting recovery does. So after being hit by this lbug one needs to restart OSS and then fsck the OST to fix the i_size error then mount the OST with abort recovery option. If we did not abort recovery LBUG hits again and the i_size is corrupted again. We found that after recovering filesystem one client would not come back. This is most likely the client that creates the problem in the first place. It sounds like a serious bug as it seem that a client operation can bring down the server. Next step for us is to update the server side to 2.1.5 and see if we still see this problem.</p> <p>Marking this as resolved/incomplete.</p> <p>If this is still a live issue on newer release, just let us know and we'll move the ticket to the correct Project.</p> <p>Thanks,<br/> ~ jfc.</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/23931" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23931</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3392" title="filter_do_bio()) ASSERTION(rw == OBD_BRW_READ)" class="issue-link" data-issue-key="LU-3392"><del>LU-3392</del></a> obdfilter: handle large file writting gracefully<br/> Project: fs/lustre-release<br/> Branch: b2_1<br/> Current Patch Set: 1<br/> Commit: d03ea264ec0aa273e3e91ef81070a66adeced965</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/23938" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23938</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3392" title="filter_do_bio()) ASSERTION(rw == OBD_BRW_READ)" class="issue-link" data-issue-key="LU-3392"><del>LU-3392</del></a> osd-ldiskfs: handle large file writting gracefully<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 622fbed8fd4203487c78ec24bdda2bbd0a9ded07</p> Duplicate Development Rank 1|hzvrs7: Rank (Obsolete) 8396 Severity