https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1475 Lustre <p>While using mdsrate to create large number files to use up the free inodes on a 48TB OST on Juelich test cluster, the following EIO error occurred:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>+ su mpiuser sh -c "/usr/lib64/openmpi/bin/mpirun -mca orte_rsh_agent rsh:ssh -np 25 -machinefile /tmp/large-lun.machines /usr/lib64/lustre/tests/mdsrate --create --verbose --ndirs 25 --dirfmt '/mnt/lustre/mdsrate/dir%d' --nfiles 1993129 --filefmt 'file%%d' " &lt;~snip~&gt; Rank 2: 71.24 creates/sec 140.38 secs (total: 260000 creates 3334.84 secs) Rank 21: 71.24 creates/sec 140.38 secs (total: 260000 creates 3334.84 secs) rank 4: open(file264701) error: Input/output error </pre> </div></div> <p>Dmesg on the MDS node showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 4988:0:(lov_request.c:572:lov_update_create_set()) error creating fid 0xf99b sub-object on OST idx 0/1: rc = -5 </pre> </div></div> <p>Maloo report:<br/> <a href="https://maloo.whamcloud.com/test_sets/379a475c-aee9-11e1-b0f7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/379a475c-aee9-11e1-b0f7-52540035b04c</a></p> <p>The same issue also occurred on 1TB OST: <a href="https://maloo.whamcloud.com/test_sets/f900e0aa-aee8-11e1-b0f7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f900e0aa-aee8-11e1-b0f7-52540035b04c</a></p> <br/> Lustre Tag: v2_1_2_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/86/">http://build.whamcloud.com/job/lustre-b2_1/86/</a><br/> Distro/Arch: RHEL6.2/x86_64<br/> Network: IB (in-kernel OFED)<br/> LU-1475

lov_update_create_set() error creating fid 0xf99b sub-object on OST idx 0/1: rc = -5

Bug Minor Resolved Cannot Reproduce Oleg Drokin Jian Yu Tue, 5 Jun 2012 05:41:42 +0000 Mon, 29 May 2017 03:56:41 +0000 Mon, 29 May 2017 03:56:41 +0000 Lustre 2.3.0 Lustre 2.1.2 Lustre 2.1.3 0 8 <p>Oleg could you please look into this one? Thanks!</p> <p>From <a href="http://wiki.whamcloud.com/display/ENG/Lustre+2.1.1+release+testing+tracker" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.whamcloud.com/display/ENG/Lustre+2.1.1+release+testing+tracker</a>, I found the large number inodes testing passed on Lustre 2.1.1 RC1 against 128TB OST on DDN SFA10KE storage appliance:<br/> <a href="https://maloo.whamcloud.com/test_sets/22872264-aef7-11e1-b0f7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/22872264-aef7-11e1-b0f7-52540035b04c</a></p> <p>The test was run on one single node at that time.</p> <p>Now, on Juelich cluster with separated Lustre Client, MDS and OSS nodes, I performed the same test on Lustre 2.1.1 RC4, the EIO error also occurred while testing against 1TB OST:<br/> <a href="https://maloo.whamcloud.com/test_sets/bbce36f6-aee8-11e1-b0f7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bbce36f6-aee8-11e1-b0f7-52540035b04c</a> </p> <p>from the logs MDS has a strong belief it got disconnected from OSTs, even though there are no messages supportign this state (evictions or whatever).</p> <p>I imagine a higher debug log might shed some light onto the situation, though I am not sure how high youcan reasonably go.</p> <p>Ideally if you increase debug_mb to 1024 and debug level to -1 that probably would be best.</p> <blockquote><p>Ideally if you increase debug_mb to 1024 and debug level to -1 that probably would be best.</p></blockquote> <p>OK.<br/> It seems there is some hardware issue on the MDS node n002. It's disconnected. I've been trying to powercycle it and even re-provision it, it still does not start up. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I'm provisioning another MDS node n001... </p> <p>Both n001 and n002 did not work. Finally, I provisioned n003 and booted it up. The testing is ongoing now.</p> <p>Hi Oleg,<br/> I reproduced the failure and gathered the debug logs with -1 level. Please find /scratch/logs/2.1.2/large-lun-145447.tar.bz2 on brent node. Thanks!</p> <p>Thanks!<br/> This log is pretty useful.</p> <p>I think I see the problem now, introduced by WangDi's patch for bug 21379, change eb7c28ff977f4e0a280558aa74e23f2a9ab0ea0c</p> <p>in lov_check_and_create_object() we start with rc = -EIO for the early exit case of no available OSCs,<br/> then we are trying to see if there's some objects available anywhere:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>+ /* check if objects has been created on this ost */ + for (stripe = 0; stripe &lt; lsm-&gt;lsm_stripe_count; stripe++) { + /* already have object at this stripe */ + if (ost_idx == lsm-&gt;lsm_oinfo[stripe]-&gt;loi_ost_idx) + break; + } </pre> </div></div> <p>If we did find something (result of a racing create completion that we can clearly see in the logs), then the next if statement is not triggered:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>+ /* check if objects has been created on this ost */ + for (stripe = 0; stripe &lt; lsm-&gt;lsm_stripe_count; stripe++) { + /* already have object at this stripe */ + if (ost_idx == lsm-&gt;lsm_oinfo[stripe]-&gt;loi_ost_idx) + break; + } </pre> </div></div> <p>And as such we exit via following RETURN(rc) with the rc still being initial -EIO.</p> <p>I think the fix should just be to either add "else rc = 0" to the last if statement, or to add rc = 0 before the last if statement.</p> <p>Wangdi, what do you think?</p> <p>Lustre Tag: v2_1_2_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/87/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/87/</a><br/> Distro/Arch: RHEL6.2/x86_64(server), RHEL6.2/i686(client)<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>sanity test 220 failed with the same issue: <a href="https://maloo.whamcloud.com/test_sets/b0aeb296-b07d-11e1-99c6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b0aeb296-b07d-11e1-99c6-52540035b04c</a></p> <p>I found this failure occurred before and was marked as <a href="https://jira.whamcloud.com/browse/LU-782" title="sanity.sh subtest test_220 failed with 3, error -107 -ENOTCONN" class="issue-link" data-issue-key="LU-782"><del>LU-782</del></a>.</p> <blockquote> <p>Wangdi, what do you think?</p></blockquote> <p>Yes, it should reset EIO to 0 if it found some objects on the stripe.</p> <p>patch at <a href="http://review.whamcloud.com/3058" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3058</a></p> <p>I uploaded debug logs in <a href="https://jira.whamcloud.com/browse/LU-782" title="sanity.sh subtest test_220 failed with 3, error -107 -ENOTCONN" class="issue-link" data-issue-key="LU-782"><del>LU-782</del></a> with similar issue, except the return code is ENOTCONN, but the real current issue is a mix of ENOTCONN and EIO.</p> <p>What is the status of the patch ? I see there was a regression introduced with the current version, but can you see why ?</p> <p>Hm, I just dug some more int logs as part of trying to improve my original patch and came to a conclusion that my initial analysis was wrong (and the "fix" is wrong too).</p> <p>What really happens (in the log file I have at hand):<br/> OST starts to feel real bad (logs have multiple messages that I interpret as i/o errors like</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">sd 6:0:15:0: attempting task abort! scmd(ffff88060e198d80) sd 6:0:15:0: [sdo] CDB: Write(10): 2a 00 01 cb 62 c0 00 00 08 00 scsi target6:0:15: handle(0x001d), sas_address(0x5000c50040cf2bc1), phy(18) scsi target6:0:15: enclosure_logical_id(0x500304800137caff), slot(6) sd 6:0:15:0: task abort: SUCCESS scmd(ffff88060e198d80) </pre> </div></div> <p>)<br/> A precreate request is sent as we start to deplete our precreated objects.<br/> This precreate requests gets stuck on the OST somewhere. The OST knows it's slow so it sends periodic early replies to let us know it's stil working on the request.<br/> Eventually we exhaust all the precreated objects. A fake request is created for the purposes of waiting for some more objects to appear, and it is placed in ptlrpcd queue.</p> <p>As no more objects are created (see OST in trouble).<br/> 10 seconds later ptlrpc decides that the fake request is sittign in the queue for way too long and should be timed out,<br/> this sets off the entire error stuff eventually propagated to userspace (because we don't have any other osts to allocate from).</p> <p>Sebasten: The LU782 log excerpt you posted there has the same sign of fake request timing out (o-1), so likely something very similar is ongoing in that case.</p> <p>Anyway for this bug we need to figure out how come OST is unhappy.<br/> YuJian, is this still repeatable? On other nodes too? Every time with scsi complaining about errors?</p> <p>Oleg,</p> <p>Looking back to the logs on the OSS when the problem first appeared, I don't see any obvious problems. However, the problem was raised after a maintenance where a writeconf was done on all the targets, and remounted all in parallel (around 200 targets at once).<br/> One of our expert suspected that there had been something wrong at re-registration time and that the configs were somehow corrupted.<br/> Reapplying the writeconf and re-registering each target one by one made the problem disappear. This is still difficult to understand why ....</p> <p>Yu Jian,<br/> Can you let us know if this is still happening often? We are trying to determine if this should still be a blocker for 2.3 Release.</p> <p>Hi Oleg and Jodi,</p> <blockquote><p>YuJian, is this still repeatable? On other nodes too? Every time with scsi complaining about errors?</p></blockquote> <blockquote><p>Can you let us know if this is still happening often? We are trying to determine if this should still be a blocker for 2.3 Release.</p></blockquote> <p>I've not run mdsrate on b2_1 branch since Lustre 2.1.2 is released. I'll find out some Toro test nodes to see whether I can reproduce it on the latest b2_1 and master branches or not.</p> <p>However, I found <a href="https://jira.whamcloud.com/browse/LU-1454" title="1.8.8-wc1&lt;-&gt;2.1.2 interop: Test failure on test suite conf-sanity, subtest test_22" class="issue-link" data-issue-key="LU-1454"><del>LU-1454</del></a> (which is a duplicate of this ticket) consistently occurred in autotest runs:<br/> <a href="http://tinyurl.com/crz74xd" class="external-link" target="_blank" rel="nofollow noopener">http://tinyurl.com/crz74xd</a></p> <p>So lu1454 was caused by the same poblem, but it's a faulty test and default behavior.</p> <p>What happens there is we have MDT that has just started and then we restart the (only) OST.<br/> Next we connect the client and try to touch a file.<br/> MDT does not yet know that OST restarted and it happens to not have any precreated objects for this OST. Attempt to get some results in finding out that OST was restarted and we no longer have a connection to it. o other OSTs to try, so we return IO error.</p> <p>For Bull reported failures the situation is less clear, on the other hand we had multiple complaints from other sites that registering many targets at the same time on MGS (many = more than 30 or so at a time) results in strange problems, mgs creashes and such. This warrants its own bug, I guess.</p> <p>RHEL6.3/x86_64 (2.1.3 Server + 1.8.8-wc1 Client):<br/> <a href="https://maloo.whamcloud.com/test_sets/bc2dffb8-e427-11e1-b6d3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bc2dffb8-e427-11e1-b6d3-52540035b04c</a></p> <p>Another instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/d98970d4-eabb-11e1-b137-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d98970d4-eabb-11e1-b137-52540035b04c</a></p> <p>RHEL6.3/x86_64 (2.3.0 RC3 Server + 1.8.8-wc1 Client):<br/> <a href="https://maloo.whamcloud.com/test_sets/e309f73e-169c-11e2-962d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e309f73e-169c-11e2-962d-52540035b04c</a></p> <p>Close old ticket.</p> Duplicate LU-1454 Development Rank 1|hzvskn: Rank (Obsolete) 8533 Severity