https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5188 Lustre <p>mounting mdt we start to see these errors then the system becomes unreponsive</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">nbp6-mds login: LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: Lustre: nbp6-MDT0000: Not available <span class="code-keyword">for</span> connect from 10.153.1.57@o2ib233 (not set up) Lustre: nbp6-MDT0000: used disk, loading Lustre: nbp6-MDT0000: Not available <span class="code-keyword">for</span> connect from 10.153.0.76@o2ib233 (not set up) Lustre: 2967:0:(mdt_handler.c:4969:mdt_process_config()) For interoperability, skip <span class="code-keyword">this</span> mdt.group_upcall. It is obsolete. Lustre: 2967:0:(mdt_handler.c:4969:mdt_process_config()) For interoperability, skip <span class="code-keyword">this</span> mdt.quota_type. It is obsolete. LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST002f-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3069:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST002f-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3069:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 11-0: nbp6-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. Lustre: nbp6-MDT0000: Will be in recovery <span class="code-keyword">for</span> at least 5:00, or until 1083 clients reconnect LustreError: 3069:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 3081:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST0032-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3081:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 212864 previous similar messages LustreError: 3069:0:(osp_sync.c:797:osp_sync_process_queues()) Skipped 229931 previous similar messages LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST002f-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 450953 previous similar messages LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) Skipped 445853 previous similar messages LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST002f-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 930108 previous similar messages LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) Skipped 925668 previous similar messages LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST002f-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 1897185 previous similar messages LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) Skipped 1898934 previous similar messages LustreError: 3081:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST0032-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3069:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 3069:0:(osp_sync.c:797:osp_sync_process_queues()) Skipped 3829999 previous similar messages LustreError: 3081:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 3851279 previous similar messages LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST002f-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 7683067 previous similar messages LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) Skipped 7758692 previous similar messages LustreError: 3081:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST0032-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3069:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 3069:0:(osp_sync.c:797:osp_sync_process_queues()) Skipped 15494392 previous similar messages LustreError: 3081:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 15461799 previous similar messages LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST002f-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) Skipped 31047376 previous similar messages LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 30898154 previous similar messages INFO: task crond:3224 blocked <span class="code-keyword">for</span> more than 120 seconds. <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. crond D 0000000000000003 0 3224 2649 0x00000080 ffff8805c5579d38 0000000000000086 ffff8805c5579d00 ffff8805c5579cfc 0000000000000000 ffff88063fc24700 ffff880028296780 0000000000000500 ffff880625ed85f8 ffff8805c5579fd8 000000000000fc40 ffff880625ed85f8 Call Trace: [&lt;ffffffff81540545&gt;] schedule_timeout+0x215/0x2e0 [&lt;ffffffff81346555&gt;] ? extract_entropy+0xe5/0x140 [&lt;ffffffff815401c3&gt;] wait_for_common+0x123/0x180 [&lt;ffffffff81063be0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffff815402dd&gt;] wait_for_completion+0x1d/0x20 [&lt;ffffffff810921c8&gt;] synchronize_sched+0x58/0x60 [&lt;ffffffff81092150&gt;] ? wakeme_after_rcu+0x0/0x20 [&lt;ffffffff8122205c&gt;] install_session_keyring_to_cred+0x6c/0xd0 [&lt;ffffffff812221f3&gt;] join_session_keyring+0x133/0x160 [&lt;ffffffff810dbff7&gt;] ? audit_syscall_entry+0x1d7/0x200 [&lt;ffffffff81220df8&gt;] keyctl_join_session_keyring+0x38/0x70 [&lt;ffffffff81221a20&gt;] sys_keyctl+0x170/0x190 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) can't send: -22 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp6-OST002f-osc-MDT0000: invalid setattr record, lsr_valid:0 LustreError: 3069:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 61763336 previous similar messages LustreError: 3081:0:(osp_sync.c:797:osp_sync_process_queues()) Skipped 62092386 previous similar messages </pre> </div></div> LU-5188

nbp6-OST002f-osc-MDT0000: invalid setattr record, lsr_valid:0

Bug Blocker Resolved Fixed Niu Yawei Mahmoud Hanafi HB Thu, 12 Jun 2014 22:27:26 +0000 Tue, 3 Mar 2015 15:53:45 +0000 Fri, 27 Jun 2014 17:34:18 +0000 Lustre 2.4.3 Lustre 2.6.0 0 8 <p>Oleg is looking into this</p> <p>Hm, it appears that this is a bug in <a href="https://jira.whamcloud.com/browse/LU-4345" title="failed to update accounting ZAP for user" class="issue-link" data-issue-key="LU-4345"><del>LU-4345</del></a> patch that you have applied.</p> <p>Patch is at <a href="http://review.whamcloud.com/10706" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10706</a> - this should help your immediate problem.</p> <p>Niu: regarding the <a href="https://jira.whamcloud.com/browse/LU-4345" title="failed to update accounting ZAP for user" class="issue-link" data-issue-key="LU-4345"><del>LU-4345</del></a> - it appears that while there is now handling for this case instead of LASSERT, it's pointless because we enter eternal retrying loop so possibly we want to have some more handling there.</p> <p>ok thanks will build and install the patch.</p> <p>NIu</p> <p>Could you please complete any follow up work on this ticket?</p> <p>Thanks</p> <p>Peter</p> <blockquote> <p>Niu: regarding the <a href="https://jira.whamcloud.com/browse/LU-4345" title="failed to update accounting ZAP for user" class="issue-link" data-issue-key="LU-4345"><del>LU-4345</del></a> - it appears that while there is now handling for this case instead of LASSERT, it's pointless because we enter eternal retrying loop so possibly we want to have some more handling there.</p></blockquote> <p>Right, I think we'd return 0 instead of -EINVAL, so that we just print an error message, and osp sync process can continue processing on other records.</p> <p>I think skipping the invalid record is OK, since LFSCK 2 should fix up the ownership of the OST objects on its next scan.</p> <p>The patched fixed our issue.</p> <p>Patch landed to master for 2.6.0.</p> <p>This patch cause <a href="https://jira.whamcloud.com/browse/LU-5244" title="conf-sanity test_32b: osp_sync_thread()) ASSERTION( count &lt; 10 ) " class="issue-link" data-issue-key="LU-5244"><del>LU-5244</del></a>. </p> <p>It seems osp_sync_new_setattr_job return 0 but did not issue RPC, so we should decrease opd_syn_rpc_in_flight and opd_sync_rpc_in_progress in this case.</p> <p><a href="http://review.whamcloud.com/10828" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10828</a></p> <p>Patch landed to Master.</p> Related LU-5249 LU-6269 LU-4345 LU-5244 Development Rank 1|hzwocn: Rank (Obsolete) 14392 Severity