https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1484 Lustre <p>This issue was created by maloo for yujian &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/743bea58-af48-11e1-a585-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/743bea58-af48-11e1-a585-52540035b04c</a>.</p> <p>The sub-test test_57 failed with the following error:</p> <blockquote> <p>== recovery-small test 57: read procfs entries causes kernel crash =================================== 05:43:48 (1338900228)<br/> fail_loc=0x80000B00<br/> Stopping client client-28vm6.lab.whamcloud.com /mnt/lustre (opts<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: recovery-small 57</p> <br/> Lustre Tag: v2_1_2_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/87/">http://build.whamcloud.com/job/lustre-b2_1/87/</a><br/> Distro/Arch: RHEL5.8/x86_64<br/> Network: TCP (1GigE)<br/> LU-1484

Test failure on test suite recovery-small, subtest test_57

Bug Blocker Resolved Fixed Nathaniel Clark Maloo Tue, 5 Jun 2012 23:40:02 +0000 Thu, 18 Apr 2013 18:14:29 +0000 Thu, 21 Feb 2013 15:04:05 +0000 Lustre 2.3.0 Lustre 2.1.2 Lustre 2.1.3 Lustre 2.1.4 Lustre 1.8.8 Lustre 2.4.0 Lustre 2.1.5 Lustre 1.8.9 0 14 <p>Console log on Client 4 (client-28vm6) showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>05:43:48:Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash =================================== 05:43:48 (1338900228) 05:43:53:LustreError: 28843:0:(fail.c:126:__cfs_fail_timeout_set()) cfs_fail_timeout id b00 sleeping for 10000ms 05:44:01:LustreError: 28843:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id b00 awake 05:46:17:INFO: task lctl:28843 blocked for more than 120 seconds. 05:46:17:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 05:46:18:lctl D 0000000000001000 0 28843 28631 (NOTLB) 05:46:18: ffff8100517dfe38 0000000000000086 ffffffff800cfa4c ffff810037d56d40 05:46:18: 0000000000000282 0000000000000007 ffff810067000080 ffff8100668c47e0 05:46:18: 000005e137f5c0e1 000000000001b643 ffff810067000268 0000000000000001 05:46:18:Call Trace: 05:46:18: [&lt;ffffffff800cfa4c&gt;] zone_statistics+0x3e/0x6d 05:46:18: [&lt;ffffffff8000f40b&gt;] __alloc_pages+0x78/0x308 05:46:19: [&lt;ffffffff8006468c&gt;] __down_read+0x7a/0x92 05:46:19: [&lt;ffffffff888d90e2&gt;] :obdclass:lprocfs_fops_read+0x82/0x1e0 05:46:19: [&lt;ffffffff8010ab77&gt;] proc_reg_read+0x7e/0x99 05:46:19: [&lt;ffffffff8000b721&gt;] vfs_read+0xcb/0x171 05:46:19: [&lt;ffffffff80011d15&gt;] sys_read+0x45/0x6e 05:46:19: [&lt;ffffffff8005d28d&gt;] tracesys+0xd5/0xe0 05:46:19: 05:46:19:INFO: task umount:28863 blocked for more than 120 seconds. 05:46:22:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 05:46:24:umount D ffff810002536420 0 28863 28862 (NOTLB) 05:46:24: ffff810057f81828 0000000000000082 0000000000000000 0000000100000002 05:46:24: 0000000000000000 0000000000000007 ffff8100668c47e0 ffffffff80319b60 05:46:24: 000005e137f5e08c 0000000000001fab ffff8100668c49c8 0000000000000000 05:46:24:Call Trace: 05:46:24: [&lt;ffffffff80064cb5&gt;] __reacquire_kernel_lock+0x2e/0x47 05:46:24: [&lt;ffffffff80063171&gt;] wait_for_completion+0x79/0xa2 05:46:24: [&lt;ffffffff8008ee74&gt;] default_wake_function+0x0/0xe 05:46:24: [&lt;ffffffff8010dcdb&gt;] remove_proc_entry+0xfb/0x1c7 05:46:25: [&lt;ffffffff888d7603&gt;] :obdclass:lprocfs_remove+0x103/0x130 05:46:25: [&lt;ffffffff888d6a46&gt;] :obdclass:lprocfs_free_stats+0x1e6/0x230 05:46:25: [&lt;ffffffff888d7a1f&gt;] :obdclass:lprocfs_obd_cleanup+0x6f/0x80 05:46:28: [&lt;ffffffff88b7ca32&gt;] :osc:osc_precleanup+0x292/0x370 05:46:28: [&lt;ffffffff888ff13c&gt;] :obdclass:lu_context_fini+0x1c/0x50 05:46:28: [&lt;ffffffff888e303f&gt;] :obdclass:class_cleanup+0xc6f/0xe30 05:46:28: [&lt;ffffffff888e6d8c&gt;] :obdclass:class_process_config+0x1e5c/0x3200 05:46:28: [&lt;ffffffff888e97f7&gt;] :obdclass:class_manual_cleanup+0xad7/0xe80 05:46:28: [&lt;ffffffff8002ff6f&gt;] __up_write+0x27/0xf2 05:46:30: [&lt;ffffffff88bba26c&gt;] :lov:lov_putref+0xb0c/0xb90 05:46:30: [&lt;ffffffff88bc2b98&gt;] :lov:lov_disconnect+0x308/0x3e0 05:46:30: [&lt;ffffffff88c66d94&gt;] :lustre:client_common_put_super+0x894/0xed0 05:46:30: [&lt;ffffffff88c676e5&gt;] :lustre:ll_put_super+0x195/0x310 05:46:30: [&lt;ffffffff800f079e&gt;] invalidate_inodes+0xce/0xe0 05:46:30: [&lt;ffffffff800e77ab&gt;] generic_shutdown_super+0x79/0xfb 05:46:30: [&lt;ffffffff800e787b&gt;] kill_anon_super+0x9/0x35 05:46:30: [&lt;ffffffff800e792c&gt;] deactivate_super+0x6a/0x82 05:46:31: [&lt;ffffffff800f1e8b&gt;] sys_umount+0x245/0x27b 05:46:31: [&lt;ffffffff800ba767&gt;] audit_syscall_entry+0x1a8/0x1d3 05:46:31: [&lt;ffffffff8005d28d&gt;] tracesys+0xd5/0xe0 </pre> </div></div> <p>For Lustre 2.1.1, we also hit recovery-small test 57 hanging, but it's another failure on client: <a href="https://jira.whamcloud.com/browse/LU-1097" title="Oops: EIP is at osc_rd_lockless_truncate+0xd/0x30 [osc]" class="issue-link" data-issue-key="LU-1097"><del>LU-1097</del></a>. </p> <p>Another instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/18995f38-aafb-11e1-b191-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/18995f38-aafb-11e1-b191-52540035b04c</a></p> <p>And here are the historical reports with status "TIMEOUT" on Maloo:<br/> <a href="http://tinyurl.com/cym3do3" class="external-link" target="_blank" rel="nofollow noopener">http://tinyurl.com/cym3do3</a></p> <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>there's a deadlock here: (!HAVE_PROCFS_USER &amp;&amp; HAVE_PROCFS_DELETED)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>proc reader proc remover proc_reg_read() LPROCFS_WRITE_ENTRY() // down_write _lprocfs_lock semaphore -------&gt; (1) pdeaux-&gt;pde_users++ lprocfs_fops_read() LPROCFS_ENTRY_AND_CHECK() // down_read _lprocfs_lock semaphore, wait here ---------------------------&gt; (2) remove_proc_entry() if (pdeaux-&gt;pde_users &gt; 0) wait_for_completion() ------------------------------------&gt; (3) ... pde_users_dec() // pdeaux-&gt;pde_users--, complete() ---------------------------------------------------------- (4) </pre> </div></div> <p>the issue is when remover get to (3), the proc reader will wait at (2), while proc remover cannot move on until proc reader reachs (4), a deadlock ensues.</p> <p>patch tracking at <a href="http://review.whamcloud.com/3455" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3455</a></p> <p>lprocfs: fix a deadlock</p> <p>There is a deadlock between proc reader and proc remover.</p> <p> <b>proc reader</b> <b>proc remover</b><br/> LPROCFS_WRITE_ENTRY() -----&gt; (1)<br/> proc_reg_read()<br/> pdeaux-&gt;pde_users++<br/> lprocfs_fops_read()<br/> LPROCFS_ENTRY_AND_CHECK() // wait semaphore &lt;----- (2)<br/> remove_proc_entry()<br/> if (pdeaux-&gt;pde_users &gt; 0)<br/> wait_for_completion() -&gt; (3)<br/> ...<br/> pde_users_dec() // pdeaux-&gt;pde_users-<del>, complete() &lt;</del>---- (4)</p> <p>when remover gets to (3), the proc reader will wait at (2), while<br/> proc remover cannot move on until proc reader reaches (4), a deadlock<br/> ensues.</p> <p>update patch</p> <p>lprocfs: refine LC_PROCFS_USERS check</p> <p>In some RHEL patched 2.6.18 kernels, pde_users member is added in<br/> another struct proc_dir_entry_aux instead of in struct proc_dir_entry<br/> in later kernel version of 2.6.23.</p> <p>Landed for 2.3</p> <p>Lustre Tag: v2_1_3_RC1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/113/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/113/</a><br/> Distro/Arch: RHEL5.8/x86_64 (kernel version: 2.6.18-308.11.1.el5)<br/> Network: TCP (1GigE)</p> <p>The same issue exists in Lustre 2.1.3: <a href="https://maloo.whamcloud.com/test_sets/89731b4c-e415-11e1-b6d3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/89731b4c-e415-11e1-b6d3-52540035b04c</a></p> <p>Will the patch for this ticket be cherry-picked/ported to b2_1 branch?</p> <p>b2_1 patch tracking at <a href="http://review.whamcloud.com/3471" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3471</a></p> <p>With commit 76bf16d1e12cd3c2d2f48a31e3e6c1ad66523638 (<a href="https://jira.whamcloud.com/browse/LU-1484" title="Test failure on test suite recovery-small, subtest test_57" class="issue-link" data-issue-key="LU-1484"><del>LU-1484</del></a> lprocfs: refine LC_PROCFS_USERS check) for the ticket, I got following build errors with RHEL 2.6.18-274.12.1.el5.i686 kernel.</p> <p> CC <span class="error">&#91;M&#93;</span> /home/bergwolf/src/lustre-testing/libcfs/libcfs/linux/linux-tracefile.o<br/> In file included from /home/bergwolf/src/lustre-testing/libcfs/include/libcfs/libcfs.h:322,<br/> from /home/bergwolf/src/lustre-testing/libcfs/libcfs/linux/linux-tracefile.c:38:<br/> /home/bergwolf/src/lustre-testing/libcfs/include/libcfs/params_tree.h:107:2: error: #error proc_dir_entry-&gt;deleted is conflicted with proc_dir_entry-&gt;pde_users<br/> make<span class="error">&#91;6&#93;</span>: *** <span class="error">&#91;/home/bergwolf/src/lustre-testing/libcfs/libcfs/linux/linux-tracefile.o&#93;</span> Error 1<br/> make<span class="error">&#91;5&#93;</span>: *** <span class="error">&#91;/home/bergwolf/src/lustre-testing/libcfs/libcfs&#93;</span> Error 2<br/> make<span class="error">&#91;4&#93;</span>: *** <span class="error">&#91;/home/bergwolf/src/lustre-testing/libcfs&#93;</span> Error 2</p> <p>config.h shows that both HAVE_PROCFS_DELETED and HAVE_PROCFS_USERS are defined.</p> <p>460 /* kernel has deleted member in procfs entry struct */<br/> 461 #define HAVE_PROCFS_DELETED 1<br/> 462<br/> 463 /* kernel has pde_users member in proc_dir_entry_aux */<br/> 464 #define HAVE_PROCFS_USERS 1</p> <p>Building b2_1, I have verified that the RHEL 5.8 updates 2.6.18-308.11.1.el5 and 2.6.18-308.16.1.el5 include both proc_dir_entry.deleted and proc_dir_entry_aux.pde_users. As Peng reported, this leads to</p> <p> libcfs/include/libcfs/params_tree.h:107:2: error: #error proc_dir_entry-&gt;deleted is conflicted with proc_dir_entry-&gt;pde_user</p> <p>anytime the source for one of these RHEL 5.8 updates is used for --with-linux=.</p> <p>Follow up in <br/> <a href="https://jira.whamcloud.com/browse/LU-2334" title="build errors with 2.6.18 kernel" class="issue-link" data-issue-key="LU-2334"><del>LU-2334</del></a> build errors with 2.6.18 kernel</p> <p>Lustre Branch: b2_1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/148" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/148</a><br/> Distro/Arch: RHEL5.8/x86_64 (kernel version: 2.6.18-308.20.1.el5)</p> <p>The same issue occurred again on recovery-small test 57:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== recovery-small test 57: read procfs entries causes kernel crash =================================== 19:13:57 (1354936437) fail_loc=0x80000B00 CMD: fat-intel-3vm6.lab.whamcloud.com grep -c /mnt/lustre' ' /proc/mounts Stopping client fat-intel-3vm6.lab.whamcloud.com /mnt/lustre (opts:) CMD: fat-intel-3vm6.lab.whamcloud.com lsof -t /mnt/lustre CMD: fat-intel-3vm6.lab.whamcloud.com umount /mnt/lustre 2&gt;&amp;1 </pre> </div></div> <p>Console log on client (fat-intel-3vm6):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>19:14:02:Lustre: DEBUG MARKER: == recovery-small test 57: read procfs entries causes kernel crash =================================== 19:13:57 (1354936437) 19:14:02:Lustre: DEBUG MARKER: grep -c /mnt/lustre' ' /proc/mounts 19:14:02:LustreError: 9173:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id b00 sleeping for 10000ms 19:14:02:Lustre: DEBUG MARKER: lsof -t /mnt/lustre 19:14:02:Lustre: DEBUG MARKER: umount /mnt/lustre 2&gt;&amp;1 19:14:13:LustreError: 9173:0:(fail.c:137:__cfs_fail_timeout_set()) cfs_fail_timeout id b00 awake 19:16:57:INFO: task lctl:9173 blocked for more than 120 seconds. 19:16:57:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 19:16:57:lctl D 0000000000001000 0 9173 8913 (NOTLB) 19:16:57: ffff810044841e38 0000000000000086 ffffffff800cfa80 ffff810037d56d40 19:16:57: 0000000000000282 0000000000000007 ffff81004de4f860 ffff81005f2120c0 19:16:57: 00002a16f848196c 0000000000020344 ffff81004de4fa48 0000000000000001 19:16:57:Call Trace: 19:16:57: [&lt;ffffffff800cfa80&gt;] zone_statistics+0x3e/0x6d 19:16:58: [&lt;ffffffff8000f47b&gt;] __alloc_pages+0x78/0x308 19:16:58: [&lt;ffffffff8006468c&gt;] __down_read+0x7a/0x92 19:16:58: [&lt;ffffffff889170c2&gt;] :obdclass:lprocfs_fops_read+0x82/0x200 19:16:58: [&lt;ffffffff8010b73e&gt;] proc_reg_read+0x7e/0x99 19:16:58: [&lt;ffffffff8000b72f&gt;] vfs_read+0xcb/0x171 19:16:58: [&lt;ffffffff80011d85&gt;] sys_read+0x45/0x6e 19:16:58: [&lt;ffffffff8005d28d&gt;] tracesys+0xd5/0xe0 19:16:58: 19:16:58:INFO: task umount:9195 blocked for more than 120 seconds. 19:16:58:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 19:16:58:umount D ffff810002536420 0 9195 9194 (NOTLB) 19:16:58: ffff810058613a08 0000000000000086 00000000ffffffff 0000000000000020 19:16:58: 00000000ffffffff 0000000000000007 ffff81005f2120c0 ffffffff8031ab60 19:16:58: 00002a16f8483cb6 000000000000234a ffff81005f2122a8 0000000000000000 19:16:58:Call Trace: 19:16:58: [&lt;ffffffff80064cb5&gt;] __reacquire_kernel_lock+0x2e/0x47 19:16:58: [&lt;ffffffff80063171&gt;] wait_for_completion+0x79/0xa2 19:16:58: [&lt;ffffffff8008ee97&gt;] default_wake_function+0x0/0xe 19:16:58: [&lt;ffffffff8010e8a2&gt;] remove_proc_entry+0xfb/0x1c7 19:16:58: [&lt;ffffffff88915573&gt;] :obdclass:lprocfs_remove+0x103/0x130 19:16:58: [&lt;ffffffff889159d0&gt;] :obdclass:lprocfs_obd_cleanup+0x90/0xa0 19:16:58: [&lt;ffffffff88caf665&gt;] :osc:osc_precleanup+0x2e5/0x3a0 19:16:58: [&lt;ffffffff88920c35&gt;] :obdclass:class_cleanup+0xc55/0xda0 19:16:58: [&lt;ffffffff889241f6&gt;] :obdclass:class_process_config+0x1b46/0x2cc0 19:16:58: [&lt;ffffffff88926a0b&gt;] :obdclass:class_manual_cleanup+0x9bb/0xd70 19:16:58: [&lt;ffffffff88d03c5d&gt;] :lov:lov_putref+0xa7d/0xaf0 19:16:58: [&lt;ffffffff88cfe623&gt;] :lov:lov_del_target+0x6d3/0x720 19:16:58: [&lt;ffffffff88d0b78b&gt;] :lov:lov_disconnect+0x39b/0x440 19:16:58: [&lt;ffffffff88de73ea&gt;] :lustre:client_common_put_super+0x83a/0xe10 19:16:58: [&lt;ffffffff88de7d15&gt;] :lustre:ll_put_super+0x1a5/0x330 19:16:58: [&lt;ffffffff800f120a&gt;] invalidate_inodes+0xce/0xe0 19:16:58: [&lt;ffffffff800e78ae&gt;] generic_shutdown_super+0x79/0xfb 19:16:58: [&lt;ffffffff800e797e&gt;] kill_anon_super+0x9/0x35 19:16:58: [&lt;ffffffff800e7a2f&gt;] deactivate_super+0x6a/0x82 19:16:58: [&lt;ffffffff800f28f7&gt;] sys_umount+0x245/0x27b 19:16:58: [&lt;ffffffff800ba78a&gt;] audit_syscall_entry+0x1a8/0x1d3 19:16:58: [&lt;ffffffff8005d28d&gt;] tracesys+0xd5/0xe0 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/a59d9126-41bc-11e2-a653-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a59d9126-41bc-11e2-a653-52540035b04c</a></p> <p>In Lustre Branch: b2_1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/148" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/148</a><br/> Distro/Arch: RHEL5.8/x86_64 (kernel version: 2.6.18-308.20.1.el5)</p> <p>In rhel5 kernel build log</p> <blockquote><p>checking if kernel has pde_users member in procfs entry struct... no</p></blockquote> <p>this test result does not match the supposed result from 2.6.18-308.20.1.el5 kernel source, my local test shows</p> <blockquote><p>checking if kernel has pde_users member in procfs entry struct... yes</p></blockquote> <p>and build failed as Stephen pointed out:</p> <blockquote><p>/root/work/lustre.clone/libcfs/include/libcfs/params_tree.h:113:2: error: #error proc_dir_entry-&gt;deleted is conflicted with proc_dir_entry-&gt;pde_users</p></blockquote> <p>This means there is something wrong about the RHEL5 kernel build.</p> <p>Also the hung stack also reveals that proc_dir_entry_aux::pre_users is used.</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>remove_proc_entry() in 2.6.18-308.20.1.el5 source</b></div><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* Wait until all existing callers into module are done. */</span> <span class="code-keyword">if</span> (pdeaux-&gt;pde_users &gt; 0) { DECLARE_COMPLETION_ONSTACK(c); pdeaux = to_pde_aux(de); <span class="code-keyword">if</span> (!pdeaux-&gt;pde_unload_completion) pdeaux-&gt;pde_unload_completion = &amp;c; spin_unlock(&amp;pdeaux-&gt;pde_unload_lock); spin_unlock(&amp;proc_subdir_lock); wait_for_completion(pdeaux-&gt;pde_unload_completion); spin_lock(&amp;proc_subdir_lock); <span class="code-keyword">goto</span> continue_removing; } </pre> </div></div> <p>I'll update a patch to make 2.6.18-308.20.1.el5 build pass.</p> <p>b2_1 patch handling build error for 2.6.18-308 RHEL5 kernel is tracked at <a href="http://review.whamcloud.com/4794" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4794</a> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>commit message</b></div><div class="preformattedContent panelContent"> <pre>LU-1484 kernel: pass RHEL5 build for 2.6.18-308 For vanilla kernel, proc_dir_entry::deleted and ::pde_users co-exists from 2.6.23 to 2.6.23.17. For some RHEL5 kernels, it defines co-existings proc_dir_entry::deleted and proc_dir_entry_aux::pde_users. </pre> </div></div> <p>strangely the RHEL5 build in <a href="http://build.whamcloud.com/job/lustre-reviews/11128/arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/11128/arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/</a> still shows that pde_users test failed. </p> <p>Chris Gearing, could you help me check out why RHEL5 (b2_1) build does not detect pde_users members? Thanks.</p> <p>Check items:</p> <ul> <li>it is defined in fs/proc/internal.h, in proc_dir_entry_aux structure.</li> <li>lustre/autoconf/lusre-core.m4 checked this member in LC_PROCFS_USERS</li> <li>after configure-ed, the config.h contains "#define HAVE_PROCFS_DELETED 1"</li> </ul> <p>Landed for 2.1.4. RHEL5 is not supported in 2.4 so this latest change is not needed to master</p> <p>Lustre Tag: v2_1_4_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/164" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/164</a><br/> Distro/Arch: RHEL5.8/x86_64</p> <p>The issue occurred again: <a href="https://maloo.whamcloud.com/test_sets/baaad7ac-4c1d-11e2-875d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/baaad7ac-4c1d-11e2-875d-52540035b04c</a></p> <p>Lustre Branch: b1_8<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/236/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/236/</a><br/> Distro/Arch: RHEL5.8/x86_64<br/> Test Group: failover</p> <p>The same issue occurred: <a href="https://maloo.whamcloud.com/test_sets/e6be996c-51b5-11e2-a904-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e6be996c-51b5-11e2-a904-52540035b04c</a></p> <p>Zhenyu Xu:</p> <p>I'm not sure what you are asking of me, but perhaps you could tell me what this line indicates?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">after configure-ed, the config.h contains <span class="code-quote">"#define HAVE_PROCFS_DELETED 1"</span> </pre> </div></div> <p>which config.h file is this?</p> <ul> <li>pde_users should be defined in fs/proc/internal.h, in proc_dir_entry_aux structure.</li> <li>lustre/autoconf/lusre-core.m4 should has checked this member in LC_PROCFS_USERS</li> <li>after configure-ed, the config.h generated under lustre build root directory should contains "#define HAVE_PROCFS_DELETED 1" and "#define HAVE_PROCFS_USERS 1"</li> </ul> <p>Lustre Branch: b1_8<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/236/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/236/</a><br/> Distro/Arch: RHEL5.8/x86_64</p> <p>The same issue occurred again: <a href="https://maloo.whamcloud.com/test_sets/6ed434e6-57ca-11e2-9cc9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6ed434e6-57ca-11e2-9cc9-52540035b04c</a></p> <p>I presume this is a build issue,</p> <p>If I look at the latest b1_8 head build on server then </p> <p>BUILD/BUILD/lustre-1.8.8.60/config.h</p> <p>does have </p> <p>#define HAVE_PROCFS_DELETED 1</p> <p>but</p> <p>/* kernel has pde_users member in procfs entry struct */<br/> /* #undef HAVE_PROCFS_USERS */</p> <p>I've attached the config file and the config.log</p> <p>I need port relevant patches to b1_8 branch then. it's tracked at <a href="http://review.whamcloud.com/4976" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4976</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>commit message</b></div><div class="preformattedContent panelContent"> <pre> LU-1484 lprocfs: refine LC_PROCFS_USERS check In some RHEL patched 2.6.18 kernels, pde_users member is added in another struct proc_dir_entry_aux instead of in struct proc_dir_entry in later kernel version of 2.6.23. </pre> </div></div> <p>Landed to b1_8</p> <p>Lustre Branch: b1_8<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/248" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/248</a><br/> Distro/Arch: RHEL5.8/x86_64 (kernel: 2.6.18-308.11.1.el5)</p> <p>The issue still occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/2b536b04-623a-11e2-b20c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2b536b04-623a-11e2-b20c-52540035b04c</a></p> <p>Another instance on Lustre build <a href="http://build.whamcloud.com/job/lustre-b1_8/249" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/249</a> :<br/> <a href="https://maloo.whamcloud.com/test_sets/3155439e-6355-11e2-ae8b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3155439e-6355-11e2-ae8b-52540035b04c</a></p> <p>b1_8 still need land another patch <a href="http://review.whamcloud.com/5129" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5129</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>commit message</b></div><div class="preformattedContent panelContent"> <pre>LU-1484 kernel: pass RHEL5 build for 2.6.18-308 For vanilla kernel, proc_dir_entry::deleted and ::pde_users co-exists from 2.6.23 to 2.6.23.17. For some RHEL5 kernels, it defines co-existings proc_dir_entry::deleted and proc_dir_entry_aux::pde_users. </pre> </div></div> <p>Lustre Branch: b1_8<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/252" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/252</a><br/> Distro/Arch: RHEL5.9 (kernel version: 2.6.18-348.1.1.el5)</p> <p>recovery-small test 57 failed again: <a href="https://maloo.whamcloud.com/test_sets/68c48694-6a28-11e2-85d4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/68c48694-6a28-11e2-85d4-52540035b04c</a></p> <p>Chris,</p> <p>Would you mind checking the build system again for the following info of the latest b1_8 build (<a href="http://build.whamcloud.com/job/lustre-b1_8/252)?" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/252)?</a></p> <ul> <li>pde_users should be defined in fs/proc/internal.h, in proc_dir_entry_aux structure.</li> <li>lustre/autoconf/lusre-core.m4 should has checked this member in LC_PROCFS_USERS</li> <li>after configure-ed, the config.h generated under lustre build root directory should contains "#define HAVE_PROCFS_DELETED 1" and "#define HAVE_PROCFS_USERS 1"</li> </ul> <p>Xu,</p> <p>Can you provide me a lot more info please, I really do not know what you are asking me to check.</p> <p>fs/proc/internal.h comes as part of the lustre source? If so how does the build system affect the presence of pde_users, and if not where does fs/proc/internal.h come from.</p> <p>I guess I'm just not understanding how the build system affects these things.</p> <p>What do I need to provide to help you debug this?</p> <p>sorry, fs/proc/internal.h is kernel's file, like linux-2.6.32-xxx/fs/proc/internal.h</p> <p>I'll also check my local 2.6.18-348.1.1.el5 kernel.</p> <p>Chris,</p> <p>This is my local VM test environment test confirm procedure (CentOS 5.9, 2.6.18-348.1.1.el5 kernel), can you confirm them on the build node?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ tail -n 15 linux-2.6.18-308.11.1.el5-b18/fs/proc/internal.h /* * RHEL internal wrapper to extend struct proc_dir_entry */ struct proc_dir_entry_aux { struct proc_dir_entry pde; int pde_users; /* number of callers into module in progress */ spinlock_t pde_unload_lock; /* proc_fops checks and pde_users bumps */ struct completion *pde_unload_completion; char name[]; /* PDE name */ }; static inline struct proc_dir_entry_aux *to_pde_aux(struct proc_dir_entry *d) { return container_of(d, struct proc_dir_entry_aux, pde); } $ ./configure --with-linux=/path-to/linux-2.6.18-308.11.1.el5-b18 ... ... checking if kernel has pde_users member in procfs entry struct... yes ... checking if kernel has deleted member in procfs entry struct... yes ... $ grep PROCFS ~/work/lustre-b18/config.h 440:#define HAVE_PROCFS_DELETED 1 443:#define HAVE_PROCFS_USERS 1 $ONLY=57 bash recovery-small.sh ... ... == test 57: read procfs entries causes kernel crash == 20:41:25 fail_loc=0x80000B00 Stopping client test3 /mnt/lustre (opts:) fail_loc=0x80000B00 Stopping /mnt/mds (opts:) Failover mds to test3 20:41:49 (1359549709) waiting for test3 network 900 secs ... 20:41:49 (1359549709) network interface is UP Starting mds: -o loop -o abort_recovery /tmp/lustre-mdt /mnt/mds lnet.debug=0x33f1504 lnet.subsystem_debug=0xffb7e3ff lnet.debug_mb=32 Started lustre-MDT0000 recovery-small.sh: line 992: kill: (1143) - No such process fail_loc=0 Starting client: test3: -o user_xattr,acl,flock test3@tcp:/lustre /mnt/lustre lnet.debug=0x33f1504 lnet.subsystem_debug=0xffb7e3ff lnet.debug_mb=32 Filesystem 1K-blocks Used Available Use% Mounted on test3@tcp:/lustre 562408 53656 478752 11% /mnt/lustre Resetting fail_loc on all nodes...done. PASS 57 (26s) ...===== recovery-small.sh test complete, duration 28 sec ====================== </pre> </div></div> <p>in client build log "config.log" I found this</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> #include &lt;linux/kernel.h&gt; | | #include "/var/lib/jenkins/workspace/lustre-b1_8/arch/x86_64/build_type/client/distro/el5/ib_stack/inkernel/BUILD/reused/usr/src/kernels/2.6.18-348.1.1.el5-x86_64/fs/proc/internal.h" | | int | main (void) | { | | struct proc_dir_entry_aux pde_aux; | | pde_aux.pde_users = 0; | | ; | return 0; | } configure:14056: result: no </pre> </div></div> <p>And I checked the client build environment Chris copied for debugging</p> <p>bobijam@brent:/scratch/help-bob-jam/client/BUILD/reused/usr/src/kernels/2.6.18-3<br/> 48.1.1.el5-x86_64$ ll fs/proc/<br/> total 12<br/> drwxr-xr-x 2 533 503 4096 Jan 31 09:50 ./<br/> drwxr-xr-x 66 533 503 4096 Jan 31 09:50 ../<br/> <del>rw-r</del><del>r</del>- 1 533 503 378 Jan 31 09:50 Makefile</p> <p>there's no files under fs/proc/, while this RHEL kernel (vanilla kernel + RHEL patches) should have C/H files under fs/proc/</p> <p>Brian, does the build process in this stage only use vanilla kernel (i.e. hasn't applied RHEL patches)? Given that the RHEL kernel src rpm only provides vanilla kernel source plus RHEL's patches, and the patches only get applied when rpmbuild executes its "%prep" stage.</p> <p>Does the client build even use full kernel source at all? It shouldn't need it I don't think, just kernel-devel. Yes, looking in lbuild in build_with_srpm() where $PATCHLESS == true:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> ! kernelrpm=$(find_linux_rpm <span class="code-quote">"-$DEVEL_KERNEL_TYPE"</span>); then fatal 1 <span class="code-quote">"Could not find the kernel-$DEVEL_KERNEL_TYPE RPM in ${KERNELRPMSBASE}/${lnxmaj}/${DISTRO}"</span> fi <span class="code-keyword">if</span> ! lnxrel=<span class="code-quote">"$lnxrel"</span> unpack_linux_devel_rpm <span class="code-quote">"$kernelrpm"</span> <span class="code-quote">"-"</span>; then fatal 1 <span class="code-quote">"Could not find the Linux tree in $kernelrpm"</span> fi </pre> </div></div> <p>and we can see in the build log in:</p> <p><tt><a href="http://build.whamcloud.com/job/lustre-b1_8/252/arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/consoleText" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/252/arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/consoleText</a></tt></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+ kernelrpm=/<span class="code-keyword">var</span>/lib/jenkins/lbuild-data/kernelrpm/2.6.18/rhel5/x86_64/kernel-devel-2.6.18-348.1.1.el5.x86_64.rpm ... + unpack_linux_devel_rpm /<span class="code-keyword">var</span>/lib/jenkins/lbuild-data/kernelrpm/2.6.18/rhel5/x86_64/kernel-devel-2.6.18-348.1.1.el5.x86_64.rpm - </pre> </div></div> <p>So indeed, it's kernel-devel that lustre's configure is pointed at in build_with_srpm()-&gt;build_lustre():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">++ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --target=x86_64-redhat-linux-gnu --program-prefix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/<span class="code-keyword">var</span> --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --with-linux=/<span class="code-keyword">var</span>/lib/jenkins/workspace/lustre-b1_8/arch/x86_64/build_type/client/distro/el5/ib_stack/inkernel/BUILD/reused/usr/src/kernels/2.6.18-348.1.1.el5-x86_64 --with-linux-obj=/<span class="code-keyword">var</span>/lib/jenkins/workspace/lustre-b1_8/arch/x86_64/build_type/client/distro/el5/ib_stack/inkernel/BUILD/reused/usr/src/kernels/2.6.18-348.1.1.el5-x86_64 --disable-server --enable-liblustre --enable-liblustre-tests --with-release=wc1_2.6.18_348.1.1.el5_g3480bb0 --enable-tests --enable-liblustre-tests </pre> </div></div> <p>If you look in kernel-devel you will find that fs/proc/ is empty because kernel-devel is "kernel headers" not full kernel source.</p> <p>Ultimate what this means is that you need to re-cook your test so as to not need kernel source but kernel headers only. This is the standard for external kernel modules: they should be buildable with kernel headers only since kernel headers represent the API and reaching behind the API is "cheating".</p> <p>Andreas,</p> <p>RHEL 5.9 hasn't revealed pre_users in its devel package, I find no other way to detect proc_dir_entry_aux::pde_users, and since the pde_users are all used in later kernels, is it ok to change the lprocfs_status.<span class="error">&#91;c|h&#93;</span> code to assume HAVE_PROCFS_USERS is always defined?</p> <p>I can't find any way to check for proc_dir_entry_aux, so we can't depend on checking it for patchless clients.</p> <p>I think what needs to change here is two things:</p> <ul class="alternate" type="square"> <li>the code in lprocfs_status.h (1.8) and param_tree.h (master) should be changed to check for HAVE_PROCFS_USERS first, then HAVE_PROCFS_DELETED secondly, so that if both are available it uses the HAVE_PROCFS_USERS method</li> <li>always check for pde_fops == NULL, regardless of whether we detect HAVE_PROCFS_USERS</li> <li>always check for deleted, if HAVE_PROCFS_DELETED is set, even if HAVE_PROCFS_USERS is also present</li> </ul> <p>At worst this causes some small race where a /proc entry will not be shown when it is just loaded or unloaded, but should be safe against crashing.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> inline <span class="code-object">int</span> LPROCFS_ENTRY_AND_CHECK(struct proc_dir_entry *dp) { <span class="code-object">int</span> deleted = 0; #ifdef HAVE_PROCFS_USERS spin_lock(&amp;dp-&gt;pde_unload_lock); #endif <span class="code-keyword">if</span> (unlikely(dp-&gt;proc_fops == NULL)) deleted = 1; #ifdef HAVE_PROCFS_USERS spin_unlock(&amp;dp-&gt;pde_unload_lock); #endif LPROCFS_ENTRY(); #<span class="code-keyword">if</span> defined(HAVE_PROCFS_DELETED) <span class="code-keyword">if</span> (unlikely(dp-&gt;deleted)) { LPROCFS_EXIT(); deleted = 1; } #endif <span class="code-keyword">return</span> deleted ? -ENODEV : 0; } </pre> </div></div> <p>I haven't tested this at all, nor even compiled it yet.</p> <p>Patch at <a href="http://review.whamcloud.com/5253" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5253</a>, let's hope it builds and tests OK.</p> <p>Lustre Branch: b1_8<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/253" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/253</a><br/> Distro/Arch: RHEL5.9/x86_64</p> <p>The issue still occurred: <a href="https://maloo.whamcloud.com/test_sets/583b7710-7009-11e2-a955-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/583b7710-7009-11e2-a955-52540035b04c</a></p> <p>Since recovery-small test_57 is intended to test proc removing while reading it, so the patch (review#5253) cannot avoid the hung of the test w/ patchless client build upon the hidden proc_dir_entry users kernels.</p> <p>Since later kernels all use proc_dir_entry users, I think we can presume it and define LPROCFS_</p> {ENTRY,END} <p> empty ops.</p> <p>Patch to assume proc_dir_entry for rhel kernels: <a href="http://review.whamcloud.com/5439" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5439</a></p> <p>Nathaniel,</p> <p>Is this patch needed for b2_1 also?</p> <p>Peter</p> <p>Per <a href="http://wiki.whamcloud.com/display/ENG/Lustre+2.1.4+release+testing+tracker" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.whamcloud.com/display/ENG/Lustre+2.1.4+release+testing+tracker</a>, the issue still exists in Lustre 2.1.4, so we need the patch on the current b2_1 branch for Lustre 2.1.5.</p> <p>Peter,</p> <p>Yes. This patch can cleanly apply to b2_1 (all the way through master). It should be applied to anything we want to support rhel 5 on. Should I submit additional patches?</p> <p>b2_1 patch:<br/> <a href="http://review.whamcloud.com/5468" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5468</a></p> <p>Landed for 1.8.9 and 2.1.5</p> Development Rank 1|hzv653: Rank (Obsolete) 4529 Severity