https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6805 Lustre <p>at_init() modifies part of struct adaptive_timeout without taking at_lock. That makes it unsafe to be used anywhere but on creation time.<br/> That is at_init() should not be used in ptlrpc_connect_interpret() and in ptlrpc_service_part_init().</p> LU-6805

at_init is not safe to use anywhere but on initialization

Bug Major Resolved Fixed WC Triage Vladimir Saveliev patch Tue, 7 Jul 2015 13:39:08 +0000 Thu, 23 Nov 2017 18:21:45 +0000 Thu, 16 Jul 2015 12:47:10 +0000 Lustre 2.8.0 0 6 <p>Vladimir Saveliev (vladimir_saveliev@xyratex.com) uploaded a new patch: <a href="http://review.whamcloud.com/15522" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15522</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6805" title="at_init is not safe to use anywhere but on initialization" class="issue-link" data-issue-key="LU-6805"><del>LU-6805</del></a> ptlrpc: use smp unsafe at_init only for initialization<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 568e59b44e4f093436ff83f1ffd860aaff114b46</p> <p>What problems are seen without this patch? Have you seen issues with this in real life, or was this found by code inspection or static analysis?</p> <p>It seems like a problem that the spinlock is being reset while it might be locked. That might cause an oops if there is spinlock debugging enabled in some rare cases. If that is the case, then including the stack trace in the big and/or patch would make it easier to correlate any future failures with this ticket and patch. </p> <p>Yes, we're seeing threads stuck in infinite loops trying to get the spinlock. This problem has been seen on at least 2 customer systems and a few internal test systems.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&gt; crash&gt; ps -m | grep '\^' &gt; ^[0 00:00:00.002] [IN] PID: 573 TASK: ffff88083370b800 CPU: 12 COMMAND: "kworker/12:1" &gt; ^[0 00:00:00.003] [IN] PID: 5698 TASK: ffff88083cb70800 CPU: 15 COMMAND: "gsock_send_1" [skip] &gt; ^[0 00:00:00.252] [IN] PID: 5785 TASK: ffff88083f074040 CPU: 0 COMMAND: "ptlrpcd_15" &gt; ^[0 00:41:02.337] [RU] PID: 5781 TASK: ffff880833673800 CPU: 2 COMMAND: "ptlrpcd_11" ^41 minutes running on CPU 2 &gt; crash&gt; bt 5781 &gt; PID: 5781 TASK: ffff880833673800 CPU: 2 COMMAND: "ptlrpcd_11" &gt; [exception RIP: _raw_spin_lock+27] &gt; RIP: ffffffff8141eabb RSP: ffff8807e29d3bc0 RFLAGS: 00000202 &gt; RAX: 0000000000000266 RBX: ffff8808378d8ba0 RCX: ffffffffa0559aa0 &gt; RDX: 0000000000001000 RSI: 0000000000000000 RDI: ffff8808378d8bd0 &gt; RBP: ffff8807e29d3bc0 R8: 0000000000000001 R9: 0000000000000001 &gt; R10: 0000000000000001 R11: 0000ffffffffff0a R12: 0000000000000258 &gt; R13: 000000000000003b R14: 0000000055944d05 R15: 0000000000000028 &gt; ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 &gt; --- &lt;NMI exception stack&gt; --- &gt; #3 [ffff8807e29d3bc0] _raw_spin_lock at ffffffff8141eabb &gt; #4 [ffff8807e29d3bc8] at_measured at ffffffffa04c0123 [ptlrpc] &gt; #5 [ffff8807e29d3c38] ptlrpc_at_adj_net_latency at ffffffffa0494547 [ptlrpc] &gt; #6 [ffff8807e29d3c78] after_reply at ffffffffa0497223 [ptlrpc] &gt; #7 [ffff8807e29d3ce8] ptlrpc_check_set at ffffffffa049c3d0 [ptlrpc] &gt; #8 [ffff8807e29d3d78] ptlrpcd_check at ffffffffa04c810b [ptlrpc] &gt; #9 [ffff8807e29d3dd8] ptlrpcd at ffffffffa04c87bb [ptlrpc] &gt; #10 [ffff8807e29d3ee8] kthread at ffffffff8107373e &gt; #11 [ffff8807e29d3f48] kernel_thread_helper at ffffffff81427bb4 spinlock is ptlrpc_request.rq_import-&gt;imp_at.iat_net_latency.at_lock &gt; crash&gt; ptlrpc &gt; &gt; Sent RPCS: ptlrpc_request_set.set_requests-&gt;rq_set_chain &gt; thread ptlrpc_request pid xid nid opc phase bulk sent/deadline &gt; =============================================================================================== &gt; ptlrpcd_11: ffff8807fa1fc000 5781 x1505497848647028 325@gni 400 RPC 0:0 1435782346/1435782478 &gt; =============================================================================================== &gt; crash&gt; ptlrpc_request ffff8807fa1fc000 | grep rq_import &gt; rq_import_generation = 1, &gt; rq_import = 0xffff8808378d8800, &gt; crash&gt; struct -o obd_import | grep imp_at &gt; [896] struct imp_at imp_at; &gt; crash&gt; struct -o imp_at | grep iat_net_latency &gt; [32] struct adaptive_timeout iat_net_latency; &gt; crash&gt; struct -o adaptive_timeout | grep at_lock &gt; [48] spinlock_t at_lock; So spinlock address: 0xffff8808378d8800 + 896 + 32 + 48 = ffff8808378d8bd0 &gt; crash&gt; spinlock_t -x ffff8808378d8bd0 &gt; struct spinlock_t { &gt; { &gt; rlock = { &gt; raw_lock = { &gt; slock = 0x6666 &gt; } &gt; } &gt; } &gt; } This at_lock is not locked. &gt; crash&gt; px xtdumpregs | grep -B 15 -A 6 ffff8808378d8bd0 &gt; }, { &gt; r15 = 0x28, &gt; r14 = 0x55944d05, &gt; r13 = 0x3b, &gt; r12 = 0x258, &gt; bp = 0xffff8807e29d3bc0, &gt; bx = 0xffff8808378d8ba0, &gt; r11 = 0xffffffffff0a, &gt; r10 = 0x1, &gt; r9 = 0x1, &gt; r8 = 0x1, &gt; ax = 0x266, &gt; cx = 0xffffffffa0559aa0, &gt; dx = 0x1000, &gt; si = 0x0, &gt; di = 0xffff8808378d8bd0, &gt; orig_ax = 0xffffffffffffffff, &gt; ip = 0xffffffff8141eabb, &gt; cs = 0x10, &gt; flags = 0x202, &gt; sp = 0xffff8807e29d3bc0, &gt; ss = 0x18 &gt;&gt; RAX: 0000000000000266 RBX: ffff8808378d8ba0 RCX: ffffffffa0559aa0 &gt; crash&gt; dis _raw_spin_lock &gt; 0xffffffff8141eaa0 &lt;_raw_spin_lock&gt;: push %rbp &gt; 0xffffffff8141eaa1 &lt;_raw_spin_lock+1&gt;: mov %rsp,%rbp &gt; 0xffffffff8141eaa4 &lt;_raw_spin_lock+4&gt;: data32 data32 data32 xchg %ax,%ax &gt; 0xffffffff8141eaa9 &lt;_raw_spin_lock+9&gt;: mov $0x100,%eax &gt; 0xffffffff8141eaae &lt;_raw_spin_lock+14&gt;: lock xadd %ax,(%rdi) &gt; 0xffffffff8141eab3 &lt;_raw_spin_lock+19&gt;: cmp %ah,%al &gt; 0xffffffff8141eab5 &lt;_raw_spin_lock+21&gt;: je 0xffffffff8141eabd &lt;_raw_spin_lock+29&gt; &gt; 0xffffffff8141eab7 &lt;_raw_spin_lock+23&gt;: pause &gt; 0xffffffff8141eab9 &lt;_raw_spin_lock+25&gt;: mov (%rdi),%al &gt; 0xffffffff8141eabb &lt;_raw_spin_lock+27&gt;: jmp 0xffffffff8141eab3 &lt;_raw_spin_lock+19&gt; &lt;--- current location &gt; 0xffffffff8141eabd &lt;_raw_spin_lock+29&gt;: leaveq &gt; 0xffffffff8141eabe &lt;_raw_spin_lock+30&gt;: retq &gt; 0xffffffff8141eabf &lt;_raw_spin_lock+31&gt;: nop So %ah == 2 and %al == 66. The value of %ah does not change in the spinlock loop. So %al needs to wrap around to 2 before the loop will exit. (Note: my assembly skills are pretty rusty so good chance I've got something wrong here, but the state of the spinlock compared to the values in use here looks suspicious. ) The dklog shows that other threads have requested this particular spinlock over 300 times since pid 5781 made this hung request. The fact that these threads are not also hung suggests that the spinlock requests were successful. &gt; 00000100:00001000:15:1435782195.117166:0:5781:0:(import.c:1586:at_measured()) add 40 to ffff880837297c48 time=33 v=40 (40 40 0 0) &gt; 00000100:00001000:14:1435782195.260830:0:5777:0:(import.c:1586:at_measured()) add 40 to ffff880837297c48 time=33 v=40 (40 40 0 0) &gt; 00000100:00001000:15:1435782195.270513:0:5773:0:(import.c:1586:at_measured()) add 40 to ffff880837297c48 time=33 v=40 (40 40 0 0) &gt; 00000100:00001000:14:1435782195.285561:0:5777:0:(import.c:1586:at_measured()) add 40 to ffff880837297c48 time=33 v=40 (40 40 0 0) &gt; 00000100:00001000:15:1435782195.291757:0:5773:0:(import.c:1586:at_measured()) add 40 to ffff880837297c48 time=33 v=40 (40 40 0 0) ... &gt; 00000100:00001000:1:1435782534.441946:0:5778:0:(import.c:1586:at_measured()) add 130 to ffff880837297c48 time=72 v=130 (130 0 40 40) </pre> </div></div> <p>One of the issue instances contains the following logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00001000:9:1435782375.689537:0:5910:0:(import.c:1586:at_measured()) add 121 to ffff8807ecb12ba0 time=146 v=96 (96 1 1 1) 00000100:00001000:19:1435782375.689537:0:5896:0:(import.c:1586:at_measured()) add 14 to ffff8807ecb12ba0 time=146 v=96 (96 1 1 1) 00000100:00001000:9:1435782375.689541:0:5910:0:(import.c:1644:at_measured()) AT ffff8807ecb12ba0 change: old=96 new=121 delta=25 (val=121) hist 121 1 1 1 00000100:00001000:19:1435782375.689542:0:5896:0:(import.c:1586:at_measured()) add 14 to ffff8807ecb12ba0 time=1435782375 v=0 (0 0 0 0) </pre> </div></div> <p>crash dump shows that pid 5896 is at_measured() &#45;&gt; spin_lock(&amp;at-&gt;at_lock);-ing on the at (ffff8807ecb12ba0) which has been recently at_init()-ed.<br/> It is possible that the spinlock got corrupted due to race between unprotected initialization and regular spin_lock()-ing.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; adaptive_timeout ffff8807ecb12ba0 struct adaptive_timeout { at_binstart = 0, at_hist = {0, 0, 0, 0}, at_flags = 0, at_current = 0, at_worst_ever = 0, at_worst_time = 1435782375, at_lock = { { rlock = { raw_lock = { slock = 514 } } } } } </pre> </div></div> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15522/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15522/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6805" title="at_init is not safe to use anywhere but on initialization" class="issue-link" data-issue-key="LU-6805"><del>LU-6805</del></a> ptlrpc: use smp unsafe at_init only for initialization<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 96ddb3b168297e7d59a2f4b7b357549f2632bcb4</p> <p>Landed for 2.8</p> Duplicate Development Rank 1|hzxhj3: Rank (Obsolete) 9223372036854775807 Severity