https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9949 Lustre <p>It looks like "<a href="https://jira.whamcloud.com/browse/LU-9480" title="LNet Dynamic Discovery" class="issue-link" data-issue-key="LU-9480"><del>LU-9480</del></a> lnet: implement Peer Discovery" commit 0f1aaad4c1b4447ee5097b8bb79a49d09eaa23c2 broke lolnd (suggested by git bisect)</p> <p>This manifests in e.g. sanity test 101b hanging with this in logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 215.914245] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 01:32:15 (1504675935) [ 215.985320] Lustre: lfs: using old ioctl(LL_IOC_LOV_GETSTRIPE) on [0x200000401:0x5:0x0], use llapi_layout_get_by_path() [ 256.717500] LNet: Service thread pid 4032 was inactive for 40.01s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 256.720328] Pid: 4032, comm: ll_ost_io00_002 [ 256.721561] Call Trace: [ 256.723391] [&lt;ffffffff81704339&gt;] schedule+0x29/0x70 [ 256.724533] [&lt;ffffffff81700972&gt;] schedule_timeout+0x162/0x2a0 [ 256.725651] [&lt;ffffffff810879f0&gt;] ? process_timeout+0x0/0x10 [ 256.726859] [&lt;ffffffffa0534e3e&gt;] target_bulk_io+0x4ee/0xb20 [ptlrpc] [ 256.729276] [&lt;ffffffff810b7ce0&gt;] ? default_wake_function+0x0/0x20 [ 256.730431] [&lt;ffffffffa05ddf08&gt;] tgt_brw_read+0xf38/0x1870 [ptlrpc] [ 256.731359] [&lt;ffffffffa01ba4a4&gt;] ? libcfs_log_return+0x24/0x30 [libcfs] [ 256.732387] [&lt;ffffffffa0579f90&gt;] ? lustre_pack_reply_v2+0x1a0/0x2a0 [ptlrpc] [ 256.733578] [&lt;ffffffffa0532800&gt;] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 256.734845] [&lt;ffffffffa057a102&gt;] ? lustre_pack_reply_flags+0x72/0x1f0 [ptlrpc] [ 256.736719] [&lt;ffffffffa057a291&gt;] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [ 256.737931] [&lt;ffffffffa05dad2b&gt;] tgt_request_handle+0x93b/0x1390 [ptlrpc] [ 256.738981] [&lt;ffffffffa05853b1&gt;] ptlrpc_server_handle_request+0x251/0xae0 [ptlrpc] [ 256.740764] [&lt;ffffffffa0589168&gt;] ptlrpc_main+0xa58/0x1df0 [ptlrpc] [ 256.741800] [&lt;ffffffff81706487&gt;] ? _raw_spin_unlock_irq+0x27/0x50 [ 256.742938] [&lt;ffffffffa0588710&gt;] ? ptlrpc_main+0x0/0x1df0 [ptlrpc] [ 256.743943] [&lt;ffffffff810a2eda&gt;] kthread+0xea/0xf0 [ 256.744963] [&lt;ffffffff810a2df0&gt;] ? kthread+0x0/0xf0 [ 256.745913] [&lt;ffffffff8170fbd8&gt;] ret_from_fork+0x58/0x90 [ 256.746933] [&lt;ffffffff810a2df0&gt;] ? kthread+0x0/0xf0 [ 256.748798] LustreError: dumping log to /tmp/lustre-log.1504675975.4032 [ 269.494952] LustreError: 2624:0:(events.c:449:server_bulk_callback()) event type 5, status -5, desc ffff8800720b3e00 </pre> </div></div> <p>Easy to reproduce, just run this on a single node: ONLY=101 REFORMAT=yes sh sanity.sh</p> LU-9949

lolnd broken

Bug Major Resolved Duplicate Amir Shehata Oleg Drokin Wed, 6 Sep 2017 06:30:01 +0000 Sun, 10 Feb 2019 00:12:07 +0000 Wed, 22 Aug 2018 17:48:54 +0000 0 8 <p>This totally breaks significant chunk of my testing with 100% failure rate. was not initially noticed because I am not well equipped to catch such hangs yet.</p> <p>Amir</p> <p>Can you please investigate</p> <p>Peter</p> <p>I ran 101b by itself, and I'm seeing the following logs:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 7 19:55:58 centos6-16 kernel: Lustre: lustre-OST0001: <span class="code-keyword">new</span> disk, initializing Sep 7 19:55:58 centos6-16 kernel: Lustre: srv-lustre-OST0001: No data found on store. Initialize space Sep 7 19:55:58 centos6-16 kernel: Lustre: lustre-MDT0000: Connection restored to 192.168.10.226@tcp (at 192.168.10.226@tcp) Sep 7 19:55:58 centos6-16 kernel: Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 Sep 7 19:55:59 centos6-16 kernel: LustreError: 4252:0:(obd_config.c:1431:class_process_proc_param()) lustre-clilov-ffff88007647f800: error writing proc <span class="code-quote">'stripecount=-1'</span>=<span class="code-quote">'-1'</span>: rc = -34 Sep 7 19:56:01 centos6-16 kernel: Lustre: lustre-OST0000: Connection restored to 192.168.10.226@tcp (at 192.168.10.226@tcp) Sep 7 19:56:01 centos6-16 rsyslogd-2013: fopen() failed: <span class="code-quote">'Read-only file system'</span>, path: <span class="code-quote">'/<span class="code-keyword">var</span>/lib/rsyslog/imjournal.state.tmp'</span>#012 [<span class="code-keyword">try</span> http:<span class="code-comment">//www.rsyslog.com/e/2013 ] </span>Sep 7 19:56:06 centos6-16 kernel: Lustre: lustre-MDT0000: Connection restored to 192.168.10.226@tcp (at 192.168.10.226@tcp) Sep 7 19:56:06 centos6-16 kernel: Lustre: Skipped 2 previous similar messages Sep 7 19:56:06 centos6-16 kernel: Lustre: Mounted lustre-client Sep 7 19:56:08 centos6-16 kernel: Lustre: DEBUG MARKER: Using TIMEOUT=20 Sep 7 19:56:15 centos6-16 kernel: LNetError: 2602:0:(lib-move.c:2395:lnet_parse_ack()) 192.168.10.226@tcp: Dropping ACK from 12345-192.168.10.226@tcp to invalid MD 0x14e239980d506816.0x18e5 Sep 7 19:56:15 centos6-16 kernel: LNetError: 2602:0:(lib-move.c:2800:lnet_parse()) 192.168.10.226@tcp, src 192.168.10.226@tcp: Dropping ACK. parse_local failed: -2 Sep 7 19:56:15 centos6-16 kernel: Lustre: DEBUG MARKER: excepting tests: 76 407 253 312 42a 42b 42c 45 68b Sep 7 19:56:15 centos6-16 kernel: Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 64b 68 71 115 300o Sep 7 19:56:16 centos6-16 kernel: Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 19:56:16 (1504828576) </pre> </div></div> <p>There appears to be a dropped ACK because of no valid MD. Prior to that there is a failure </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 4252:0:(obd_config.c:1431:class_process_proc_param()) lustre-clilov-ffff88007647f800: error writing proc <span class="code-quote">'stripecount=-1'</span>=<span class="code-quote">'-1'</span>: rc = -34 </pre> </div></div> <p>error is ERANGE. Is this expected in that test?</p> <p>LNet stats:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - net type: tcp local NI(s): - nid: 192.168.10.226@tcp status: up statistics: send_count: 809 recv_count: 808 drop_count: 1 sent_stats: put: 777 get: 32 reply: 0 ack: 0 hello: 0 received_stats: put: 758 get: 16 reply: 16 ack: 18 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 1 hello: 0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: tcp bonding: 0 dev cpt: -1 CPT: <span class="code-quote">"[0]"</span> </pre> </div></div> <p>The only issue the stats seem to point to is the dropped ACK.</p> <p>Please note that this is on the socklnd and not the lolnd. There appears to be no problems on the lolnd.</p> <p>&gt; Prior to that there is a failure</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 4252:0:(obd_config.c:1431:class_process_proc_param()) lustre-clilov-ffff88007647f800: error writing proc 'stripecount=-1'='-1': rc = -34 </pre> </div></div> <p>&gt; error is ERANGE. Is this expected in that test?</p> <p>This is fine.</p> <p>I believe that lnet is smart enough to use lolnd when it notices that the target nid is the same as its own.</p> <p>Simplified reproducer (on a single Oleg test node):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dd if=/dev/zero of=/mnt/lustre/f1 bs=1M count=1 lctl set_param ldlm.namespaces.*-osc-*.lru_size=clear lctl set_param osd-ldiskfs.*.read_cache_enable=0 dd if=/mnt/lustre/f1 of=/dev/null bs=1M </pre> </div></div> <p>I'm seeing this from time to time which is preventing me from mounting a test file system.</p> <p>00000020:00000080:4.0:1505255706.626213:0:6697:0:(obd_config.c:431:class_attach()) OBD: dev 1 attached type mgc with refcount 1<br/> 00000020:00000080:4.0:1505255706.626217:0:6697:0:(obd_config.c:1138:class_process_config()) processing cmd: cf003<br/> 00000100:00000100:5.0:1505255706.627789:0:6697:0:(client.c:96:ptlrpc_uuid_to_connection()) cannot find peer MGC10.37.248.196@o2ib1<br/> _0!<br/> 00010000:00080000:5.0:1505255706.627791:0:6697:0:(ldlm_lib.c:74:import_set_conn()) can't find connection MGC10.37.248.196@o2ib1_0<br/> 00010000:00020000:5.0:1505255706.627793:0:6697:0:(ldlm_lib.c:483:client_obd_setup()) can't add initial connection<br/> 00000020:00000080:1.0:1505255706.638190:0:6711:0:(genops.c:1204:class_import_destroy()) destroying import ffff88101ff0a000 for MGC<br/> 10.37.248.196@o2ib1<br/> 00000020:00020000:6.0:1505255706.639014:0:6697:0:(obd_config.c:563:class_setup()) setup MGC10.37.248.196@o2ib1 failed (-2)<br/> 00000020:00020000:6.0:1505255706.650104:0:6697:0:(obd_mount.c:202:lustre_start_simple()) MGC10.37.248.196@o2ib1 setup error -2<br/> 00000020:00000080:6.0:1505255706.661495:0:6697:0:(obd_config.c:1138:class_process_config()) processing cmd: cf002</p> <p>Is this the same bug or such I open another ticket?</p> <p>The problem happens when read_cache_enable=0. If read_cache_enable=1 then the test passes (both 101b and John's reproducer)</p> <p>In the case when read_cache_enable=0 generic_error_remove_page() is being called for each page in osd_read_prep(). Looking at this function it unmaps the pages. This seems to cause a problem on Oleg's setup. After the pages are unmapped the socklnd does not seem to complete the bulk transfer. The server appears to call the kernel API to send the page, but the client socklnd (which is common with the server, since it's the same node) never receives the data. </p> <p>I haven't been able to reproduce this issue on my local VM. Oleg's VM is running a debug Kernel, and I'm wondering if its behavior is different. I'll attempt to step through the code and see if the data is being dropped somewhere in the kernel call stack.</p> <p>Oleg have you tested this on a multiple node setup using the same debug kernel?</p> <p>&gt; I believe that lnet is smart enough to use lolnd when it notices that the target nid is the same as its own.</p> <p>This is no longer the case after 0f1aaad4c1b4447ee5097b8bb79a49d09eaa23c2 <a href="https://jira.whamcloud.com/browse/LU-9480" title="LNet Dynamic Discovery" class="issue-link" data-issue-key="LU-9480"><del>LU-9480</del></a> lnet: implement Peer Discovery. Which seems like a bug.</p> <p>Multinode setup appears to be working ok</p> <p>One outstanding question that should be answered: Is it safe for generic_error_remove_page() to be called on the pages before they are passed to the LND?</p> <p>According to Alex: yes, it's safe as page is removed from the mapping (so another threads can't find it), but it can't get reused for anything else until last page_put() is called</p> <p>to summarize, the issue seems to be observed only on Oleg's VM on a single node setup running a debug kernel. It appears that calling generic_error_remove_page() which unmaps the pages before they are sent by the socklnd causes the socklnd send to succeed, but not to actually send the pages. It is still not known exactly why that happens.</p> <p>This issue occurred after 0f1aaad4c1b4447ee5097b8bb79a49d09eaa23c2 which discovers the loopback interface when it's initially used then due to the lolnd having no credits always prefers the other interfaces. This issue has been resolved in <a href="https://jira.whamcloud.com/browse/LU-9992" title="Multi-Rail: use lolnd when sending locally" class="issue-link" data-issue-key="LU-9992"><del>LU-9992</del></a>.</p> <p>Should this still be a blocker?</p> <p>Sounds like a duplicate </p> Duplicate LU-9992 Related LU-9992 Development Rank 1|hzzjmf: Rank (Obsolete) 9223372036854775807 Severity