https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7794 Lustre <p>We saw this during DNE failover soak-test.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: 4374:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1455822992/real 1455822992] req@ffff8807bba3b980 x1526529651962748/t0(0) o250-&gt;MGC192.168.1.108@o2ib10@0@lo:26/25 lens 520/544 e 0 to 1 dl 1455823038 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 4374:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 37 previous similar messages LustreError: 8115:0:(tgt_lastrcvd.c:1464:tgt_clients_data_init()) soaked-MDT0001: duplicate export for client generation 3 LustreError: 8115:0:(obd_config.c:578:class_setup()) setup soaked-MDT0001 failed (-114) LustreError: 8115:0:(obd_config.c:1666:class_config_llog_handler()) MGC192.168.1.108@o2ib10: cfg command failed: rc = -114 Lustre: cmd=cf003 0:soaked-MDT0001 1:soaked-MDT0001_UUID 2:1 3:soaked-MDT0001-mdtlov 4:f </pre> </div></div> <p>And it cause the failover MDT can not be mounted on the new MDS. Similar as <a href="https://jira.whamcloud.com/browse/LU-7430" title="General protection fault: 0000 upon mounting MDT" class="issue-link" data-issue-key="LU-7430"><del>LU-7430</del></a>, but no panic this time.</p> LU-7794

tgt_clients_data_init()) soaked-MDT0001: duplicate export for client generation 1

Bug Critical Resolved Duplicate Gregoire Pichon Di Wang soak Thu, 18 Feb 2016 19:54:10 +0000 Thu, 3 Nov 2022 03:10:07 +0000 Thu, 3 Nov 2022 03:10:07 +0000 Lustre 2.8.0 0 7 <p>debuglog snippet </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:01000000:8.0:1455822976.430423:0:8057:0:(obd_config.c:1562:class_config_llog_handler()) For 2.x interoperability, rename obd type from lov to lod (soaked-MDT0001) 00000020:00000080:8.0:1455822976.430426:0:8057:0:(obd_config.c:1148:class_process_config()) processing cmd: cf001 00000020:00000080:8.0:1455822976.430429:0:8057:0:(obd_config.c:362:class_attach()) attach type lod name: soaked-MDT0001-mdtlov uuid: soaked-MDT0001-mdtlov_UUID 00000020:00000080:8.0:1455822976.430526:0:8057:0:(genops.c:371:class_newdev()) Adding new device soaked-MDT0001-mdtlov (ffff8807b479f078) 00000020:00000080:8.0:1455822976.430530:0:8057:0:(obd_config.c:432:class_attach()) OBD: dev 59 attached type lod with refcount 1 00000020:00000080:8.0:1455822976.430534:0:8057:0:(obd_config.c:1148:class_process_config()) processing cmd: cf003 00080000:01000000:8.0:1455822976.430613:0:8057:0:(osd_handler.c:6597:osd_obd_connect()) connect #1 00000020:00000080:8.0:1455822976.430617:0:8057:0:(genops.c:1167:class_connect()) connect: client soaked-MDT0001-osd_UUID, cookie 0xe84502d68e3d0d9a 00000020:00000080:8.0:1455822976.430707:0:8057:0:(obd_config.c:552:class_setup()) finished setup of obd soaked-MDT0001-mdtlov (uuid soaked-MDT0001-mdtlov_UUID) 00000020:01000000:8.0:1455822976.430711:0:8057:0:(obd_config.c:1487:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:8.0:1455822976.430714:0:8057:0:(obd_config.c:1148:class_process_config()) processing cmd: cf010 00000020:00000080:8.0:1455822976.430715:0:8057:0:(obd_config.c:1218:class_process_config()) marker 8 (0x2) soaked-MDT0001-m lov setup 00000020:01000000:8.0:1455822976.430717:0:8057:0:(obd_config.c:1487:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:8.0:1455822976.430720:0:8057:0:(obd_config.c:1148:class_process_config()) processing cmd: cf010 00000020:00000080:8.0:1455822976.430721:0:8057:0:(obd_config.c:1218:class_process_config()) marker 9 (0x1) soaked-MDT0001 add mdt 00000020:00000080:8.0:1455822976.430724:0:8057:0:(obd_config.c:1148:class_process_config()) processing cmd: cf001 00000020:00000080:8.0:1455822976.430725:0:8057:0:(obd_config.c:362:class_attach()) attach type mdt name: soaked-MDT0001 uuid: soaked-MDT0001_UUID 00000020:00000080:8.0:1455822976.430816:0:8057:0:(genops.c:371:class_newdev()) Adding new device soaked-MDT0001 (ffff8807a97fd038) 00000020:00000080:8.0:1455822976.430819:0:8057:0:(obd_config.c:432:class_attach()) OBD: dev 60 attached type mdt with refcount 1 00000020:00000080:8.0:1455822976.430822:0:8057:0:(obd_config.c:1148:class_process_config()) processing cmd: cf007 00000020:00000080:8.0:1455822976.430823:0:8057:0:(obd_config.c:1176:class_process_config()) mountopt: profile soaked-MDT0001 osc soaked-MDT0001-mdtlov mdc (null) 00000020:01000000:8.0:1455822976.430826:0:8057:0:(obd_config.c:873:class_add_profile()) Add profile soaked-MDT0001 00000020:00000080:8.0:1455822976.430831:0:8057:0:(obd_config.c:1148:class_process_config()) processing cmd: cf003 00000020:01000004:8.0:1455822976.430883:0:8057:0:(obd_mount_server.c:175:server_get_mount()) get mount ffff8807c8232400 from soaked-MDT0001, refs=2 00000020:00000080:8.0:1455822976.430908:0:8057:0:(obd_config.c:362:class_attach()) attach type mdd name: soaked-MDD0001 uuid: soaked-MDD0001_UUID 00000020:00000080:8.0:1455822976.430998:0:8057:0:(genops.c:371:class_newdev()) Adding new device soaked-MDD0001 (ffff8807ea567078) 00000020:00000080:8.0:1455822976.431001:0:8057:0:(obd_config.c:432:class_attach()) OBD: dev 61 attached type mdd with refcount 1 00000004:01000000:8.0:1455822976.431060:0:8057:0:(lod_dev.c:1687:lod_obd_connect()) connect #0 00000020:00000080:8.0:1455822976.431064:0:8057:0:(genops.c:1167:class_connect()) connect: client soaked-MDT0001-mdtlov_UUID, cookie 0xe84502d68e3d0daf 00000020:00000080:8.0:1455822976.431096:0:8057:0:(obd_config.c:552:class_setup()) finished setup of obd soaked-MDD0001 (uuid soaked-MDD0001_UUID) 00000004:01000000:8.0:1455822976.431103:0:8057:0:(mdd_device.c:1178:mdd_obd_connect()) connect #000000020:00000080:8.0:1455822976.431106:0:8057:0:(genops.c:1167:class_connect()) connect: client soaked-MDD0001_UUID, cookie 0xe84502d68e3d0db6 00080000:01000000:8.0:1455822976.431113:0:8057:0:(osd_handler.c:6597:osd_obd_connect()) connect #2 00000020:00000080:8.0:1455822976.431115:0:8057:0:(genops.c:1167:class_connect()) connect: client soaked-MDT0001-osd_UUID, cookie 0xe84502d68e3d0dbd 00000001:00080000:8.0:1455822976.461440:0:8057:0:(tgt_lastrcvd.c:1427:tgt_clients_data_init()) RCVRNG CLIENT uuid: soaked-MDT0000-mdtlov_UUID idx: 0 lr: 17182107883 srv lr: 17180780327 lx: 1526530761809700 gen 0 00000001:00080000:8.0:1455822976.461464:0:8057:0:(tgt_lastrcvd.c:1427:tgt_clients_data_init()) RCVRNG CLIENT uuid: soaked-MDT0003-mdtlov_UUID idx: 1 lr: 17181767205 srv lr: 17180780327 lx: 1526529647176604 gen 0 00000001:00080000:8.0:1455822976.461475:0:8057:0:(tgt_lastrcvd.c:1427:tgt_clients_data_init()) RCVRNG CLIENT uuid: soaked-MDT0002-mdtlov_UUID idx: 2 lr: 17182149935 srv lr: 17180780327 lx: 1526529649871472 gen 0 00000001:00080000:8.0:1455822976.461483:0:8057:0:(tgt_lastrcvd.c:1427:tgt_clients_data_init()) RCVRNG CLIENT uuid: soaked-MDT0005-mdtlov_UUID idx: 3 lr: 17182105205 srv lr: 17180780327 lx: 1526525227761264 gen 0 00000001:00080000:8.0:1455822976.461493:0:8057:0:(tgt_lastrcvd.c:1427:tgt_clients_data_init()) RCVRNG CLIENT uuid: soaked-MDT0004-mdtlov_UUID idx: 4 lr: 17182152956 srv lr: 17180780327 lx: 1526525229666900 gen 0 00000001:00080000:8.0:1455822976.461503:0:8057:0:(tgt_lastrcvd.c:1427:tgt_clients_data_init()) RCVRNG CLIENT uuid: soaked-MDT0007-mdtlov_UUID idx: 5 lr: 17182138513 srv lr: 17180780327 lx: 1526536609097660 gen 0 00000001:00080000:8.0:1455822976.461512:0:8057:0:(tgt_lastrcvd.c:1427:tgt_clients_data_init()) RCVRNG CLIENT uuid: soaked-MDT0006-mdtlov_UUID idx: 6 lr: 17181782621 srv lr: 17180780327 lx: 1526536607769984 gen 0 00000001:00080000:8.0:1455822976.461520:0:8057:0:(tgt_lastrcvd.c:1427:tgt_clients_data_init()) RCVRNG CLIENT uuid: c2781877-e222-f9a8-07f4-a1250eba2af6 idx: 7 lr: 0 srv lr: 17180780327 lx: 0 gen 1 00000001:00080000:8.0:1455822976.461530:0:8057:0:(tgt_lastrcvd.c:1427:tgt_clients_data_init()) RCVRNG CLIENT uuid: 150208e6-71a8-375e-d139-d478eec5b761 idx: 8 lr: 0 srv lr: 17180780327 lx: 0 gen 1 00000001:00020000:8.0:1455822976.461539:0:8057:0:(tgt_lastrcvd.c:1464:tgt_clients_data_init()) soaked-MDT0001: duplicate export for client generation 1 </pre> </div></div> <p>Gregoire</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Honestly, I have no idea what could lead to that duplicate export situation.</p> <p>Could you detail what tests were running on the filesystem ?</p> <p>Do you think you could rebuild the operations (client connections/disconnections, MDT mount/unmount, other MDTs connections/disconnections, ...) that occurred on the filesystem from the node logs ?</p> <p>Hmm, I am not sure we can reproduce the problem easily. I am not sure there are node logs. If you need debug log, maybe I can upload it somewhere.</p> <p>Yes, could upload debug logs as attachment to this JIRA ticket ?</p> <p>Are the D_INFO messages written to the logs ?<br/> It would be helpful to look for the D_INFO messages logged in tgt_client_new()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">CDEBUG(D_INFO, <span class="code-quote">"%s: <span class="code-keyword">new</span> client at index %d (%llu) with UUID <span class="code-quote">'%s'</span> "</span> <span class="code-quote">"generation %d\n"</span>, tgt-&gt;lut_obd-&gt;obd_name, ted-&gt;ted_lr_idx, ted-&gt;ted_lr_off, ted-&gt;ted_lcd-&gt;lcd_uuid, ted-&gt;ted_lcd-&gt;lcd_generation); </pre> </div></div> <p>And see in which context they were called.</p> <p>And again, could you detail what tests were running on the filesystem, please ?</p> <p>I upload the debug log to ftp.hpdd.intel.com:/uploads/lu-7794/</p> <p>The whole process does like this, 4 MDS, each MDS has 2 MDTs. MDS0 (MDT0/1), MDS1(MDT2/3), MDS2(MDT4/5), MDS3(MDT6/7). MDS0 and MDS1 are configured as active/active failover, and MDS2 and MDS3 are configured as active/active failover.</p> <p>And the test is randomly chosen one of MDS to reboot. then its MDTs will be failover to its pair MDS. In this case, it is MDS0 is restarted, then MDT0/MDT1 should be mounted on MDS1, but failed because of this. No, I do not think this can be easily produced.</p> <p>Maybe Frank or Cliff will know more.</p> <p>Gregoire: I hope the info's below will answer you're question. Please let me know if something is missing or unclear.</p> <h3><a name="smallenvironmentinfoaddition"></a>small environment info addition</h3> <ul> <li><tt>lola-8</tt> --&gt; MDS0, <tt>lola-9</tt> --&gt; MDS1, <tt>lola-10</tt> --&gt; MDS2, <tt>lola-11</tt> --&gt; MDS3</li> <li>MDTs formatted with <em>ldiskfs</em> , OSTs using <em>zfs</em></li> <li>Failover procedure <ul> <li>Triggered by automated framework</li> <li>(Random) selected node is powercycled at (randomly) chosen time</li> <li>Wait till node is up again</li> <li>Mount resources on failover partner:<br/> Mount MDT resources in sequence <br/> Wait for mount command to complete<br/> If error occurrs retry to mount Lustre MDT(s) again.</li> <li>Framework is configured for test session NOT to wait for RECOVERY process to complete</li> <li>umount MDTs on secondary node</li> <li>mount MDTs on primary node again</li> </ul> </li> <li>NOTE:<br/> I checked the soak framework. I'm very sure that the implementation <b>won't'</b> execute multiple <br/> <tt>mount</tt> commands at the same time or start a new <tt>mount</tt> command while the one started before<br/> haven't finished, yet. <br/> The framework <b>didn't</b> check whether there's already a mount command running executed from outside the framework.</li> </ul> <h3><a name="Concerning%27sequenceofevents%27"></a>Concerning 'sequence of events'</h3> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2016-02-18 11:07:14,437:fsmgmt.fsmgmt:INFO triggering fault mds_failover 2016-02-18 11:07:14,438:fsmgmt.fsmgmt:INFO reseting MDS node lola-8 (--&gt; Node was powercycled!) 2016-02-18 11:07:14,439:fsmgmt.fsmgmt:INFO executing cmd pm -h powerman -c lola-8&gt; /dev/null 2016-02-18 11:07:28,291:fsmgmt.fsmgmt:INFO trying to connect to lola-8 ... 2016-02-18 11:07:38,307:fsmgmt.fsmgmt:INFO trying to connect to lola-8 ... 2016-02-18 11:07:46,410:fsmgmt.fsmgmt:INFO trying to connect to lola-8 ... ... ... 2016-02-18 11:13:36,132:fsmgmt.fsmgmt:INFO trying to connect to lola-8 ... 2016-02-18 11:13:37,060:fsmgmt.fsmgmt:INFO lola-8 is up!!! 2016-02-18 11:13:48,072:fsmgmt.fsmgmt:INFO Failing over soaked-MDT0001 ... 2016-02-18 11:13:48,073:fsmgmt.fsmgmt:INFO Mounting soaked-MDT0001 on lola-9 ... 2016-02-18 11:16:16,760:fsmgmt.fsmgmt:ERROR ... mount of soaked-MDT0001 on lola-9 failed with 114, retrying ... 2016-02-18 11:16:16,760:fsmgmt.fsmgmt:INFO mount.lustre: increased /sys/block/dm-10/queue/max_sectors_kb from 1024 to 16383 mount.lustre: increased /sys/block/dm-8/queue/max_sectors_kb from 1024 to 16383 mount.lustre: increased /sys/block/sdg/queue/max_sectors_kb from 1024 to 16383 mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mount.lustre: mount /dev/mapper/360080e50002ffd820000024f52013094p1 at /mnt/soaked-mdt1 failed: Operation already in progress The target service is already running. (/dev/mapper/360080e50002ffd820000024f52013094p1) 2016-02-18 11:18:12,738:fsmgmt.fsmgmt:ERROR ... mount of soaked-MDT0001 on lola-9 failed with 114, retrying ... 2016-02-18 11:18:12,739:fsmgmt.fsmgmt:INFO mount.lustre: mount /dev/mapper/360080e50002ffd820000024f52013094p1 at /mnt/soaked-mdt1 failed: Operation already in progress The target service is already running. (/dev/mapper/360080e50002ffd820000024f52013094p1) 2016-02-18 11:26:58,512:fsmgmt.fsmgmt:INFO ... soaked-MDT0001 mounted successfully on lola-9 2016-02-18 11:26:58,513:fsmgmt.fsmgmt:INFO ... soaked-MDT0001 failed over 2016-02-18 11:26:58,513:fsmgmt.fsmgmt:INFO Failing over soaked-MDT0000 ... 2016-02-18 11:26:58,513:fsmgmt.fsmgmt:INFO Mounting soaked-MDT0000 on lola-9 ... 2016-02-18 11:27:51,049:fsmgmt.fsmgmt:INFO ... soaked-MDT0000 mounted successfully on lola-9 2016-02-18 11:27:51,049:fsmgmt.fsmgmt:INFO ... soaked-MDT0000 failed over 2016-02-18 11:28:11,430:fsmgmt.fsmgmt:DEBUG Recovery Result Record: {'lola-9': {'soaked-MDT0001': 'RECOVERING', 'soaked-MDT0000': 'RECOVERING', 'soaked-MDT0003': 'COMPLETE', 'soaked-MDT0002': 'COMPLETE'}} 2016-02-18 11:28:11,431:fsmgmt.fsmgmt:INFO soaked-MDT0001 in status 'RECOVERING'. 2016-02-18 11:28:11,431:fsmgmt.fsmgmt:INFO soaked-MDT0000 in status 'RECOVERING'. 2016-02-18 11:28:11,431:fsmgmt.fsmgmt:INFO Don't wait for recovery to complete. Failback MDT's immediately 2016-02-18 11:28:11,431:fsmgmt.fsmgmt:INFO Failing back soaked-MDT0001 ... 2016-02-18 11:28:11,431:fsmgmt.fsmgmt:INFO Unmounting soaked-MDT0001 on lola-9 ... 2016-02-18 11:28:12,078:fsmgmt.fsmgmt:INFO ... soaked-MDT0001 unmounted successfully on lola-9 2016-02-18 11:28:12,079:fsmgmt.fsmgmt:INFO Mounting soaked-MDT0001 on lola-8 ... 2016-02-18 11:29:03,122:fsmgmt.fsmgmt:INFO ... soaked-MDT0001 mounted successfully on lola-8 2016-02-18 11:29:03,122:fsmgmt.fsmgmt:INFO ... soaked-MDT0001 failed back 2016-02-18 11:29:03,123:fsmgmt.fsmgmt:INFO Failing back soaked-MDT0000 ... 2016-02-18 11:29:03,123:fsmgmt.fsmgmt:INFO Unmounting soaked-MDT0000 on lola-9 ... 2016-02-18 11:29:09,942:fsmgmt.fsmgmt:INFO ... soaked-MDT0000 unmounted successfully on lola-9 2016-02-18 11:29:09,942:fsmgmt.fsmgmt:INFO Mounting soaked-MDT0000 on lola-8 ... 2016-02-18 11:29:24,023:fsmgmt.fsmgmt:INFO ... soaked-MDT0000 mounted successfully on lola-8 2016-02-18 11:29:24,023:fsmgmt.fsmgmt:INFO ... soaked-MDT0000 failed back 2016-02-18 11:29:24,024:fsmgmt.fsmgmt:INFO mds_failover just completed 2016-02-18 11:29:24,024:fsmgmt.fsmgmt:INFO next fault in 1898s </pre> </div></div> <p>The error message at <tt>2016-02-18 11:16:16,760 ... failed with 114 ...</tt> is strange as (stated above) the soak framework<br/> don't execute multiple mounts of the same device file. <br/> I'm not sure whether some manual mount (outside the soak framework) of MDT1 took place. I'm sure I didn't execute anything on the node at this time.</p> <h3><a name="applicationsexecuted"></a>applications executed</h3> <ul> <li>mdtest (single shared file, file per process)</li> <li>IOR (single shared file, file per process)</li> <li>simul</li> <li>blogbench</li> <li>kcompile</li> <li>pct (producer, consumer inhouse application)<br/> All applications are initiated with random size, file count, block size. If needed I could provide the (slurm) list of active jobs at the time the error occured.</li> </ul> <p>it's interesting that in this case and in LDEV-180 the duplicated entry had generation=1. meaning that for a reason the counter got reset. </p> <p>Di Wang,</p> <p>Unfortunately, I cannot get the debug log from ftp.hpdd.intel.com:/uploads/lu-7794/, since the anonymous login only provides write access.</p> <p>Would it be another way to provide these logs ?</p> <p>Why the error reported in the description of the ticket (<tt>soaked-MDT0001: duplicate export for client generation 3</tt>) is not the same than the error reported in the first comment from Di Wang (<tt>soaked-MDT0001: duplicate export for client generation 1</tt>) ?</p> <p>Does it mean there has been several occurences of the issue ?<br/> Could it be possible to have the logs of the first occurence ?</p> <p>By the way, how many Lustre clients are connected to the filesystem ?<br/> Are the two clients with uuid <tt>c2781877-e222-f9a8-07f4-a1250eba2af6</tt> and <tt>150208e6-71a8-375e-d139-d478eec5b761</tt> different from others ?</p> <p>There are 8 clients connected to the filesystem. I will check on client differences, but afaik they are uniform.</p> <p>Grégoire</p> <p>To turn the question around - is there someone you would like the logs uploaded to so that you can access them?</p> <p>Peter</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Does it mean there has been several occurences of the issue ? Could it be possible to have the logs of the first occurrence ? </pre> </div></div> <p>Yes, there are several occurrences according to the log. You can tell me one place, and I can upload the debug log there. Unfortunately the debug log is too big (more 100M) to be uploaded here.</p> <p>I have been able to access to the log yesterday. Unfortunately, I have not found any usefull information that could help identify the cause of the problem.</p> <p>Anyway, I wonder if the <tt>tgt_client_new()</tt> routine could be called after <tt>tgt_init()</tt> and before <tt>tgt_clients_data_init()</tt> leading to assign the client lcd_generation using a wrong <tt>lut_client_generation</tt> value. The <tt>lut_client_generation</tt> value is initialized to 0 in <tt>tgt_init()</tt> but updated with the highest client generation read from last_rcvd file in case of recovery.</p> <p>In case of recovery, if a client connects to the MDT in parallel of the target initialization, are its <tt>obd_export</tt> and <tt>tg_export_data</tt> structures immediately initialized ?</p> Duplicate LU-15935 Related LU-7430 Development Rank 1|hzy1s7: Rank (Obsolete) 9223372036854775807 Severity