https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6612 Lustre <p>Hi </p> <p>We seeing that the MDT occupied lot of disk space and most of the space was occupied by OBJECTS/* files of the MDT. We took the backup of MDT and restored the Same MDT in the test lab and deleted all the files residing in the OBJECTS/* and removed the CATALOG file and remounted -t lustre and collected the logs. Below is a snippet showing the pattern of logs generated for each file removed. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 00000040:00020000:0.0:1431530471.926250:0:17345:0:(llog_obd.c:320:cat_cancel_cb()) Cannot find handle for log 0x2793959b: -116 00000040:00080000:0.0:1431530471.926254:0:17345:0:(llog_obd.c:346:cat_cancel_cb()) cancel log 0x2793959b:f6d1b675 at index 58627 of catalog 0x13600004 00000040:00080000:0.0:1431530471.926255:0:17345:0:(llog_obd.c:315:cat_cancel_cb()) processing log 0x2792f036:f6d21832 at index 58628 of catalog 0x13600004 00000040:00020000:0.0:1431530471.926259:0:17345:0:(llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile 0x2792f036:0xf6d21832: rc -116 00000040:00020000:0.0:1431530471.926261:0:17345:0:(llog_cat.c:174:llog_cat_id2handle()) error opening log id 0x2792f036:f6d21832: rc -116 00000040:00020000:0.0:1431530471.926262:0:17345:0:(llog_obd.c:320:cat_cancel_cb()) Cannot find handle for log 0x2792f036: -116 00000040:00080000:0.0:1431530471.926265:0:17345:0:(llog_obd.c:346:cat_cancel_cb()) cancel log 0x2792f036:f6d21832 at index 58628 of catalog 0x13600004 00000040:00080000:0.0:1431530471.926267:0:17345:0:(llog_obd.c:315:cat_cancel_cb()) processing log 0x2793959d:f6d27b18 at index 58629 of catalog 0x13600004 00000040:00020000:0.0:1431530471.926270:0:17345:0:(llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile 0x2793959d:0xf6d27b18: rc -116 ... </pre> </div></div> <p>This seems the Lustre bug related to the <a href="https://jira.whamcloud.com/browse/LU-1749" title="llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-1749"><del>LU-1749</del></a>, which is reporting false error messages even the CATALOG file is deleted and if not then are we seeing any issues here related to those deleting files form the OBJECTS directory or is that going to affect the MDT data or going to lead to any data corruption. </p> <p>Thank You,<br/> Manish</p> Lustre Server 2.1.6 LU-6612

(llog_obd.c:346:cat_cancel_cb()) cancel log + llog_obd.c:315:cat_cancel_cb()) processing log + llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile

Bug Minor Resolved Incomplete Bruno Faccini Manish Patel Mon, 18 May 2015 14:42:33 +0000 Tue, 26 Sep 2023 08:00:36 +0000 Tue, 5 Sep 2017 19:33:50 +0000 Lustre 2.1.6 Lustre 2.16.0 Lustre 2.15.4 0 7 <p>Since format of the following log confirms that you are running with patch for <a href="https://jira.whamcloud.com/browse/LU-1749" title="llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-1749"><del>LU-1749</del></a> in 2.1.6:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 00000040:00020000:0.0:1431530471.926262:0:17345:0:(llog_obd.c:320:cat_cancel_cb()) Cannot find handle for log 0x2792f036: -116 </pre> </div></div> <p>I think you are safe.<br/> Do you still see these messages ?</p> <p>Hi Bruno,</p> <p>Yes I still see those messages. On the other hand do we still have to worry about the file system data or this error messages has nothing to do with the consistency of the data. Can you confirm about this.</p> <p>Thank You,<br/> Manish</p> <p>Still with the same log names/ids and index numbers ?</p> <p>Humm, in fact after re-reading your submission text, I wonder if your OSTs were stopped at the time you have removed the files from the OBJECTS directory and the CATALOG file on your MDT ?? Because if not, msgs may come from old/invalid LLOGs reference from OSTs. If this has been the case, you may find corresponding msgs on OSSs, can you check ?</p> <p>Last, are you using ChangeLogs ?</p> <p>Manish, the OBJECTS files are still in use in Lustre 2.1, so they can't just be deleted - they are used to track files being deleted on the MDT but not yet deleted on the OSTs. It is only with Lustre 2.5 and later that the OBJECTS files are obsolete.</p> <p>I'd agree with Bruno - it seems that these OBJECTS files are not being cleaned up because either the ChangeLog has been enabled but there is no consumer, or there is a missing or disabled OST that has objects deleted (or ownership changed) and the MDS is logging these changes until the OST is returned.</p> <p>Bruno, it is worthwhile to check whether disabling all ChangeLog consumers will delete the ChangeLog records, or if they are left behind?</p> <p>Hi Andreas,</p> <p>Changelog was enabled for a period of time until the changelog filled up which resulted in client mount issues with "Lustre clients receive -14 "Bad address" from MDS <span class="error">&#91;REF:1638415019758&#93;</span>" and then changelog user was unregistered to fixt that .</p> <p>The current state is as follows:</p> <p><span class="error">&#91;root@dc2mds01 ~&#93;</span># cat /proc/fs/lustre/mdd/dc2-MDT0000/changelog_users<br/> current index: 10970114278<br/> ID index</p> <p><span class="error">&#91;root@dc2mds01 ~&#93;</span># cat /proc/fs/lustre/mdd/dc2-MDT0000/changelog_mask<br/> MARK CREAT MKDIR HLINK SLINK MKNOD UNLNK RMDIR RENME RNMTO OPEN CLOSE LYOUT TRUNC SATTR XATTR HSM MTIME CTIME</p> <p>Thank You,<br/> Manish</p> <p>Manish,<br/> My initial inclination is that the messages are harmless but I need to investigate more fully to confirm. I have several possible theories as to what kind of scenario could cause this situation. I will be in touch again when I have a clearer picture.<br/> BTW, you did not answer about my previous question regarding the fact the OSTs were stopped or not at the time you removed OBJECTS/* and CATALOG files on MDT??</p> <p>Hi Bruno,</p> <p>The test which customer performed was on an isolated test server and network. I did not use any OSTs in my tests.</p> <p>Here is another question from customer.</p> <p>"I wasn't aware that interacting with the MDT while mounted as ldiskfs could trigger Lustre operations with the OSTs. I suppose I need to do a bit more reading."</p> <p>Currently customer is planning for upcoming maintenance to resize the grown OI.16 file to free up some space.</p> <p>And if this resize process of oi.16 file created issues then can we simply restore the original oi.16 file? Or must we restore the entire dd based MDT backup?</p> <p>In either restoration case, how would we track/clean up any orphaned objects on the OSTs? Could we record enough information about every file we create during testing that we could use to surgically clean up? If so, how? Or, must we perform a full file system lfsck? Would we run into problems if we reverted to our backup and left a few hundred orphaned OST objects?</p> <p>1. So are we OK to delete those files from OBJECTS directory with Lustre v2.1.6. to clean extra occupied space by files residing in the OBJECTS directory on MDT or we upgrade the Lustre with v2.5.x or higher version and then perform the deletion of those files residing in the OBJECTS directory.<br/> 2. How would we track/clean up any orphaned objects on the OSTs with Lustre v2.1.6? Is an lfsck is an option here, if so then please provide the procedure to clean the orphan objects.</p> <p>Thank You,<br/> Manish</p> <p>Manish,<br/> I am sorry but I am puzzled here, do you mean that you got the original msgs/errors for this ticket after restoring a MDT backup, removing OBJECTS/* and CATALOG files, and then mount it single (ie, without the original OSTs) ? And this just to test/see what will happen ? Did I understand the situation ? Because, if yes I don't think this "test" could be representative of the reality.</p> <p>And now, you speak about removing the OI.16.* files ?? But can you really explain what is the purpose of this test ? To prepare a 2.5 upgrade with a manual pre-cleaning of MDT, and a possible downgrade due to problems ? If this is the case, just do the upgrade and then run lfsck, it will do the cleanup for you and better than all of these manual and risky actions, you can trust me!</p> <p>To answer some of your numerous questions: if you remove all OI.16.* files they will be recreated upon MDT start/mount, removing of OBJECTS/* and CATALOG file should work even if there may be a few possible side effects/msgs (like orphaned objects on OSTs, ...), if you want to record all infos to be able to manually clean orphans on OSTs upon an MDT restore best choice is RobinHood, "offline" lfsck version available in 2.1 can also do it for you, "online" lfsck version available in 2.5 will do it much better and easier!</p> <p>Hi Bruno,</p> <p>Yes you understood correctly and we were just trying to see if there is ant risk involved in the upcoming maintenance process, so we decided to postpone the removing the files from the OBJECTS directory and will work on that part with Lustre 2.5.x releases.</p> <p>Yes we hit with the bug DDN-77 and to fix that issue we have to go through this process and right now customer is still testing Lustre v2.5 and want's to keep on hold for now to get on v2.5 so till we have to go through this process of fixing the OI.16 file under Lustre v2.1.6.</p> <p>Thank You,<br/> Manish</p> <p>Hello Manish, sorry to be late on this.<br/> Not sure to fully understand your last comment, but according to my previous indications and DDN-77 content, I am definitely convinced that any OBJECTS/* and OI files removal should only occur post v2.5 upgrade and after a full filesystem stop/umount from Clients and Servers.</p> <p>Hi Bruno,</p> <p>Well we have already did e.g. "mv OBJECTS OBJECTS.DDN-191; mv CATALOGS CATALOGS.DDN-191;" to that directory when we encountered some issues with hitting Directory index limit with DDN-191 ticket. Currently Customer is still on Lustre v2.1.6.</p> <p>Since we already renamed the original OBJECTS directory that contains 487G of llogs. Is it safe to delete the old OBJECTS directory to reclaim space during our next maintenance window?</p> <p>Why logs continue to accumulate in the OBJECTS directory and is there something we can do to stop it.</p> <p>Thank You,<br/> Manish</p> <p>Hi Bruno, Andreas,</p> <p>Any updates/comments on my previous questions? Other maintenance schedule is heading closer and we have to plan accordingly for that maintenance schedule, so please suggest on this issue.</p> <p>Thank You,<br/> Manish</p> <p>Hello Manish, sorry to be late on this.<br/> I definitely need more infos to determine what's going on at your site causing the new plain LLOGs to accumulate under OBJECTS directory :<br/> _ "lctl dl" cmd output on MDS<br/> _ "ls -li" of MDS's /OBJECTS directory output, when ldiskfs-mounted<br/> _ "llog_reader CATALOGS" cmd output, when ldiskfs-mounted<br/> _ binary copy of CATALOGS file</p> <p>And yes, as far as you have indicated that you are already running fine with moved OBJECT/CATALOGS directory/file, it should be fine to remove the old/renamed OBJECTS directory, but I would suggest you do this during dedicated time and all Clients/Servers unmounted.</p> <p>Hi Bruno,</p> <p>Customer tried to run that command on their test cluster with backup mdt data and got error as below:</p> <p>dc-oss01# llog_reader CATALOGS<br/> Memory Alloc for recs_buf error.<br/> Could not pack buffer; rc=-12</p> <p>Do you know any alternative workaround to get those data output and their test cluster has 48G of RAM and production cluster has 128G.</p> <p>Thank You,<br/> Manish</p> <p>Hello Manish,<br/> I don't think this error is related to some real memory lack, but more likely be the consequence of wrong parsing of CATALOGS file format in llog_reader where following code :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* the llog header not countable here.*/ recs_num = le32_to_cpu((*llog)-&gt;llh_count)-1; recs_buf = malloc(recs_num * sizeof(struct llog_rec_hdr *)); </pre> </div></div> <p>can easily return &#45;ENOMEM if (*llog)-&gt;llh_count value was mistakenly be read as a 0.</p> <p>Can you attach the "binary copy of CATALOGS file" I already/also have asked you ? And also the other infos I had requested too ??</p> <p>Also have you been able to remove the old/moved OBJECTS directory content/files ??</p> <p>Hi Bruno,</p> <p>Can you be more clear about the "binary copy of CATALOGS file". Is that you we just need to mount the MDT with ldiskfs and use normal cp command to copy that CATALOGS file and send it to you or there is some different process involved.</p> <p>Thank You,<br/> Manish</p> <p>Correct, just a raw copy of CATALOGS file.</p> <p>Here is the CATALOGS file from MDT.</p> <p>Hi Bruno,</p> <p>Attached is a copy of the current CATALOGS file.</p> <p><a href="https://jira.hpdd.intel.com/secure/attachment/18394/CATALOGS.20150707" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/secure/attachment/18394/CATALOGS.20150707</a></p> <p>The llog_reader command failed with the following error:</p> <p>Memory Alloc for recs_buf error.<br/> Could not pack buffer; rc=-12</p> <p>So please advice how to get that output and what needs to be done to get rid of that "recs_buf error"</p> <p>We removed the OLD CATALOGS and OBJECTS directories, which we renamed about a month ago. Today customer site has scheduled for maintenance so it would be nice to have next step befre we run out of this maintenance scheulde.</p> <p>Thank You,<br/> Manish</p> <p>Thanks for the CATALOGS binary file, but what about the :<br/> _ "lctl dl" cmd output on MDS<br/> _ "ls -li" of MDS's /OBJECTS directory output, when ldiskfs-mounted<br/> other infos I already asked you to also provide ??<br/> These infos are also required to understand what is going on.</p> <p>Concerning the "recs_buf error" from "llog_reader CATALOGS" cmd, they should come from the fact "llog_reader" don't know know howto interpret CATALOGS formet, but I can manage to get interesting stuff from CATALOGS binary file.</p> <p>Bruno requested rest the data.</p> <p>Hi Bruno,</p> <p>Here are the other requested files.</p> <p><a href="https://jira.hpdd.intel.com/secure/attachment/18414/ddn_sr44330_lctl_dl.txt" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/secure/attachment/18414/ddn_sr44330_lctl_dl.txt</a><br/> <a href="https://jira.hpdd.intel.com/secure/attachment/18415/ddn_sr44330_ls_objects.txt" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/secure/attachment/18415/ddn_sr44330_ls_objects.txt</a><br/> <a href="https://jira.hpdd.intel.com/secure/attachment/18416/llog_reader.txt" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/secure/attachment/18416/llog_reader.txt</a></p> <p>Thank You,<br/> Manish</p> <p>Hello Manish,<br/> According to your latest provided files/infos, here is what I think :<br/> _ the llog_reader.txt file, shows the the oi.16 file as being huge, and as we already discussed before, how to cure this has been already described to you in DDN-77.</p> <p> _ the llog_reader.txt file, also again highlights the llog_reader error when used on the CATALOG file that is not of expected LLOG format. I will push a patch soon to have llog_reader gracefully handle such case.</p> <p> _ the ddn_sr44330_ls_objects.txt file shows that there are 336 "100uvw:68b06xyz" small files (2x per OST and half of them as the only referenced in CATALOG file), in OBJECTS/ dir. All are dated "Jun 9 10:56". These should be the normal OSP sync LLOGs catalogs.</p> <p> _ the ddn_sr44330_ls_objects.txt file also shows 190 more recent and others files of reasonable size (all are &lt; 3MBs). Are these files te one you suspect to leak and to have consumed a huge space in your MDT before you have removed them ?</p> <p> _ last, do you still see the "(llog_obd.c:320:cat_cancel_cb())<br/> Cannot find handle for log 0x2793959b: -116" original messages for this ticket ?</p> <p>Hi Bruno,</p> <p>We have already cured the hug oi.16 file issues and it's shrinked to normal size. Regarding rest of the logs files related questiosn I will have to check with customer to get their input and I will update this threads.</p> <p>Thank You,<br/> Manish</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15654" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15654</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6612" title="(llog_obd.c:346:cat_cancel_cb()) cancel log + llog_obd.c:315:cat_cancel_cb()) processing log + llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-6612"><del>LU-6612</del></a> utils: strengthen llog_reader vs wrong format/header<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bdc32b6164c08484b2f88b1a3a0bac7b9f223dea</p> <p>Hi Bruno,</p> <p>It seems that any file over a few hours old in the OBJECTS directory is leaked. It would be nice if you can take a look at the ls -lah output from the MDT OBJECTS directory in the attached file "find_exec_ls_alh_manish.txt file" to this case for a more complete history of the number of files, and their timestamp and size. The file lists most of the nearly 74k files that were removed during our last maintenance window.</p> <p><a href="https://jira.hpdd.intel.com/secure/attachment/18474/find_exec_ls_alh_manish.txt" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/secure/attachment/18474/find_exec_ls_alh_manish.txt</a></p> <p>The most recent llog_obd.c:320:cat_cancel_cb() error logged was on 7/9 for a different log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jul 8 00:06:11 dc2mds01 kernel: : LustreError: 8949:0:(llog_obd.c:320:cat_cancel_cb()) Cannot find handle for log 0x27924bbc: -116 Jul 8 00:06:11 dc2mds01 kernel: : LustreError: 8949:0:(llog_obd.c:320:cat_cancel_cb()) Skipped 26419 previous similar messages </pre> </div></div> <p>Thank You,<br/> Manish</p> <p>Hi Bruno,</p> <p>Customer is scheduled to have maintenance window on upcoming next Tuesday, so can you take a look at that file which I listed in my previous post and see if you need anything else from the customer to be collected during the maintenance downtime.</p> <p>Thank You,<br/> Manish </p> <p>Hello Manish,<br/> In the last find_exec_ls_alh_manish.txt you have provided, I don't see any files dated for the last 2 June/July monthes ...<br/> Also, it looks much more different than the previous ddn_sr44330_ls_objects.txt content I have already described!<br/> Can you detail how/when these 2x "ls" cmd reports have been done ?</p> <p>Hi Bruno,</p> <p>Customer took the MDT backup and loaded the MDT data on their test cluster and mounted with ldiskfs and those data are captured on May-12-2015. The command used to get the file data "find_exec_ls_alh_manish.txt " is as below:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>find OBJECTS -name "[0-9][0-9]*:[0-9a-f][0-9a-f]*" -exec ls -lah {} \; </pre> </div></div> <p>Let me know if you need anything else to troubleshoot further.</p> <p>Thank You,<br/> Manish</p> <p>Humm I was also asking for some feedback about the differences I have found in previous ddn_sr44330_ls_objects.txt file/infos ...<br/> BTW, if OBJECTS files still appear to leak on customer site, would be cool if you can attach some (old + recent) of them. And with a recent version of the CATALOGS file too.</p> <p>Hi Bruno,</p> <p>Customer collected a listing of the OBJECTS directory, a copy of the CATALOGS file, and a tarball of OBJECTS directory. Here is new data which you requested.</p> <p><a href="http://ddntsr.com/ftp/2015-08-04-SR44330_20150804.tar" class="external-link" target="_blank" rel="nofollow noopener">http://ddntsr.com/ftp/2015-08-04-SR44330_20150804.tar</a></p> <p>Let us know if you need anything else.</p> <p>Thank You,<br/> Manish</p> <p>Hi Bruno,</p> <p>Any updates on the data which I uploaded in the previous post. </p> <p>Thank You,<br/> Manish</p> <p>Manish,<br/> Sorry to be late on this, but unfortunately the <a href="http://ddntsr.com/ftp/2015-08-04-SR44330_20150804.tar" class="external-link" target="_blank" rel="nofollow noopener">http://ddntsr.com/ftp/2015-08-04-SR44330_20150804.tar</a> returns me "ERROR 404: Not Found." when I try to download it. Can you verify what is going on? Thanks in advance.</p> <p>Hi Bruno,</p> <p>I have uploaded that logs to Intel FTP site with directory name to this Jira Case <a href="https://jira.whamcloud.com/browse/LU-6612" title="(llog_obd.c:346:cat_cancel_cb()) cancel log + llog_obd.c:315:cat_cancel_cb()) processing log + llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-6612"><del>LU-6612</del></a>. Let me know if you are able to get those logs from there.</p> <p>Thank You,<br/> Manish</p> <p>Adding recent email/answer sent to concerned people inclusing Manish :<br/> =======================================================<br/> I have started to work on the latest infos from the site that Manish has provided.<br/> I am not fully done, because LLOGs analysis is quite a long process, but my 1st results seem to indicate that the files growing/leaking in OBJECTS directory are OSP Sync logs. If further analysis of all LLOGs confirms this, that means that either some of the osp_sync_thread’s are stuck or that some chown &amp; unlink operations on MDT are never commited by some OSTs, or last, that some OSTs are stopped.<br/> <a href="https://jira.whamcloud.com/browse/LU-5297" title="osp_sync_thread can&#39;t handle invalid record gracefully" class="issue-link" data-issue-key="LU-5297"><del>LU-5297</del></a>, <a href="https://jira.whamcloud.com/browse/LU-6687" title="ALL osp-sync in D state" class="issue-link" data-issue-key="LU-6687"><del>LU-6687</del></a> and <a href="https://jira.whamcloud.com/browse/LU-7079" title="OSP shouldn&#39;t discard requests due to imp_peer_committed_transno" class="issue-link" data-issue-key="LU-7079"><del>LU-7079</del></a> may be related.</p> <p>Since Manish is in copy, I am using this email to ask him more infos :<br/> _ does the site have 168 OSTs configured?<br/> _ do you again confirm that none of the OSTs are stopped?<br/> _ can you check osp_sync_thread’s state on MDS?<br/> _ since how many times now the concerned Lustre filesystem has been started? Would be nice to get the exact date.<br/> _ do you confirm site is running with 2.1.6?</p> <p>Thanks again for your help,<br/> ============================================================</p> <p>I have done more work on latest provided datas, and here are the results :<br/> _ CATALOGS file points to 168 Catalog files, presumably one for each OST/osp_sync log<br/> _ each of these catalogs only contain 5-7 valid+sparse records pointing to plain LLOG files.<br/> _ number of Catalog files (168) + number of all of their referenced LLOG files = number of files in OBJECTS dir<br/> _ I have not checked all of the LLOG referenced files, but all I had a look to only contain a few+sparsed records of MDS_UNLINK_REC type.</p> <p>So <a href="https://jira.whamcloud.com/browse/LU-7079" title="OSP shouldn&#39;t discard requests due to imp_peer_committed_transno" class="issue-link" data-issue-key="LU-7079"><del>LU-7079</del></a> seems to look as the best choice/candidate, but I need to verify if it also applies to 2.1.6.</p> <p>Hi Bruno,</p> <p>-&gt; Yes this site is having 168 OST's configured.<br/> -&gt; Yes none of the OST's are stopped.<br/> -&gt; yes site is running with 2.1.6</p> <p>I will have to get the rest of the data from the customer and post it once I get those details. So how do I check the "osp_sync_thread" state on MDS", is there any command which I can use it.</p> <p>Thank You,<br/> Manish</p> <p>Bruno</p> <p><a href="https://jira.whamcloud.com/browse/LU-7079" title="OSP shouldn&#39;t discard requests due to imp_peer_committed_transno" class="issue-link" data-issue-key="LU-7079"><del>LU-7079</del></a> was only introduced with Lustre 2.4.x so that cannot be the exact issue at play on this 2.1.6 deployment so I think that discounts the request to check osp_sync_thread’s state on the MDS. Are the other items still required? Anything else that DDN should gather to help support your ongoing investigation?</p> <p>Peter</p> <p>Since it is established that our 2.1 problem could not be related/linked to <a href="https://jira.whamcloud.com/browse/LU-7079" title="OSP shouldn&#39;t discard requests due to imp_peer_committed_transno" class="issue-link" data-issue-key="LU-7079"><del>LU-7079</del></a>, and also that my previous request to check "osp_sync_thread state on MDS" is inacurate.<br/> Would it be possible for you to provide MDS and OSSs syslogs/Console history for the past 2/3 months time period ?<br/> On the other hand I am parsing the 2.1.6 Lustre source code to find any code-path that may explain such leakage.</p> <p>Hi Bruno,</p> <p>I have uploaded the logs to intel FTP under <a href="https://jira.whamcloud.com/browse/LU-6612" title="(llog_obd.c:346:cat_cancel_cb()) cancel log + llog_obd.c:315:cat_cancel_cb()) processing log + llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-6612"><del>LU-6612</del></a> dir with file name as "2015-09-03-SR44330_es_lustre_showall_2015-09-03_151236.tar.bz2"</p> <p>File system has be restarted so many times and it's gone be hard to get all those data but here is recent activity.</p> <p>06-03-2015 : File System is taken down for oi.16 shrink procedure and restarted during scheduled maintenance window.<br/> 06-07-2015 - 06-09-2015 : The File System MDS experienced issues which required the MDT to be unmounted multiple times. During this unplanned outage is when Andreas instructed us to renamed the OBJECTS directory on the MDT.<br/> 07-07-2015 : File System was restarted during scheduled maintenance window<br/> 08-04-2015 : File System was taken offline during regular scheduled maintenance window to collect MDT OBJECTS and CATALOGS files information for Intel.<br/> 09-02-2015 : File System was restarted to accommodate a MDT backup.</p> <p>Thank You,<br/> Manish</p> <p>Thanks to provide the logs.<br/> Looks like OSSs/OSTs quite frequently report the following errors/msgs sequence :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>messages:Sep 2 04:05:50 dc2oss12 kernel: : LustreError: 14888:0:(ldlm_resource.c:1090:ldlm_resource_get()) lvbo_init failed for resource 110829994: rc -2 messages:Sep 2 04:05:50 dc2oss12 kernel: : LustreError: 7212:0:(filter.c:3136:__filter_oa2dentry()) dc2-OST0075: filter_sync on non-existent object: 110829994:0 messages:Sep 2 04:05:50 dc2oss12 kernel: : LustreError: 7212:0:(ost_handler.c:1648:ost_blocking_ast()) Error -2 syncing data on lock cancel </pre> </div></div> <p>I am presently analyzing the related code path to see if it can be the cause of the leaking MDS_UNLINK_REC records in the OSP-SYNC LLOGs.</p> <p>There also the following error msgs from filter_destroy() which could be much more related :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ grep filter_destroy es_lustre_showall_2015-09-03_151236/*oss*/logs/messages-* es_lustre_showall_2015-09-03_151236/dc2oss01/logs/messages-20150816:Aug 14 03:23:13 dc2oss01 kernel: : [&lt;ffffffffa0ca5a24&gt;] filter_destroy+0xf24/0x13d0 [obdfilter] es_lustre_showall_2015-09-03_151236/dc2oss01/logs/messages-20150816:Aug 14 03:24:56 dc2oss01 kernel: : LustreError: 15252:0:(filter.c:4267:filter_destroy()) error on commit, err = -30 es_lustre_showall_2015-09-03_151236/dc2oss01/logs/messages-20150816:Aug 14 03:24:56 dc2oss01 kernel: : LustreError: 15252:0:(filter.c:4267:filter_destroy()) Skipped 1 previous similar message es_lustre_showall_2015-09-03_151236/dc2oss01/logs/messages-20150816:Aug 14 08:19:45 dc2oss01 kernel: : LustreError: 16201:0:(filter.c:4151:filter_destroy()) dc2-OST0006: can not find olg of group 0 es_lustre_showall_2015-09-03_151236/dc2oss01/logs/messages-20150816:Aug 14 08:19:45 dc2oss01 kernel: : LustreError: 16201:0:(filter.c:4151:filter_destroy()) dc2-OST0006: can not find olg of group 0 es_lustre_showall_2015-09-03_151236/dc2oss01/logs/messages-20150816:Aug 14 08:19:57 dc2oss01 kernel: : LustreError: 16201:0:(filter.c:4151:filter_destroy()) dc2-OST0006: can not find olg of group 0 es_lustre_showall_2015-09-03_151236/dc2oss01/logs/messages-20150816:Aug 14 08:19:57 dc2oss01 kernel: : LustreError: 16201:0:(filter.c:4151:filter_destroy()) Skipped 7 previous similar messages es_lustre_showall_2015-09-03_151236/dc2oss01/logs/messages-20150816:Aug 14 08:19:59 dc2oss01 kernel: : LustreError: 16189:0:(filter.c:4151:filter_destroy()) dc2-OST0006: can not find olg of group 0 es_lustre_showall_2015-09-03_151236/dc2oss01/logs/messages-20150816:Aug 14 08:19:59 dc2oss01 kernel: : LustreError: 16189:0:(filter.c:4151:filter_destroy()) Skipped 9 previous similar messages es_lustre_showall_2015-09-03_151236/dc2oss01/logs/messages-20150816:Aug 14 08:20:01 dc2oss01 kernel: : LustreError: 16189:0:(filter.c:4151:filter_destroy()) dc2-OST0006: can not find olg of group 0 es_lustre_showall_2015-09-03_151236/dc2oss02/logs/messages-20150816:Aug 15 10:56:03 dc2oss02 kernel: : [&lt;ffffffffa0c2f218&gt;] ? filter_destroy+0x718/0x13d0 [obdfilter] es_lustre_showall_2015-09-03_151236/dc2oss08/logs/messages-20150823:Aug 18 06:29:24 dc2oss08 kernel: : [&lt;ffffffffa0cb2218&gt;] filter_destroy+0x718/0x13d0 [obdfilter] es_lustre_showall_2015-09-03_151236/dc2oss10/logs/messages-20150809:Aug 5 14:31:51 dc2oss10 kernel: : LustreError: 15496:0:(filter.c:1621:filter_destroy_internal()) destroying objid 94281010 ino 36943382 nlink 1 count 3 </pre> </div></div> <p>and could be the start-point of MDS_UNLINK_REC records and containing LLOG plain file leakage.</p> <p>Since problem seems to occur on any OST, I have decided to concentrate on OST0000, and this OST's Catalog presently only contains 5 references/records of plain LLOGs :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Header size : 8192 Time : Tue Jun 9 07:48:06 2015 Number of records: 5 Target uuid : ----------------------- #01 (064)ogen=68B06AFD name=0:1048793 #102 (064)ogen=8E05AC60 name=0:1049057 #103 (064)ogen=B52DE2C3 name=0:1049144 #104 (064)ogen=EEE3D5B name=0:1049420 #193 (064)ogen=B9CE5F6A name=0:1049449 </pre> </div></div> <p>if we use the 1st one, it contains (like the others) a few valid records :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rec #1 type=10612404 len=40 &lt;&lt;&lt;&lt;&lt; mainly MDS_UNLINK_REC records rec #2 type=10612404 len=40 rec #3 type=10612404 len=40 rec #4 type=10612404 len=40 rec #5 type=10612404 len=40 rec #6 type=10612404 len=40 rec #7 type=10612404 len=40 rec #8 type=10612404 len=40 rec #9 type=10612404 len=40 rec #10 type=10612404 len=40 rec #11 type=10612404 len=40 rec #12 type=10612404 len=40 rec #13 type=10612404 len=40 rec #14 type=10612404 len=40 rec #15 type=10612404 len=40 rec #19 type=10612404 len=40 rec #20 type=10612404 len=40 rec #21 type=10612404 len=40 rec #24 type=10612404 len=40 rec #27 type=10612404 len=40 rec #28 type=10612404 len=40 rec #29 type=10612404 len=40 rec #30 type=10612404 len=40 rec #32 type=10612404 len=40 rec #33 type=10612404 len=40 rec #37 type=10612404 len=40 rec #38 type=10612404 len=40 rec #39 type=10612404 len=40 rec #42 type=10612404 len=40 rec #43 type=10612404 len=40 rec #44 type=10612404 len=40 rec #45 type=10612404 len=40 rec #46 type=10612404 len=40 rec #47 type=10612404 len=40 rec #52 type=10612404 len=40 rec #53 type=10612404 len=40 rec #54 type=10612404 len=40 rec #55 type=10612404 len=40 rec #56 type=10612404 len=40 rec #57 type=10612404 len=40 rec #58 type=10692401 len=56 &lt;&lt;&lt;&lt;&lt; but also a MDS_SETATTR64_REC rec #59 type=10612404 len=40 rec #60 type=10612404 len=40 rec #61 type=10612404 len=40 rec #62 type=10612404 len=40 rec #63 type=10612404 len=40 rec #64 type=10612404 len=40 rec #65 type=10612404 len=40 rec #66 type=10612404 len=40 rec #67 type=10612404 len=40 rec #68 type=10612404 len=40 rec #69 type=10612404 len=40 rec #70 type=10612404 len=40 rec #71 type=10612404 len=40 rec #72 type=10612404 len=40 rec #73 type=10612404 len=40 rec #74 type=10612404 len=40 rec #75 type=10612404 len=40 rec #76 type=10612404 len=40 rec #77 type=10612404 len=40 rec #78 type=10612404 len=40 rec #79 type=10612404 len=40 rec #80 type=10612404 len=40 rec #81 type=10612404 len=40 rec #82 type=10612404 len=40 rec #83 type=10612404 len=40 rec #84 type=10612404 len=40 rec #85 type=10612404 len=40 rec #86 type=10612404 len=40 rec #87 type=10612404 len=40 rec #88 type=10612404 len=40 rec #89 type=10612404 len=40 </pre> </div></div> <p>Thus, could you check if OBJID 0x532cb79 (87214969) still exists on OST0000 and also if OBJID 0x532ccc6 (87215302) still exists on OST0000, and the later with 0xb8517 (754967) uid and 0xd2 (210) gid ?<br/> To do so, you can use debugfs on associated device and look for OBJIDs with the following commands :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>debugfs -R "stat O/0/d25/87214969" /dev/... debugfs -R "stat O/0/d6/87215302" /dev/... </pre> </div></div> <p>Would be also very helpful if you could again provide a similar package containing MDS's CATALOGS and OBJECTS/* files, for me to understand problem's evolution/consequences based on the recently provided logs content.</p> <p>Hi Bruno,</p> <p>Can you provide more details on what exactly you need for </p> <p>"Would be also very helpful if you could again provide a similar package containing MDS's CATALOGS and OBJECTS/* files, for me to understand problem's evolution/consequences based on the recently provided logs content."</p> <p>Also do you need any out with 0xb8517 (754967) uid and 0xd2 (210) gid, then let us know the command to gather those data.</p> <p>I will ask the customer to get the debugfs output and update the post.</p> <p>Thank You,<br/> Manish</p> <p>&gt; Can you provide more details on what exactly you need for<br/> Exactly the same stuff/infos than what you have provided and commented in this ticket on 04/Aug/15, but from now.</p> <p>&gt; Also do you need any out with 0xb8517 (754967) uid and 0xd2 (210) gid, then let us know the command to gather those data.<br/> No that will be part of the infos returned for OBJID 0x532ccc6 (87215302).</p> <p>Hi Bruno,</p> <p>Here is the debugfs output which you requested and rest of other data we can get during next maintenance window happening in October.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@dc2oss01 ~]# debugfs -R "stat O/0/d25/87214969" /dev/mapper/ost_dc2_0 2&gt;&amp;1 | tee /tmp/sr44330_87214969.log debugfs 1.42.7.wc1 (12-Apr-2013) O/0/d25/87214969: File not found by ext2_lookup </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@dc2oss01 ~]# debugfs -R "stat O/0/d6/87215302" /dev/mapper/ost_dc2_0 2&gt;&amp;1 | tee /tmp/sr44330_87215302.log debugfs 1.42.7.wc1 (12-Apr-2013) Inode: 94402427 Type: regular Mode: 0666 Flags: 0x80000 Generation: 1758127277 Version: 0x00000000:21734617 User: 754967 Group: 210 Size: 2914894 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 5696 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x55770077:00000000 -- Tue Jun 9 11:04:23 2015 atime: 0x55770086:00000000 -- Tue Jun 9 11:04:38 2015 mtime: 0x55770077:00000000 -- Tue Jun 9 11:04:23 2015 crtime: 0x5577003b:611b92d8 -- Tue Jun 9 11:03:23 2015 Size of extra inode fields: 28 Extended attributes stored in inode body: fid = "fe 0d 00 5b 76 f2 10 02 00 00 00 00 00 00 00 00 c6 cc 32 05 00 00 00 00 00 00 00 00 00 00 00 00 " (32) fid: objid=87215302 seq=0 parent=[0x210f2765b000dfe:0x0:0x0] stripe=0 EXTENTS: (0-255):2009370624-2009370879, (256-711):2009379072-2009379527 </pre> </div></div> <p>Thank You,<br/> Manish</p> <p>Thanks Manish,<br/> This is really interesting to see that for 2 Unlink and Setattr leaked records, respectivelly the 1st is no longer present (but may have been simply unlinked as expected!), and the 2nd is present and has its uid/gid set as expected with the values in its LLOG record.<br/> Due to these last infos, I have worked more looking into the 2.1.6 source code and I think I have a good idea about how LLOG records may leak now. Based on this, I will now try to setup a reproducer along to push a debug patch.<br/> More to come soon now.</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16373" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16373</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6612" title="(llog_obd.c:346:cat_cancel_cb()) cancel log + llog_obd.c:315:cat_cancel_cb()) processing log + llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-6612"><del>LU-6612</del></a> obdfilter: more debug upon filter_find_olg() failure<br/> Project: fs/lustre-release<br/> Branch: b2_1<br/> Current Patch Set: 1<br/> Commit: 28b1448ac3f757c7df42cd15208cd36e4b20d4eb</p> <p>Hi Bruno,</p> <p>So is that patch is with the fix or it's just the debug patch.</p> <p>Thank You,<br/> Manish</p> <p>Manish</p> <p>Based on the comment above and the commit message it seems that this is a debug patch</p> <p>Peter</p> <p>Hi Bruno,</p> <p>Were you able to create a reproducer and found anything?</p> <p>Thank You,<br/> Manish</p> <p>Hello Manish,<br/> Peter is right, my patch is only to give more infos on the code paths where I suspect we can leak LLOG records finally causing the orphan LLOG files accumulating under OBJECTS directory.<br/> I am presently working to reproduce running with my patch, will let you know soon now about any progress/success.</p> <p>Unfortunately I am presently unable to reproduce even if trying to create scenarios likely to generate the previously suspected msgs ...<br/> On the other hand working on the reproducer has permitted me to find other paths in code where there seems to be the possibility of records leakage in MDS&lt;-&gt;OSTs SYNC LLOGs. Recovery code may also be concerned because the MDS should be able to ask OSTs to replay such recorded operations upon restart ...<br/> Due to this I am presently writing an enhanced version of my previous debug patch, if its further testing will be successful, do you think there is a chance that it could be installed on-site? MDS/OSSs are targeted.</p> <p>Also about the 2.1.6 version you have claimed for this problem, the logs yopu have provided state "Lustre: Build Version: EXAScaler-ddn1.0--PRISTINE-2.6.32-358.11.1.el6_lustre.es359.devel.x86_64", can you give me more details about the exact content (Tag it is based on, additional patches, ...) ?</p> <p>And again, as I have already requested, in my 2 previous updates on 7+9/Sep/15, can you provide « Exactly the same stuff/infos than what you have provided and commented in this ticket on 04/Aug/15, but from now » ???</p> <p>Hi Bruno,</p> <p>Yes if you can provide that new debug patch in few days then we can build new RPMS and test it and install it at next maintenance window, which is most likely to be happening on October/06 and at same time we can collect those previous set of data which was provided on 04/Aug/15.</p> <p>Here is more details about this Lustre Build Patches applied to that Build.</p> <p><a href="https://jira.hpdd.intel.com/secure/attachment/18930/IU_Lustre_Build_Patches_06_08_2015.txt" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/secure/attachment/18930/IU_Lustre_Build_Patches_06_08_2015.txt</a></p> <p>Thank You,<br/> Manish</p> <p>Hi Bruno,</p> <p>Let us know if you need anything else other than below listed data, </p> <p>a listing of the OBJECTS directory, a copy of the CATALOGS file, and a tarball of OBJECTS directory.</p> <p>Customer is having a downtime tomorrow and if you have that enhanced version of patch ready then let us know.</p> <p>Thank You,<br/> Manish </p> <p>Hello Manish,<br/> &gt; a listing of the OBJECTS directory, a copy of the CATALOGS file, and a tarball of OBJECTS directory.<br/> Yes, that will be very helpful, but even more if you can get the same infos after filesystem has been fully stopped, and again after filesystem has been fully re-started (and any recovery is completed).</p> <p>If you could also ensure to run the whole restart with the default (ioctl neterror warning error emerg ha config console lfsck) debug mask set, on MDS, and then take a full trace at the end of restart/recovery, that would also be very helpful.</p> <p>All MDS/OSSs syslogs/messages taken sometimes after restart will also be required.</p> <p>Last, you can add/use my debug patch available at <a href="http://review.whamcloud.com/16373" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16373</a> for Lustre version to be used for the restart, it will print more debug msgs/infos on some identified paths in code that may cause the unlink/setattr LLOG records leakage.</p> <p>If customer agrees to set the MDS debug mask as I have requested (and also to grow the size of the log buffer), I need to better detail the procedure to apply it.<br/> To be sure that you will catch and log the whole restart and recovery process on the MDS node, you will need to first load the libcfs module by running the "modprobe libcfs" command and change/verify the debug log mask (echo "ioctl neterror warning error emerg ha config console lfsck" &gt;/proc/sys/lnet/debug) and buffer size (echo 8192 &gt;/proc/sys/lnet/debug_mb, for 8GB but you can use less if MDS memory size is small), before mounting the MDT.<br/> Then you will need to dump the debug log to a disk file using the "lctl dk /tmp/lustre_debug.log" command, right after the filesystem start has completed.</p> <p>Also, you can reset the mask and buffer log size, to their original values, after filesystem has been fully started.</p> <p>Hi Bruno,</p> <p>We need more clarification about this part. </p> <p>"Yes, that will be very helpful, but even more if you can get the same infos after filesystem has been fully stopped, and again after filesystem has been fully re-started (and any recovery is completed)."</p> <p>So do you mean that after full stopped we collect the data using ldiskfs and then get back file system online and once the recovery is completed we take down the file system again and mount it with ldiskfs and collect those same set of data? Or you have other way around to collect those data while file system is up and online using 'debugfs' or other method. So can you please clarify about this part and provide detail steps to get those data.</p> <p>Also we won't be able to deploy the patch today since it has to go through their test lab testing and probably it will be deployed at next maintenance window.</p> <p>Thank You,<br/> Manish</p> <p>You will not need to stop the filesystem again, you can safely mount the MDT as ldiskfs, but from the same node/MDS only, and this in parallel of normal operations, until you don't modify anything on it. You can also use debugfs if you want, but dump'ing all the OBJECTS/ dir files could be a long process.</p> <p>Ok, for the patch I will let you know if I modify/enhance it in between.</p> <p>Hi Bruno,</p> <p>Here are the logs which you requested.</p> <p><a href="http://ddntsr.com/ftp/2015-10-06-SR44330_20151006.tar" class="external-link" target="_blank" rel="nofollow noopener">http://ddntsr.com/ftp/2015-10-06-SR44330_20151006.tar</a></p> <p>It contains the following files:</p> <p>SR44330_ls-li_MDT_OBJECTS_20151006.txt<br/> SR44330_MDT_CATALOGS_201501006.txt<br/> SR44330_MDT_OBJECTS_20151006.tar.gz<br/> lustre_debug.20151006.log<br/> SR44330_ls-li_MDT_OBJECTS_20151006b.txt<br/> SR44330_MDT_CATALOGS_201501006b.txt<br/> SR44330_MDT_OBJECTS_20151006b.tar.gz</p> <p>The files with a suffix of *b.txt and *b.tar.gz were collected after customer brought the file system back online.</p> <p>Here are the syslogs if needed.<br/> <a href="http://ddntsr.com/ftp/2015-10-06-SR44330_es_lustre_showall_2015-10-06_125006.tar.bz2" class="external-link" target="_blank" rel="nofollow noopener">http://ddntsr.com/ftp/2015-10-06-SR44330_es_lustre_showall_2015-10-06_125006.tar.bz2</a></p> <p>Let us know if you need anything else.</p> <p>Thank You,<br/> Manish</p> <p>Thanks Manish, I am presently reviewing all of these new infos, will let you know asap about any progress.</p> <p>Sorry, it took me some time to parse all these new datas ...<br/> The MDS debug log taken upon FS restart, when it is expected for the Unlink/Setattr LLOG records to be replayed between MDS and each OSTs, is particularly interesting because it shows that the concerned LLOG files of each OSTs are parsed. More debug-level are missing to understand why, but their records are not canceled/removed, thus LLOG files are kept.<br/> Thus I have decided to again try to reproduce problem in-house using the following scenario :<br/> _ Client creates a file<br/> _ deactivate MDT/OST connexion<br/> _ Client remove the file<br/> _ stop MDT<br/> _ set full debug<br/> _ restart MDT<br/> And this has permitted me to find that even if each MDS/OST Unlink/Setattr LLOG records are parsed/replayed, they are effectivelly not/never canceled :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00002000:00000001:27.0:1445357157.777095:0:5009:0:(filter_log.c:266:filter_recov_log_mds_ost_cb()) Process entered 00002000:00000001:27.0:1445357157.777097:0:5009:0:(filter_log.c:166:filter_recov_log_unlink_cb()) Process entered 00002000:00000010:27.0:1445357157.777099:0:5009:0:(filter_log.c:169:filter_recov_log_unlink_cb()) slab-alloced '((oa))': 208 at ffff8808362f4cb0. 00002000:00000001:27.0:1445357157.777103:0:5009:0:(filter.c:4119:filter_destroy()) Process entered 00002000:00000001:27.0:1445357157.777105:0:5009:0:(filter_capa.c:127:filter_auth_capa()) Process entered 00002000:00000001:27.0:1445357157.777107:0:5009:0:(filter_capa.c:141:filter_auth_capa()) Process leaving (rc=0 : 0 : 0) 00002000:00000002:27.0:1445357157.777110:0:5009:0:(filter.c:4133:filter_destroy()) lustre-OST0000: filter_destroy(group=0,oid=2) 00002000:00000001:27.0:1445357157.777113:0:5009:0:(filter.c:1474:filter_fid2dentry()) Process entered 00002000:00000002:27.0:1445357157.777116:0:5009:0:(filter.c:1499:filter_fid2dentry()) looking up object O/d2/2 00002000:00000002:27.0:1445357157.777177:0:5009:0:(filter.c:1518:filter_fid2dentry()) got child objid 2: ffff880834151740, count = 1 00002000:00000001:27.0:1445357157.777180:0:5009:0:(filter.c:1522:filter_fid2dentry()) Process leaving (rc=18446612167547754304 : -131906161797312 : ffff880834151740) 00002000:00000002:27.0:1445357157.777183:0:5009:0:(filter.c:4142:filter_destroy()) destroying non-existent object 2:0 00002000:00000001:27.0:1445357157.777185:0:5009:0:(filter.c:4161:filter_destroy()) Process leaving via cleanup (rc=18446744073709551614 : -2 : 0xfffffffffffffffe) 00002000:00000002:27.0:1445357157.777187:0:5009:0:(filter.c:221:f_dput()) putting 2: ffff880834151740, count = 0 00002000:00000001:27.0:1445357157.777191:0:5009:0:(lustre_quota.h:653:lquota_adjust()) Process entered 00040000:00000001:27.0:1445357157.777193:0:5009:0:(quota_master.c:577:filter_quota_adjust()) Process entered 00040000:00000001:27.0:1445357157.777195:0:5009:0:(quota_master.c:580:filter_quota_adjust()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:27.0:1445357157.777197:0:5009:0:(lustre_quota.h:657:lquota_adjust()) Process leaving (rc=0 : 0 : 0) 00002000:00000010:27.0:1445357157.777198:0:5009:0:(filter_log.c:201:filter_recov_log_unlink_cb()) slab-freed '((oa))': 208 at ffff8808362f4cb0. 00002000:00000001:27.0:1445357157.777200:0:5009:0:(filter_log.c:203:filter_recov_log_unlink_cb()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:27.0:1445357157.777202:0:5009:0:(filter_log.c:313:filter_recov_log_mds_ost_cb()) Process leaving (rc=0 : 0 : 0) </pre> </div></div> <p>and according to the code in b2_1, this should occur by calling either llog_cancel() or filter_cancel_cookies_cb() directly or as a callback, but only if OBD_MD_FLCOOKIE is not set in obdo valid flags, and this stands for both the unlink and setattr records case :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> int filter_setattr_internal(struct obd_export *exp, struct dentry *dentry, struct obdo *oa, struct obd_trans_info *oti) { unsigned int orig_ids[MAXQUOTAS] = {0, 0}; struct llog_cookie *fcc = NULL; struct filter_obd *filter; int rc, err, sync = 0; .................... if (oa-&gt;o_valid &amp; OBD_MD_FLCOOKIE) { OBD_ALLOC(fcc, sizeof(*fcc)); if (fcc != NULL) *fcc = oa-&gt;o_lcookie; } ................... if (oa-&gt;o_valid &amp; OBD_MD_FLFLAGS) { rc = fsfilt_iocontrol(exp-&gt;exp_obd, dentry, FSFILT_IOC_SETFLAGS, (long)&amp;oa-&gt;o_flags); } else { rc = fsfilt_setattr(exp-&gt;exp_obd, dentry, handle, &amp;iattr, 1); if (fcc != NULL) /* set cancel cookie callback function */ sync = fsfilt_add_journal_cb(exp-&gt;exp_obd, 0, handle, filter_cancel_cookies_cb, &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; fcc); } ................... if (sync) { filter_cancel_cookies_cb(exp-&gt;exp_obd, 0, fcc, rc); fcc = NULL; } .......................... int filter_destroy(struct obd_export *exp, struct obdo *oa, struct lov_stripe_md *md, struct obd_trans_info *oti, struct obd_export *md_exp, void *capa) { unsigned int qcids[MAXQUOTAS] = {0, 0}; struct obd_device *obd; struct filter_obd *filter; struct dentry *dchild = NULL, *dparent = NULL; struct lustre_handle lockh = { 0 }; struct lvfs_run_ctxt saved; void *handle = NULL; struct llog_cookie *fcc = NULL; int rc, rc2, cleanup_phase = 0, sync = 0; struct iattr iattr; unsigned long now; ENTRY; rc = filter_auth_capa(exp, NULL, oa-&gt;o_seq, (struct lustre_capa *)capa, CAPA_OPC_OSS_DESTROY); if (rc) RETURN(rc); obd = exp-&gt;exp_obd; filter = &amp;obd-&gt;u.filter; push_ctxt(&amp;saved, &amp;obd-&gt;obd_lvfs_ctxt, NULL); cleanup_phase = 1; CDEBUG(D_INODE, "%s: filter_destroy(group="LPU64",oid=" LPU64")\n", obd-&gt;obd_name, oa-&gt;o_seq, oa-&gt;o_id); dchild = filter_fid2dentry(obd, NULL, oa-&gt;o_seq, oa-&gt;o_id); if (IS_ERR(dchild)) GOTO(cleanup, rc = PTR_ERR(dchild)); cleanup_phase = 2; if (dchild-&gt;d_inode == NULL) { CDEBUG(D_INODE, "destroying non-existent object "POSTID"\n", oa-&gt;o_id, oa-&gt;o_seq); /* If object already gone, cancel cookie right now */ if (oa-&gt;o_valid &amp; OBD_MD_FLCOOKIE) { struct llog_ctxt *ctxt; struct obd_llog_group *olg; olg = filter_find_olg(obd, oa-&gt;o_seq); if (!olg) { CERROR("%s: can not find olg of group %d " "for objid "LPU64"\n", obd-&gt;obd_name, (int)oa-&gt;o_seq, oa-&gt;o_id); GOTO(cleanup, rc = PTR_ERR(olg)); } fcc = &amp;oa-&gt;o_lcookie; ctxt = llog_group_get_ctxt(olg, fcc-&gt;lgc_subsys + 1); llog_cancel(ctxt, NULL, 1, fcc, 0); &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; llog_ctxt_put(ctxt); fcc = NULL; /* we didn't allocate fcc, don't free it */ } GOTO(cleanup, rc = -ENOENT); } rc = filter_prepare_destroy(obd, oa-&gt;o_id, oa-&gt;o_seq, &amp;lockh); if (rc) GOTO(cleanup, rc); /* Our MDC connection is established by the MDS to us */ if (oa-&gt;o_valid &amp; OBD_MD_FLCOOKIE) { OBD_ALLOC(fcc, sizeof(*fcc)); if (fcc != NULL) *fcc = oa-&gt;o_lcookie; &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; } /* we're gonna truncate it first in order to avoid possible deadlock: * P1 P2 * open trasaction open transaction * down(i_zombie) down(i_zombie) * restart transaction * (see BUG 4180) -bzzz * * take i_alloc_sem too to prevent other threads from writing to the * file while we are truncating it. This can cause lock ordering issue * between page lock, i_mutex &amp; starting new journal handle. * (see bug 20321) -johann */ now = jiffies; down_write(&amp;dchild-&gt;d_inode-&gt;i_alloc_sem); LOCK_INODE_MUTEX(dchild-&gt;d_inode); fsfilt_check_slow(exp-&gt;exp_obd, now, "i_alloc_sem and i_mutex"); /* VBR: version recovery check */ rc = filter_version_get_check(exp, oti, dchild-&gt;d_inode); if (rc) { UNLOCK_INODE_MUTEX(dchild-&gt;d_inode); up_write(&amp;dchild-&gt;d_inode-&gt;i_alloc_sem); GOTO(cleanup, rc); } handle = fsfilt_start_log(obd, dchild-&gt;d_inode, FSFILT_OP_SETATTR, NULL, 1); if (IS_ERR(handle)) { UNLOCK_INODE_MUTEX(dchild-&gt;d_inode); up_write(&amp;dchild-&gt;d_inode-&gt;i_alloc_sem); GOTO(cleanup, rc = PTR_ERR(handle)); } /* Locking order: i_mutex -&gt; journal_lock -&gt; dqptr_sem. LU-952 */ ll_vfs_dq_init(dchild-&gt;d_inode); iattr.ia_valid = ATTR_SIZE; iattr.ia_size = 0; rc = fsfilt_setattr(obd, dchild, handle, &amp;iattr, 1); rc2 = fsfilt_commit(obd, dchild-&gt;d_inode, handle, 0); UNLOCK_INODE_MUTEX(dchild-&gt;d_inode); up_write(&amp;dchild-&gt;d_inode-&gt;i_alloc_sem); if (rc) GOTO(cleanup, rc); if (rc2) GOTO(cleanup, rc = rc2); /* We don't actually need to lock the parent until we are unlinking * here, and not while truncating above. That avoids holding the * parent lock for a long time during truncate, which can block other * threads from doing anything to objects in that directory. bug 7171 */ dparent = filter_parent_lock(obd, oa-&gt;o_seq, oa-&gt;o_id); if (IS_ERR(dparent)) GOTO(cleanup, rc = PTR_ERR(dparent)); cleanup_phase = 3; /* filter_parent_unlock */ LOCK_INODE_MUTEX(dchild-&gt;d_inode); handle = fsfilt_start_log(obd, dparent-&gt;d_inode,FSFILT_OP_UNLINK,oti,1); if (IS_ERR(handle)) { UNLOCK_INODE_MUTEX(dchild-&gt;d_inode); GOTO(cleanup, rc = PTR_ERR(handle)); } cleanup_phase = 4; /* fsfilt_commit */ /* Quota release need uid/gid of inode */ obdo_from_inode(oa, dchild-&gt;d_inode, NULL, OBD_MD_FLUID|OBD_MD_FLGID); filter_fmd_drop(exp, oa-&gt;o_id, oa-&gt;o_seq); /* this drops dchild-&gt;d_inode-&gt;i_mutex unconditionally */ rc = filter_destroy_internal(obd, oa-&gt;o_id, oa-&gt;o_seq, dparent, dchild); EXIT; cleanup: switch(cleanup_phase) { case 4: if (fcc != NULL) sync = fsfilt_add_journal_cb(obd, 0, oti ? oti-&gt;oti_handle : handle, filter_cancel_cookies_cb, &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; fcc); /* If add_journal_cb failed, then filter_finish_transno * will commit the handle and we will do a sync * on commit. then we call callback directly to free * the fcc. */ rc = filter_finish_transno(exp, NULL, oti, rc, sync); if (sync) { filter_cancel_cookies_cb(obd, 0, fcc, rc); &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; fcc = NULL; } rc2 = fsfilt_commit(obd, dparent-&gt;d_inode, handle, 0); if (rc2) { CERROR("error on commit, err = %d\n", rc2); if (!rc) rc = rc2; } else { fcc = NULL; } case 3: filter_parent_unlock(dparent); case 2: filter_fini_destroy(obd, &amp;lockh); f_dput(dchild); if (fcc != NULL) OBD_FREE(fcc, sizeof(*fcc)); case 1: pop_ctxt(&amp;saved, &amp;obd-&gt;obd_lvfs_ctxt, NULL); break; default: CERROR("invalid cleanup_phase %d\n", cleanup_phase); LBUG(); } /* trigger quota release */ qcids[USRQUOTA] = oa-&gt;o_uid; qcids[GRPQUOTA] = oa-&gt;o_gid; rc2 = lquota_adjust(filter_quota_interface_ref, obd, qcids, NULL, rc, FSFILT_OP_UNLINK); if (rc2) CERROR("filter adjust qunit! (rc:%d)\n", rc2); return rc; } </pre> </div></div> <p>then, having a look to the code of callers :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/* Callback for processing the unlink log record received from MDS by * llog_client_api. */ static int filter_recov_log_unlink_cb(struct llog_ctxt *ctxt, struct llog_rec_hdr *rec, struct llog_cookie *cookie) { struct obd_export *exp = ctxt-&gt;loc_obd-&gt;obd_self_export; struct llog_unlink_rec *lur; struct obdo *oa; obd_count count; int rc = 0; ENTRY; lur = (struct llog_unlink_rec *)rec; OBDO_ALLOC(oa); if (oa == NULL) RETURN(-ENOMEM); oa-&gt;o_valid |= OBD_MD_FLCOOKIE; &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; oa-&gt;o_id = lur-&gt;lur_oid; oa-&gt;o_seq = lur-&gt;lur_oseq; oa-&gt;o_valid = OBD_MD_FLID | OBD_MD_FLGROUP; &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; bad !!! oa-&gt;o_lcookie = *cookie; /* objid gap may require to destroy several objects in row */ count = lur-&gt;lur_count + 1; /* This check is only valid before FID-on-OST and it should * be removed after FID-on-OST is implemented */ if (oa-&gt;o_seq &gt; FID_SEQ_OST_MAX) { CERROR("%s: invalid group number "LPU64" &gt; MAX_CMD_GROUP %u\n", exp-&gt;exp_obd-&gt;obd_name, oa-&gt;o_seq, FID_SEQ_OST_MAX); RETURN(-EINVAL); } while (count &gt; 0) { rc = filter_destroy(exp, oa, NULL, NULL, NULL, NULL); if (rc == 0) CDEBUG(D_RPCTRACE, "object "LPU64" is destroyed\n", oa-&gt;o_id); else if (rc != -ENOENT) CEMERG("error destroying object "LPU64": %d\n", oa-&gt;o_id, rc); else rc = 0; count--; oa-&gt;o_id++; } OBDO_FREE(oa); RETURN(rc); } /* Callback for processing the setattr log record received from MDS by * llog_client_api. */ static int filter_recov_log_setattr_cb(struct llog_ctxt *ctxt, struct llog_rec_hdr *rec, struct llog_cookie *cookie) { struct obd_device *obd = ctxt-&gt;loc_obd; struct obd_export *exp = obd-&gt;obd_self_export; struct obd_info oinfo = { { { 0 } } }; obd_id oid; int rc = 0; ENTRY; OBDO_ALLOC(oinfo.oi_oa); if (oinfo.oi_oa == NULL) RETURN(-ENOMEM); if (rec-&gt;lrh_type == MDS_SETATTR_REC) { struct llog_setattr_rec *lsr = (struct llog_setattr_rec *)rec; oinfo.oi_oa-&gt;o_id = lsr-&gt;lsr_oid; oinfo.oi_oa-&gt;o_seq = lsr-&gt;lsr_oseq; oinfo.oi_oa-&gt;o_uid = lsr-&gt;lsr_uid; oinfo.oi_oa-&gt;o_gid = lsr-&gt;lsr_gid; } else { struct llog_setattr64_rec *lsr = (struct llog_setattr64_rec *)rec; oinfo.oi_oa-&gt;o_id = lsr-&gt;lsr_oid; oinfo.oi_oa-&gt;o_seq = lsr-&gt;lsr_oseq; oinfo.oi_oa-&gt;o_uid = lsr-&gt;lsr_uid; oinfo.oi_oa-&gt;o_gid = lsr-&gt;lsr_gid; } oinfo.oi_oa-&gt;o_valid |= (OBD_MD_FLID | OBD_MD_FLUID | OBD_MD_FLGID | OBD_MD_FLCOOKIE); &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; oinfo.oi_oa-&gt;o_valid = OBD_MD_FLID | OBD_MD_FLGROUP; &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; bad !! oinfo.oi_oa-&gt;o_lcookie = *cookie; oid = oinfo.oi_oa-&gt;o_id; rc = filter_setattr(exp, &amp;oinfo, NULL); OBDO_FREE(oinfo.oi_oa); if (rc == -ENOENT) { CDEBUG(D_RPCTRACE, "object already removed, send cookie\n"); llog_cancel(ctxt, NULL, 1, cookie, 0); RETURN(0); } if (rc == 0) CDEBUG(D_RPCTRACE, "object "LPU64" is chown/chgrp\n", oid); RETURN(rc); } </pre> </div></div> <p>Will push a b2_1 patch to fix this now, and see how it works.</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16906" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16906</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6612" title="(llog_obd.c:346:cat_cancel_cb()) cancel log + llog_obd.c:315:cat_cancel_cb()) processing log + llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-6612"><del>LU-6612</del></a> obdfilter: do not overwrite OBD_MD_FLCOOKIE flag<br/> Project: fs/lustre-release<br/> Branch: b2_1<br/> Current Patch Set: 1<br/> Commit: 1e04568d15e0c623faa8256d0583552a79331a9f</p> <p>Debug patch at <a href="http://review.whamcloud.com/16373" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16373</a> has been abandonned.</p> <p>Hi Bruno,</p> <p>I see that this patch <a href="http://review.whamcloud.com/16373" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16373</a> is abandoned and there is one more patch which is in progress <a href="http://review.whamcloud.com/#/c/16906/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16906/</a> so do you want us to apply any of those patches and need something from our end. </p> <p>Also were you able to reproduce this issues in your test lab and if so then did your Fix patch worked to fix those issues?</p> <p>Thank You,<br/> Manish</p> <p>Hello Manish,<br/> #16373 debug patch has been abandoned because bug has been found and real fix has been pushed as #16906.<br/> Yes, I have been able to reproduce issue/leak in lab, and no I have presently missed time to verify fix. But even if the bug was quite difficult to identify, the fix is definitely obvious.<br/> So it would be nice if you could give a try to patch #16906.</p> <p>This is quite old and no longer an active issue. Please close.</p> <p>ok - thanks Oz</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/15654/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/15654/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6612" title="(llog_obd.c:346:cat_cancel_cb()) cancel log + llog_obd.c:315:cat_cancel_cb()) processing log + llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-6612"><del>LU-6612</del></a> utils: strengthen llog_reader vs wrong format/header<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 45291b8c06eebf33d3654db3a7d3cfc5836004a6</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48309" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48309</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6612" title="(llog_obd.c:346:cat_cancel_cb()) cancel log + llog_obd.c:315:cat_cancel_cb()) processing log + llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-6612"><del>LU-6612</del></a> utils: strengthen llog_reader vs wrong format/header<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 7e3f7d1792eba8f3001a618de48397f05046e5f6</p> <p>"Jian Yu &lt;yujian@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48900" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48900</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6612" title="(llog_obd.c:346:cat_cancel_cb()) cancel log + llog_obd.c:315:cat_cancel_cb()) processing log + llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-6612"><del>LU-6612</del></a> utils: strengthen llog_reader vs wrong format/header<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 6e98a6e0e0c6bce20c5f3cb8fae1b0ffd9532efb</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48900/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48900/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6612" title="(llog_obd.c:346:cat_cancel_cb()) cancel log + llog_obd.c:315:cat_cancel_cb()) processing log + llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-6612"><del>LU-6612</del></a> utils: strengthen llog_reader vs wrong format/header<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: badba63a54e905129dbdf28e31026580453ea337</p> Gantt End to Start Related Development End date Tue, 1 Dec 2015 14:42:33 +0000 Rank 1|hzxdjj: Rank (Obsolete) 9223372036854775807 Severity Start date Mon, 18 May 2015 14:42:33 +0000