https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2440 Lustre <p>I have received notice of an issue a user at TACC is having:</p> <blockquote> <p>I am having a problem where my git repositories on TACC systems get corrupted, leading to errors like this:</p></blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>login4$ git status fatal: index file smaller than expected login4$ git status error: object file .git/objects/01/ee9f4bfe74aaee027a3e8418d70d337e1235d3 is empty fatal: loose object 01ee9f4bfe74aaee027a3e8418d70d337e1235d3 (stored in .git/objects/01/ee9f4bfe74aaee027a3e8418d70d337e1235d3) is corrupt login4$ git status error: object file .git/objects/8d/6083737dae5cb67906ac26702465ca2d70bc95 is empty fatal: loose object 8d6083737dae5cb67906ac26702465ca2d70bc95 (stored in .git/objects/8d/6083737dae5cb67906ac26702465ca2d70bc95) is corrupt login4$ git status error: object file .git/objects/bc/61c57143652fbf198de898ca7bb9d5659a5de0 is empty fatal: loose object bc61c57143652fbf198de898ca7bb9d5659a5de0 (stored in .git/objects/bc/61c57143652fbf198de898ca7bb9d5659a5de0) is corrupt </pre> </div></div> EL5 LU-2440

git repositories get corrupted

Story Major Resolved Cannot Reproduce Richard Henwood Richard Henwood Thu, 6 Dec 2012 16:04:56 +0000 Tue, 30 May 2017 05:21:40 +0000 Tue, 30 May 2017 05:21:40 +0000 Lustre 1.8.6 0 5 <p>If the git objects are zero length, there could be a number of potential causes. I had a similar problem on my local filesystem due to it filling up, and then Git proceeded to create and reference new objects that were empty.</p> <p>Is there possibly a quota limit in effect, some OST being full, other? Alternately, if git was in the middle of updating and the client crashed or was evicted before the data was written to disk something similar could also happen.</p> <p>Conversely, if there are errors on the console (dmesg) for that client when these objects are accessed that would clearly indicate some problem with Lustre.</p> <p>Is the problem reproducible (i.e. some sequence of steps during normal operation results in a broken repository)? At one time in the distant past, there was a metadata bug that CVS triggered, and we have a CVS test as the reproducer (sanity.sh test_99<span class="error">&#91;a-f&#93;</span>). If we can distill the steps that git takes to reproduce this problem (preferably starting with a new, empty archive to save space), that would definitely speed up understanding and fixing any problem found.</p> <p>It is probably worth mentioning that there is a nice long thread on the git mailing list about other issues of corruption and just general mis-behaviour of git on Lustre. At least some of it might be due to <a href="https://jira.whamcloud.com/browse/LU-2276" title="is open() idempotent in regards to being restarted after a signal interrupts it?" class="issue-link" data-issue-key="LU-2276"><del>LU-2276</del></a>, and yet some other complaints seem to sound like <a href="https://jira.whamcloud.com/browse/LU-305" title="utime() fails with EINTR : not conform to POSIX standard" class="issue-link" data-issue-key="LU-305"><del>LU-305</del></a> and some others unknown as of yet.</p> <p>Perhaps using git on lustre ought to be one of our regression tests.</p> <p>Thanks Brian, this is useful.</p> <p>I'm playing around at the moment, trying to create a reproducer.</p> <p>One of our users has what seems to be the exact same problem with git. We are running lustre clients 1.8.8, and 1.8.4 and 1.8.5 servers. The user is able to reproduce the problem every now and then by running a "git gc" in a crontab. The problem seems to appear once or twice per week.</p> <p>Thanks for this update. I have been able to reproduce this issue, leaving a repo overnight did it in my case.</p> <p>I'm following up to see if this is also an issue with Master, and trying to shorten the time to reproduce.</p> <p>Richard,<br/> "overnight" might just be the time it takes for idle DLM locks to be cancelled. What would be useful is:</p> <ul class="alternate" type="square"> <li>enable full debug logging, like <tt>lctl set_param debug=-1</tt></li> <li>do "git update" or "git gc" or whatever is the trigger</li> <li>dump debug logs, like <tt>lctl dk /tmp/git_update.log</tt></li> <li>verify repository is not corrupted <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/></li> <li>get checksums of all of the files under .git, like <tt>find .git -type f | xargs md5sum &gt; git_before.md5sum</tt></li> <li>cancel all of the DLM locks on the client, like <tt>lctl set_param ldlm.namespaces.*.lru_size=clear</tt></li> <li>dump debug logs, like {{lctl dk /tmp/git_dlm_cancel.log</li> <li>get checksums of all the .git files again (into a new file)</li> <li>compare checksums of before and after lock cancel</li> </ul> <p>If the checksums are different, then there is some problem with the cache flushing or similar.</p> <p>However, without a more specific reproducer, it won't be very easy to isolate when this is happening.</p> <p>I've seen a transient corruption:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ git fsck error: 6419686540529fe8937aa6a7f01989109c7be7c6: object corrupt or missing error: af074dd53d04eda6d5db0f2368f0390d4060ca70: object corrupt or missing error: ffdec01d82af9ca59bec7b1fdf941e4a8d84db2e: object corrupt or missing fatal: index file smaller than expected login2$ </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ diff ~/git_before.md5sum ~/git_after.md5sum 0a1 &gt; 2b1bc9e225f27e10a228974b47281ff9 .git/refs/heads/master 20a22,24 &gt; 122257de7cf6016e026760c7791d1d5a .git/objects/af/074dd53d04eda6d5db0f2368f0390d4060ca70 &gt; be5e8c66fd02b1d9662a6deb2d4d325a .git/objects/64/19686540529fe8937aa6a7f01989109c7be7c6 &gt; 9963ce604411643548921a66fc0a67d2 .git/objects/ff/dec01d82af9ca59bec7b1fdf941e4a8d84db2e 29,32c33,36 &lt; 1414af68fbd29b3dafa9152a49453010 .git/logs/refs/heads/master &lt; 1414af68fbd29b3dafa9152a49453010 .git/logs/HEAD &lt; fde41e17523926db4a56131b9a313c54 .git/COMMIT_EDITMSG &lt; 99e1f2253855d6cf020dce0fff06fdfd .git/index --- &gt; 1f8f81bd507eac9467924aab7cbe9995 .git/logs/refs/heads/master &gt; 1f8f81bd507eac9467924aab7cbe9995 .git/logs/HEAD &gt; 42bee3bb1f71aec0b3e61f0fcf4f65d6 .git/COMMIT_EDITMSG &gt; a73a56d1dc1af89cbcb0abc836864f82 .git/index </pre> </div></div> <p>and then try git fsck again:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>login2$ git fsck login2$ </pre> </div></div> <p>I notice the difference here is <tt>object corrupt or missing</tt> compared to the reported <tt>$git status ... loose object ... is corrupt</tt>.</p> <p>NOTE: These results are from a machine where <tt>sync</tt> is not available to the user.</p> <p>Hi,<br/> Just as a note, we had one user who had this problem. About 6 weeks ago, our sysadmin increased two parameters on the lustre clients :<br/> LRU<br/> MaxDirtyMegabytes</p> <p>He increased the LRU to 10 000, and MaxDirtyMegabytes to 256MB. </p> <p>Since then, our user did not get an error.</p> <p>Might be worth investigating those parameters.</p> <p>Regards,</p> <p>Maxime Boissonneault</p> <p>Thanks for this info Maxime.</p> <p>Can you provide the version of Lustre you observed this problem with, the mount options and if sync is enabled on your machine.</p> <p>If you can readily reproduce this problem - and can share a reproducible configuration - that would be very helpful.</p> <p>cheers,<br/> Richard</p> <p>Hi Richard,<br/> Here is the<br/> <span class="error">&#91;mboisson@colosse2 ~&#93;</span>$ cat /proc/fs/lustre/version<br/> lustre: 1.8.8<br/> kernel: patchless_client<br/> build: jenkins-wc1-gbc88c4c-PRISTINE-2.6.18-308.16.1.el5</p> <p>The mount options are : <br/> mds2-ib0@o2ib,mds1-ib0@o2ib:/lustre1 on /lustre type lustre (rw,noauto,localflock)<br/> mds4-ib0@o2ib,mds3-ib0@o2ib:/lustre2 on /lustre2 type lustre (rw,noauto,localflock)<br/> 10.225.16.3@o2ib0:/fs1 on /lustre3 type lustre (rw,noauto,localflock)</p> <p>How do I know if sync is enabled ?</p> <p>We can not readily reproduce the problem. First, it does not seem to happen anymore, and second, when it did happen, it was at random times.</p> <p>Best,</p> <p>Maxime</p> <p>on the topic of sync: I've seen clients on some systems with /bin/sync as only executable by root.</p> <p>Hi Richard,<br/> /bin/sync can be executed by others on our system.</p> <p>Best,</p> <p>Maxime</p> <p>Close old issue.</p> Related LU-2276 LU-305 Development Rank 1|hzvdcn: Rank (Obsolete) 5768