[LU-2440] git repositories get corrupted

• Type: Story
• Resolution: Cannot Reproduce
• Priority: Major
• Affects Version/s: Lustre 1.8.6
• Environment: EL5
• Assignee: Richard Henwood (Inactive)
• Reporter: Richard Henwood (Inactive)

Description

I have received notice of an issue a user at TACC is having:

I am having a problem where my git repositories on TACC systems get corrupted, leading to errors like this:

login4$ git status
fatal: index file smaller than expected
login4$ git status
error: object file .git/objects/01/ee9f4bfe74aaee027a3e8418d70d337e1235d3
is empty
fatal: loose object 01ee9f4bfe74aaee027a3e8418d70d337e1235d3 (stored in
.git/objects/01/ee9f4bfe74aaee027a3e8418d70d337e1235d3) is corrupt
login4$ git status
error: object file .git/objects/8d/6083737dae5cb67906ac26702465ca2d70bc95
is empty
fatal: loose object 8d6083737dae5cb67906ac26702465ca2d70bc95 (stored in
.git/objects/8d/6083737dae5cb67906ac26702465ca2d70bc95) is corrupt
login4$ git status
error: object file .git/objects/bc/61c57143652fbf198de898ca7bb9d5659a5de0
is empty
fatal: loose object bc61c57143652fbf198de898ca7bb9d5659a5de0 (stored in
.git/objects/bc/61c57143652fbf198de898ca7bb9d5659a5de0) is corrupt

Issue Links

• is related to: LU-2276 is open() idempotent in regards to being restarted after a signal interrupts it?
• is related to: LU-305 utime() fails with EINTR : not conform to POSIX standard

Comments

[Andreas Dilger]

Close old issue.

[Maxime Boissonneault]

Hi Richard,
/bin/sync can be executed by others on our system.

Best,

Maxime

[Richard Henwood (Inactive)]

on the topic of sync: I've seen clients on some systems with /bin/sync as only executable by root.

[Maxime Boissonneault]

Hi Richard,
Here is the
[mboisson@colosse2 ~]$ cat /proc/fs/lustre/version
lustre: 1.8.8
kernel: patchless_client
build: jenkins-wc1-gbc88c4c-PRISTINE-2.6.18-308.16.1.el5

The mount options are :
mds2-ib0@o2ib,mds1-ib0@o2ib:/lustre1 on /lustre type lustre (rw,noauto,localflock)
mds4-ib0@o2ib,mds3-ib0@o2ib:/lustre2 on /lustre2 type lustre (rw,noauto,localflock)
10.225.16.3@o2ib0:/fs1 on /lustre3 type lustre (rw,noauto,localflock)

How do I know if sync is enabled ?

We can not readily reproduce the problem. First, it does not seem to happen anymore, and second, when it did happen, it was at random times.

Best,

Maxime

[Richard Henwood (Inactive)]

Thanks for this info Maxime.

Can you provide the version of Lustre you observed this problem with, the mount options and if sync is enabled on your machine.

If you can readily reproduce this problem - and can share a reproducible configuration - that would be very helpful.

cheers,
Richard

[Maxime Boissonneault]

Hi,
Just as a note, we had one user who had this problem. About 6 weeks ago, our sysadmin increased two parameters on the lustre clients :
LRU
MaxDirtyMegabytes

He increased the LRU to 10 000, and MaxDirtyMegabytes to 256MB.

Since then, our user did not get an error.

Might be worth investigating those parameters.

Regards,

Maxime Boissonneault