[LU-5514] After upgrade from 1.8.7 to 2.5.2 stack trace cfs_hash_bd_lookup_intent Created: 20/Aug/14 Updated: 13/Oct/21 Resolved: 13/Oct/21 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major |
| Reporter: | Dave Bond (Inactive) | Assignee: | Lai Siyao |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Attachments: |
lustre-log.1408470549.15310.txt.7z
|
||||||||
| Issue Links: |
|
||||||||
| Epic/Theme: | Lustre-2.5.2 | ||||||||
| Severity: | 3 | ||||||||
| Rank (Obsolete): | 15371 | ||||||||
| Description |
|
We have upgraded from 1.8.7 to 2.5.2 After mounting the FS at approximately 17:29 At 18:49 we see the following Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 15310 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: |
| Comments |
| Comment by Dave Bond (Inactive) [ 20/Aug/14 ] |
|
When unmounting on MDS03-01 so I can mount on MDS03-02 I see the following on MDS03-01 Message from syslogd@cs04r-sc-mds03-01 at Aug 20 09:41:30 ... Message from syslogd@cs04r-sc-mds03-01 at Aug 20 09:41:30 ... I won't elaborate too much further at the moment as I do not want to confuse the issue. |
| Comment by Peter Jones [ 20/Aug/14 ] |
|
Lai is looking into this ticket |
| Comment by Lai Siyao [ 20/Aug/14 ] |
|
I don't see anything particular in the log. And there is a patch for a similar issue: http://review.whamcloud.com/#/c/10816/, however the patch looks not included in 2.5.2, could you verify this? Besides, http://review.whamcloud.com/#/c/10816/ may not be able to fix this issue because that's in getattr context, but this in rename. There may be a similar error in rename. |
| Comment by Frederik Ferner (Inactive) [ 20/Aug/14 ] |
|
Lai, I'm not sure I fully understand your comment, did you want us (Diamond) to verify if that patch is included in 2.5.2? Also, after looking at the patch and issues referenced in the patch, I'm still not clear what might have happened? I assume it needs special circumstances during a rename/getattr to trigger this. Are there any more details know? Any hints where we should start looking for a reproducer? FYI, the file system is not yet seeing normal use as we're currently running tests before letting the users back on... Cheers, |
| Comment by Peter Jones [ 20/Aug/14 ] |
|
Frederik Lai is suggesting that you should apply the patch - which is missing from 2.5.2 but will be in the upcoming 2.5.3 release. Regards Peter |
| Comment by Lai Siyao [ 21/Aug/14 ] |
|
racer.sh test script is the reproducer for http://review.whamcloud.com/#/c/10816/, which you may use as well. |
| Comment by Lai Siyao [ 21/Aug/14 ] |
|
This turns out to be a duplicate of This is the same issue as http://review.whamcloud.com/#/c/10816/, but this happens in rename, while that's in getattr. It's a race in lookup the same object twice while object is destroyed in the mean time. I'd suggest you apply these three patches to fully solve such issue. |
| Comment by Frederik Ferner (Inactive) [ 21/Aug/14 ] |
|
Peter, Lai, thanks for clarifying this. We'll look into compiling lustre 2.5.2 + those 3 patches, but if there is an easier way for us to get the correct compiled versions, let us know. I assume, initially it should be sufficient to run this patches version on the MDS, correct? Kind regards, |
| Comment by Lai Siyao [ 21/Aug/14 ] |
|
Yes, it's sufficient with these 3 patches. Once those two patches for |
| Comment by Peter Jones [ 21/Aug/14 ] |
|
Frederik Is rename commonly used during regular production usage or is this an issue that is more likely to hit in a stressing test run? Peter |
| Comment by Frederik Ferner (Inactive) [ 21/Aug/14 ] |
|
There's a number of processes/applications that tend to create files with a temporary extension and than rename them to the proper name, to avoid any race conditions where processing starts before the data is there, so not that uncommon. And considering that we first saw it with basically only ior test runs, I'd say, something is doing it quite frequently... Cheers, |
| Comment by Peter Jones [ 21/Aug/14 ] |
|
ok fair enough then. I just did not want you to move forward with patches for something that was a rare edge case |