kern.log.gz
[LU-179] lustre client lockup when under memory pressure Created: 30/Mar/11 Updated: 26/Feb/16 Resolved: 06/Jul/11 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | Lustre 1.8.6 |
| Type: | Bug | Priority: | Minor |
| Reporter: | Shuichi Ihara (Inactive) | Assignee: | Zhenyu Xu |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Environment: |
Client is running 2.6.27.45-lustre-1.8.3.ddn3.3. Connectivity is 10GigE |
||
| Attachments: |
kern.log.gz
|
| Severity: | 3 |
| Rank (Obsolete): | 10103 |
| Description |
|
A customer is seeing a problem on a client where the client loses access to Lustre when the node is subjected to memory pressure from an errant application. Lustre starts reporting -113 (No route to host) errors for certain NIDS in the filesystem despite the TCP/IP network being functional. After the memory pressure is relieved the Lustre errors remain. I am collecting logs currently. From the customer report: Lnet is reporting no-route-to-host for a significant number of OSS / MDSs (client log attached). Mar 29 09:23:27 cgp-bigmem kernel: [589295.826095] LustreError: 4980:0:(events.c:66:request_out_callback()) @@@ type 4, status but from user-space on the client, all those nodes are pingable: cgp-bigmem:/var/log# ping 172.17.128.130 however a lnet ping hangs: From another client, the ping works as expected farm2-head1:# lctl ping 172.17.128.130@tcp cgp-bigmem:~# lfs check servers | grep -v active |
| Comments |
| Comment by Peter Jones [ 30/Mar/11 ] |
|
Bobijam Could you please look into this one? Thanks Peter |
| Comment by Zhenyu Xu [ 01/Apr/11 ] |
|
Can we get more logs? |
| Comment by Liang Zhen (Inactive) [ 01/Apr/11 ] |
|
I think this could be fixed by patch on bug 21776, just FYI |
| Comment by Zhenyu Xu [ 01/Apr/11 ] |
|
patch https://bugzilla.lustre.org/attachment.cgi?id=29521&action=edit for 1.8.1 can also be applied on v1_8_3 We'd still like to check logs to see whether this issue fits the phenomenon of bz 21776. |
| Comment by Ashley Pittman (Inactive) [ 13/Apr/11 ] |
|
The customer is happy that it's the above issue and is preparing to update. Unless the problem re-occurs they are going to wait for 1.8.6 and go straight to that. We'll reopen if we observe the issue again once the patch is applied. |
| Comment by Zhenyu Xu [ 13/Apr/11 ] |
|
dup of bug 21776 |
| Comment by Cory Spitz [ 16/Apr/11 ] |
|
If it is a dup of 21776, note that the o2iblnd does not use PF_MEMALLOC with the patches that were landed to 21776. Also, we (Cray) are chasing a similar instance down attachMD() so other paths might need more help. |
| Comment by Cory Spitz [ 19/Apr/11 ] |
|
Re the issues with LNetMDAttach(), ptl_send_rpc() calls libcfs_memory_pressure_get_and_set(), but not until after ptlrpc_register_bulk() which will call LNetMDAttach(). We have seen LNetMDAttach() fail to allocate in that case, which then causes the writeback to hang and deadlock ensues. I'm afraid of creeping the scope where PF_MEMALLOC is used to bail out ptlrpc, however. Comments? |
| Comment by Liang Zhen (Inactive) [ 19/Apr/11 ] |
|
o2iblnd will always try to preallocate memory descriptors, so we don't have patch for it.
if (request->rq_memalloc) So I think we can fix it just by moving these two lines ahead. |
| Comment by Zhenyu Xu [ 19/Apr/11 ] |
|
patch tracking at http://review.whamcloud.com/439 |
| Comment by Zhenyu Xu [ 19/Apr/11 ] |
|
1.8.6 needs minor fix |
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Zhenyu Xu [ 20/Apr/11 ] |
|
minor fix landed on b1_8 |
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Build Master (Inactive) [ 20/Apr/11 ] |
|
Integrated in Johann Lombardi : 134461a981b134de896d9aa9cc6ec2d816cfa044
|
| Comment by Peter Jones [ 21/Apr/11 ] |
|
Does this fix need landing to master? |
| Comment by Zhenyu Xu [ 21/Apr/11 ] |
|
no, it was already in master. |
| Comment by Cory Spitz [ 27/Apr/11 ] |
|
Hi. I agree with the fix to relocate the register bulk. Are you planning on submitting it upstream (to Oracle)? |
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Build Master (Inactive) [ 08/Jun/11 ] |
|
Integrated in Johann Lombardi : 08b76cd92b2a4b6854ce3910a07531996449a9fd
|
| Comment by Guy Coates [ 10/Jun/11 ] |
|
Client log |
| Comment by Guy Coates [ 10/Jun/11 ] |
|
We've just had a re-occurrence of this problem running 1.8.5.56 (as tagged in git). |
| Comment by Guy Coates [ 13/Jun/11 ] |
|
Was able to get output from top from the last client lockup; pdflush is sat in 100% CPU. top - 09:10:36 up 2 days, 20:08, 2 users, load average: 801.64, 799.78, 796.51 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND |
| Comment by Ashley Pittman (Inactive) [ 20/Jun/11 ] |
|
As above the customer is was still observing this problem using the latest code on the 10th Jun, could you reopen this bug accordingly. |
| Comment by Zhenyu Xu [ 20/Jun/11 ] |
|
is it the same pattern as lnet ping hangs while ping works ok? |
| Comment by Ashley Pittman (Inactive) [ 20/Jun/11 ] |
|
Yes. The system is a NUMA system with 512Gb ram currently. The problem seems to happen during memory pressure, a figure of 70% has been quoted but it's worth saying the application is single-threaded so it's quite likely that some NUMA regions are experiencing 100% memory usage. One thing I've suggested is pinning the application to a different NUMA region to the lustre kernel threads (if this is even possible) so the application wouldn't starve Lustre of memory so easily. |
| Comment by Zhenyu Xu [ 20/Jun/11 ] |
|
can you help checkout what lustre threads was doing during this hang? (better to have thread stacks) |
| Comment by Guy Coates [ 06/Jul/11 ] |
|
We upgraded the kernel on this machine from 2.6.27/SLES11 kernel + 1.8.5.56 lustre client to 2.6.32 kernel +1.8.5.56 lustre client , and the problems seems to have stopped. You can close this issue. Thanks, Guy |
| Comment by Zhenyu Xu [ 06/Jul/11 ] |
|
close ticket per Guy Coates' update info. |
