[LU-8043] MDS running lustre 2.5.5+ OOM when running with Lustre 2.8 GA clients

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Critical
• Fix Version/s: Lustre 2.5.5
• Affects Version/s: Lustre 2.5.5
• Environment: Cray clients running unpatched lustre 2.8 GA clients. Server side running Lustre 2.5.5 with a patch set in a RHEL6.7 environment.
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: James A Simmons

Description

Today we performed a test shot on our smaller Cray Aries cluster (700 nodes) with a non-patched lustre 2.8 GA client specially build for this system. The test were run against our atlas file system which is running a RHEL6.7 distro with the lustre version 2.5.5 with patches. During our test shot while running an IOR single shared file test across all nodes with the stripe count of 1008 the MDS server ran out of memory. I attached the dmesg output to this ticket.

Issue Links

• is related to: LU-7535 mdt_intent_layout does not care about lock handles
• is related to: LU-5367 (mdt_handler.c:2782:mdt_lock_handle_fini()) ASSERTION( !lustre_handle_is_used(&lh->mlh_pdo_lh) ) failed

Comments

[Gerrit Updater]

Comment deleted (wrong LU in commit message).

[Gerrit Updater]

Comment deleted (wrong LU in commit message).

[Jian Yu]

Thank you, Matt.

[Matt Ezell]

We attempted to reproduce this assertion on Tuesday using the same conditions as last time, but it never crashed. After that, we moved to a server with 19717 and 18060 to hopefully prevent it in the future. I think we can close this ticket and reopen if we see it again. Thanks.

[John Hammond]

It's hard to say for sure without more information but the failed assertion may be addresses by http://review.whamcloud.com/#/c/18060/.

[Matt Ezell]

We have been unable to reproduce on our testbed systems, and we haven't had an opportunity to reproduce on the production systems.

[John Hammond]

I understood that ORNL was going to reproduce with a stronger debug mask. Has that been done?

[James A Simmons]

Any progress on fixing the assertion?

[Matt Ezell]

crash> struct ptlrpc_request.rq_pill ffff8837a0061000
  rq_pill = {
    rc_req = 0xffff8837a0061000, 
    rc_fmt = 0xffffffffa0819240 <RQF_LDLM_INTENT_LAYOUT>, 
    rc_loc = RCL_SERVER, 
    rc_area = {{4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295}, {4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295}}
  }