[LU-736] LBUG and kernel panic on client unmount

• Status: Closed
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: Christopher Morrone
• Assignee: Christopher Morrone
• Resolution: Won't Fix
• Labels: llnl

Environment

1.8.5.0-5chaos. https://github.com/chaos/lustre/tree/1.8.5.0-5chaos

• Severity: 3
• Bugzilla ID: 23,861

Description

We had a few hundred clients all LBUG and then kernel panic on unmount of a lustre filesystem recently. All the ones that I checked have the same backtrace. See the attached sierra32_console.txt.

It looks like others have hit this in earlier 1.8 versions. See bugzilla.lustre.org bug 23861.

Comments

[Christopher Morrone]

To make this issue more searchable, the LBUG is here:

2011-09-29 07:51:30 LustreError: 19065:0:(ldlm_lock.c:1568:ldlm_lock_cancel()) ### lock still has references ns: lsa-MDT0000-mdc-ffff810332040400 lock: ffff810263a92e00/0xb23761f5d085be87 lrc: 4/0,1 mode: PW/PW res: 578792285/4020328757 rrc: 2 type: FLK pid: 21451 [0->9223372036854775807] flags: 0x22002890 remote: 0x1f055096a089059 expref: -99 pid: 21451 timeout: 0
2011-09-29 07:51:30 LustreError: 19065:0:(ldlm_lock.c:1569:ldlm_lock_cancel()) LBUG

[Peter Jones]

HongChao

Could you please look into this one?

Thanks

Peter

[Peter Jones]

Hongchao

Could you please provide a status update?

Thanks

Peter

[Hongchao Zhang]

the readers/writers of the flock's LDLM lock isn't zero until it is canceled by unlock request, the reference will only be
dropped by "ldlm_flock_completion_ast" in "cleanup_resource" by setting "LDLM_FL_LOCAL_ONLY|LDLM_FL_FAILED" flag in the
LDLM lock.

in this case, the flag of the lock is "0x22002890", only contains LDLM_FL_FAILED, no LDLM_FL_LOCAL_ONLY. and during umount
this flag will be set only if obd->obd_force is set,

if there are flock's LDLM lock during umount and obd->obd_force isn't set, then this issue wil be triggered.

Hi Chris,
Do you add "-f" flag when you umount the Lustre client? thanks.

[Hongchao Zhang]

Hi Chris,
could you please also check whether your application running on the Lustre client leave some flocks unlocked?
I have tested locally with leaving some flock unlocked deliberately and umount Lustre, which triggers this LBUG.

Thanks

[Christopher Morrone]

I will find out what the admins did to umount lustre.

It is going to be rather difficult to track down whether an of the various applications are using flock, and how. Most of our users won't know the answer to that that, even if there application IS using flock.

Perhaps it is relevant that we are mounting with the "flock" option enabled.

[Christopher Morrone]

As far as they can recall, they did not use the umount -f option.

[Hongchao Zhang]

the initial patch is tracked at http://review.whamcloud.com/#change,2535

[Christopher Morrone]

Thanks. FYI unless this is also a problem for 2.1, this ticket is very low priority compared to our many 2.1 bugs. We do not plan to fix any 1.8 bugs in production.

[Hongchao Zhang]

Hi Chris,
Do you need any more work on this ticket? Or are we OK to close it? Thanks

[Christopher Morrone]

This is so old that I think you can close it with resolution "Won't Fix".

[Hongchao Zhang]

Chris, Thanks!