[LU-12198] lnetctl peer show hangs for ~2600 clients, ioctl getting E2BIG

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0, Lustre 2.12.5
• Affects Version/s: Lustre 2.12.0
• Environment: x86 servers, 2.12 no patches, RHEL 7.6
• Severity: 3
• Assignee: Dominique Martinet (Inactive)
• Reporter: Ruth Klundt (Inactive)

Description

command `lnetctl peer show` appears to hang, strace shows looping on:

ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x65, 0x64, 0xb8), 0x7fffffffccf0) = -1 E2BIG (Argument list too long)

There are 2605 lines in /sys/kernel/debug/lnet/peers.

Issue Links

• is related to: LU-9680 Improve the user land to kernel space interface for lustre

Comments

[James A Simmons]

Is this with 2.14?  I suspect we are reaching the limits of using ioctls.

[Joe Grund]

I'm seeing something similar to this while calling both

lnetctl export

and

lnetctl peer show

These commands hang and running strace shows the above.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37720/
Subject: LU-12198 libcfs: always copy ioctl header back to user
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: a3c687a943233a7c5ae7e3fb906d1913b063c95c

[Mahmoud Hanafi]

It would be nice if this could land in the next 2.12.x.

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37720
Subject: LU-12198 libcfs: always copy ioctl header back to user
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 321a757880445089a48d26acfe0554853750ca3f

[Peter Jones]

Landed for 2.14

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37559/
Subject: LU-12198 libcfs: always copy ioctl header back to user
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9e02ef474f8caa833d6a1b5e0068d5323a57e8c4

[Gerrit Updater]

Dominique Martinet ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37559
Subject: LU-12198 libcfs: always copy ioctl header back to user
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 818fc691a0e29e5764bfcd65d2a1918c5369fe7c

[Dominique Martinet (Inactive)]

The problem is that the hdr header in kernel does not go back to userspace on error, so when lnet_get_peer_list writes back to *sizep it stays in kernel and does not fill back the value for lnetctl to grow the buffer.

[Mahmoud Hanafi]

We are hitting this issue on our routers.

 lnetctl peer show

will hang and strace show 

 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x65, 0x64, 0xb8), 0x7fffffffe6b0) = -1 E2BIG (Argument list too long)