[LU-12198] lnetctl peer show hangs for ~2600 clients, ioctl getting E2BIG

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0, Lustre 2.12.5
• Affects Version/s: Lustre 2.12.0
• Environment: x86 servers, 2.12 no patches, RHEL 7.6
• Severity: 3
• Assignee: Dominique Martinet (Inactive)
• Reporter: Ruth Klundt (Inactive)

Description

command `lnetctl peer show` appears to hang, strace shows looping on:

ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x65, 0x64, 0xb8), 0x7fffffffccf0) = -1 E2BIG (Argument list too long)

There are 2605 lines in /sys/kernel/debug/lnet/peers.

Issue Links

• is related to: LU-9680 Improve the user land to kernel space interface for lustre

Comments

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37720
Subject: LU-12198 libcfs: always copy ioctl header back to user
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 321a757880445089a48d26acfe0554853750ca3f

[Peter Jones]

Landed for 2.14

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37559/
Subject: LU-12198 libcfs: always copy ioctl header back to user
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9e02ef474f8caa833d6a1b5e0068d5323a57e8c4

[Gerrit Updater]

Dominique Martinet ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37559
Subject: LU-12198 libcfs: always copy ioctl header back to user
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 818fc691a0e29e5764bfcd65d2a1918c5369fe7c

[Dominique Martinet (Inactive)]

The problem is that the hdr header in kernel does not go back to userspace on error, so when lnet_get_peer_list writes back to *sizep it stays in kernel and does not fill back the value for lnetctl to grow the buffer.

[Mahmoud Hanafi]

We are hitting this issue on our routers.

 lnetctl peer show

will hang and strace show 

 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x65, 0x64, 0xb8), 0x7fffffffe6b0) = -1 E2BIG (Argument list too long)

[Ruth Klundt (Inactive)]

fyi a bit further down the code appears to retry with count and size returned from ioctl. those values must not be getting across correctly though because the call loops indefinitely.

[Sonia Sharma (Inactive)]

Yes, this is because we have this limit of 1000 peers while allocating buffer to get the peer list- 

2996         count = 1000;                                                                                                                                        
2997         size = count * sizeof(struct lnet_process_id);                                                                                                       
2998         list = malloc(size);

009                         LIBCFS_IOC_INIT_V2(peer_info, prcfg_hdr);                                                                                            
3010                         peer_info.prcfg_hdr.ioc_len = sizeof(peer_info);                                                                                     
3011                         peer_info.prcfg_size = size;                                                                                                         
3012                         peer_info.prcfg_bulk = list;                                                                                                         
3013                                                                                                                                                              
3014                         l_errno = 0;                                                                                                                         
3015                         rc = l_ioctl(LNET_DEV_ID, IOC_LIBCFS_GET_PEER_LIST,                                                                                  
3016                                      &peer_info);                                                                                                            
3017                         count = peer_info.prcfg_count;                                                                                                       
3018                         if (rc == 0)                                                                                                                         
3019                                 break;                                                                                                                       
3020                         l_errno = errno;