[LU-16634] Null pointer dereference in lustre_set_wire_obdo

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.16.0
• Type: Bug
• Priority: Critical
• Reporter: Tao Lyu
• Assignee: Andreas Dilger
• Resolution: Fixed

Environment

Lustre commit: 9ddcdee2c8b9ec14986b93cf3180d946cd4869f7

Three server nodes and one client. Kernel version: Ubuntu-5.4.0-90.101

1. MGS
   mkfs.lustre --fsname=lustre --mgs /dev/vda
   mount -t lustre /dev/vda /root/lustre-server

1. MDS
   mkfs.lustre --fsname=lustre --index=0 --mgsnode=$start_ip@tcp0 --mdt /dev/vda
   mount -t lustre /dev/vda /root/lustre-server

1. OSS
   mkfs.lustre --ost --fsname=lustre --index=1 --reformat --mgsnode=$start_ip@tcp0 /dev/vda
   mount -t lustre /dev/vda /root/lustre-server

1. Client
   mount -t lustre $start_ip@tcp0:/lustre /root/lustre-client

Sub-Tasks

Key	Summary	Type	Status	Assignee
LU-17489	fix 'lfs getname' ioctl breakage	Technical task	Open	Andreas Dilger

• Key: Summary
• Summary: Type
• Type: Status
• Status: Assignee
• Severity: 3

Description

[ 279.518552] kasan: GPF could be caused by NULL-ptr deref or user memory access
[ 279.520881] general protection fault: 0000 1 SMP KASAN NOPTI
[ 279.523366] CPU: 1 PID: 555 Comm: test Tainted: G O 5.4.148+ #7
[ 279.527232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
[ 279.530776] RIP: 0010:lustre_set_wire_obdo+0x7e/0x570 [obdclass]
[ 279.556223] Call Trace:
[ 279.556666] osc_getattr+0x1eb/0x950 [osc]
[ 279.558046] osc_iocontrol+0x4f1/0xe80 [osc]
[ 279.559241] lov_iocontrol+0x4ba/0x5de0 [lov]
[ 279.567254] ll_dir_ioctl+0x2834/0x17cc0 [lustre]
[ 279.590353] do_vfs_ioctl+0x405/0x660
[ 279.590784] ksys_ioctl+0x5e/0x90

Issue Links

• is related to: LU-12613 Lustre lustre_msg_string() bug
• is related to: LU-13107 clean up lctl commands
• is related to: LU-16616 crash in osc_brw_prep_request() ASSER...
• is related to: LU-16617 A null-pointer dereference in osc_req...
• is related to: LU-16688 Client crash due to an assertion in l...

Comments

[Andreas Dilger]

Hello tao.lyu, this bug needs to have more information in it before there is any chance someone could identify the issue.

At a minimum, a stack trace and any console logs from before the crash. Preferably also "gdb> list *(lustre_set_wire_obdo+NNN)" that shows which line in that function the crash happened. 

[Tao Lyu]

Hi, sorry for the incomplete information.

This crashed location is lustre/obdclass/obdo.c:182

root@dfs:~/lustre-cserver# gdb /lib/modules/5.4.148+/updates/fs/obdclass.ko
Reading symbols from /lib/modules/5.4.148+/updates/fs/obdclass.ko...
(gdb) list *(lustre_set_wire_obdo+0x7e)
0xaa7ee is in lustre_set_wire_obdo (/root/lustre-cserver/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/obdclass/obdo.c:182).
177	 */
178	void lustre_set_wire_obdo(const struct obd_connect_data *ocd,
179				  struct obdo *wobdo,
180				  const struct obdo *lobdo)
181	{
182		*wobdo = *lobdo;
183		if (ocd == NULL)
184			return;
185	
186		if (!(wobdo->o_valid & OBD_MD_FLUID))

The stack trace is shown below.

[  279.516647] kasan: CONFIG_KASAN_INLINE enabled
[  279.518552] kasan: GPF could be caused by NULL-ptr deref or user memory access
[  279.520881] general protection fault: 0000 [#1] SMP KASAN NOPTI
[  279.523366] CPU: 1 PID: 555 Comm: test Tainted: G           O      5.4.148+ #7
[  279.527232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
[  279.530776] RIP: 0010:lustre_set_wire_obdo+0x7e/0x570 [obdclass]
[  279.533624] Code: 40 84 c6 0f 85 78 04 00 00 84 c9 0f 95 c2 0f 9e c0 84 c2 0f 85 68 04 00 00 48 ba 00 00 00 00 00 fc ff df 4c 89 e0 48 c1 e8 03 <0f> b6 0c 10 49 8d 84 24 cf 00 00 00 48 89 c6 83 e0 07 48 c1 ee 03
[  279.540387] RSP: 0018:ff110002456ff678 EFLAGS: 00010206
[  279.541933] RAX: 0000000000000003 RBX: ff1100024a8462a0 RCX: 0000000000000000
[  279.543988] RDX: dffffc0000000000 RSI: 1fe22000495ac501 RDI: ff1100024a8462a0
[  279.545662] RBP: ff1100024ad628e8 R08: ff11000245455ac0 R09: ffe21c0049508c52
[  279.547314] R10: ffe21c0049508c51 R11: ff1100024a84628b R12: 0000000000000018
[  279.548570] R13: ff1100024964b000 R14: 0000000000000018 R15: ff110002509536d8
[  279.549797] FS:  00007ffff7fc4540(0000) GS:ff11000257280000(0000) knlGS:0000000000000000
[  279.551165] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  279.552129] CR2: 00007ffff7f4fffc CR3: 0000000247498001 CR4: 0000000000761ee0
[  279.553346] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  279.554550] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  279.555759] PKRU: 55555554
[  279.556223] Call Trace:
[  279.556666]  osc_getattr+0x1eb/0x950 [osc]
[  279.558046]  osc_iocontrol+0x4f1/0xe80 [osc]
[  279.559241]  lov_iocontrol+0x4ba/0x5de0 [lov]
[  279.567254]  ll_dir_ioctl+0x2834/0x17cc0 [lustre]
[  279.590353]  do_vfs_ioctl+0x405/0x660
[  279.590784]  ksys_ioctl+0x5e/0x90
[  279.591175]  __x64_sys_ioctl+0x16/0x20
[  279.591626]  do_syscall_64+0x48/0x140
[  279.592038]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

[Tao Lyu]

After setup the file system, executing the binary compiled from attached poc.c triggers this bug.

[Andreas Dilger]

Tao, thanks for filing your report.

After setup the file system, executing the binary compiled from attached poc.c triggers this bug.

Sorry, I don't see any poc.c attachment on this ticket. Alternately, if you consider this to be a security issue, you can email me the poc.c program at [e-mail] and I can make it available internally for review until after we release a fix.

Looking at the stack trace and command being run it looks like you are running some kind of ioctl/syscall fuzzer for a research project? It would be helpful to disclose this kind of information in the initial bug report.

It will be interesting to see what the fuzzer is doing.

[Tao Lyu]

Hi Andreas,

Oh, I'm sorry. I forgot to upload the POC. I've sent it to you.

Yes, we are testing lustre through a distributed fuzzing framework. It generally tests lustre by feeding lots of mutated syscall sequences.

[Tao Lyu]

By the way, does lustre support to run servers and clients in the same node?
We tried to compile and install server and client modules in the same node. Unfortunately, it reports the server and client modules are in conflict.
Is this its nature?

[Andreas Dilger]

By the way, does lustre support to run servers and clients in the same node?
We tried to compile and install server and client modules in the same node. Unfortunately, it reports the server and client modules are in conflict.

This is just a misunderstanding of how the packaging is done. If you build and install the "server" package it can also mount the exported filesystem as a client. The "client" package can only mount the exported filesystem and cannot be a server. The client packages have reduced functionality to make them more easily portable to different kernels/distros.

[Tao Lyu]

Okay, got it. Thanks a lot for your explaination.

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50314
Subject: LU-16634 misc: standardize handling of ioctl params
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 496b2b5190703ae6cec88568a6837f0347185fd9

[Andreas Dilger]

Hello tao.lyu, I've created a patch that should address the class of issues that you uncovered. The Gerrit link is above, and I will also attach the current version of the patch against the master branch to this ticket.

There is one cosmetic defect in the current version that is causing regression test failures (returning -EINVAL instead of -ENOTTY in one test case), but it should be ok for your testing.

Please let me know if this resolved the issue you hit. Once it has been reviewed and passes regression testing the patch will be landed to master and backpirted to the maintenance branches.

Please file a separate Jira ticket you uncover other unrelated issues.

[Andreas Dilger]

PS: I will add "Reported-by: Tao Lyu <[e-mail]>" in the updated version of the patch.

[Tao Lyu]

Thanks, I'll test it later and reply you.

Could you please take a look at another two issues I reported:
LU-16617
LU-16616

Thanks!

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50331
Subject: LU-16634 build: improve checkpatch warnings
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b0b4408491d1414848db289c594524a58cffddba

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50333
Subject: LU-16634 misc: remove unnecessary ioctl typecasts
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2dfbbce062dfcfa7e08d1fe6f7745ba2ced0a6b0

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50334
Subject: LU-16634 obdclass: improve iocontrol error messages
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 08517e8ffb81f2a3092057d4f905afd8e224bc6f

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50335
Subject: LU-16634 llite: move common ioctl code to ll_iocontrol()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0609508786a8192e6fa691f632433ee71b209eee

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50390
Subject: LU-16634 misc: replace obsolete ioctl numbers
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 882314ade0a2b51f3b119ac71d5c1841e48571f1

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50331/
Subject: LU-16634 build: improve checkpatch warnings
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fe13a1b01c61801065571b4049e8c6bd544742b7

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50333/
Subject: LU-16634 misc: remove unnecessary ioctl typecasts
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4a1465577e1310ce09834103f7136bef0e1a1eeb

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50334/
Subject: LU-16634 obdclass: improve iocontrol error messages
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1f4825eff026321a8e769739868d4d482026c3ee

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50335/
Subject: LU-16634 llite: move common ioctl code to ll_iocontrol()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3be425883918528ef981af697b34de187c3937b3

[Andreas Dilger]

tao.lyu, have you been able to validate that the provided patch (or the latest patch https://review.whamcloud.com/50314 on master) fixes the issue reported here and in LU-16617?

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50314/
Subject: LU-16634 misc: standardize iocontrol param handling
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 5eae8514f5f1730fe93d55d348400f5ecf681078

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50390/
Subject: LU-16634 misc: replace obsolete ioctl numbers
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 594b780062db7e015837faefa75f7352293f54ae

[Peter Jones]

All patches seem to have landed for 2.16

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51596
Subject: LU-16634 build: remove obsolete checkpatch strings
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4566b6d452f2ccc6b89c23b1865f3a5b5d0b2038

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51596/
Subject: LU-16634 build: remove obsolete checkpatch strings
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cf50755cce8b19c9cbfb20c12137b0eb3c3b66da