Details
-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
None
-
None
-
3
-
9223372036854775807
Description
when running a real workload reading a dataset that is eligible to be cached with PCC. This dataset is made up of only a few small files (~6). One of the files is very large (>128GB). This particular job crash seems to be consistently reproducible:
[405658.893856] BUG: kernel NULL pointer dereference, address: 00000000000000e8
[405658.902513] #PF: supervisor read access in kernel mode
[405658.909000] #PF: error_code(0x0000) - not-present page
[405658.915491] PGD 71c2d55067 P4D 71c2d55067 PUD 0
[405658.921382] Oops: 0000 [#4] SMP NOPTI
[405658.926166] CPU: 144 PID: 576639 Comm: python Tainted: P D OE 5.4.0-42-generic #46-Ubuntu
[405658.937145] Hardware name: NVIDIA DGXA100 920-23687-2530-000/DGXA100, BIOS 0.27 07/28/2020
[405658.947090] RIP: 0010:pcc_fault+0x38/0x320 [lustre]
[405658.953205] Code: 41 56 41 55 41 54 49 89 d4 53 48 89 fb 48 83 ec 10 4c 8b af a0 00 00 00 4c 8b b7 b0 00 00 00 49 8b 85 c8 00 00 00 4d 8b 45 20 <4c> 8b b8 e8 00 00 00 f6 05 16 96 82 ff 01 74 0d f6 05 11 96 82 ff
[405658.975336] RSP: 0000:ffffb4c61e8bfc80 EFLAGS: 00010282
[405658.981774] RAX: 0000000000000000 RBX: ffff919c880a0820 RCX: 0000000000000018
[405658.990347] RDX: ffffb4c61e8bfcfd RSI: ffffb4c61e8bfdc0 RDI: ffff919c880a0820
[405658.998904] RBP: ffffb4c61e8bfcb8 R08: ffff91a38c21b448 R09: ffffb4c61e8bfdc0
[405659.007452] R10: 0000000000000000 R11: 0000000000000000 R12: ffffb4c61e8bfcfd
[405659.015965] R13: ffff91cd159d5c00 R14: ffffffffbc482de0 R15: ffff91d0bd9e7b40
[405659.024489] FS: 00007f4c1b7fe700(0000) GS:ffff91dacf800000(0000) knlGS:0000000000000000
[405659.034090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[405659.041074] CR2: 00000000000000e8 CR3: 0000007e93cc4000 CR4: 0000000000340ee0
[405659.049615] Call Trace:
[405659.052905] ll_fault+0x43/0x770 [lustre]
[405659.057932] ? unlock_page_memcg+0x12/0x20
[405659.063105] __do_fault+0x3c/0x130
[405659.067432] do_fault+0x24b/0x640
[405659.071654] __handle_mm_fault+0x4c5/0x7a0
[405659.076746] handle_mm_fault+0xca/0x200
[405659.081564] do_user_addr_fault+0x1f9/0x450
[405659.086752] ? do_futex+0x160/0x1e0
[405659.091166] __do_page_fault+0x58/0x90
[405659.095855] do_page_fault+0x2c/0xe0
[405659.100346] page_fault+0x34/0x40
[405659.104539] RIP: 0033:0x7f4e6cfc77e0
[405659.109030] Code: 0f 0f 84 83 00 00 00 48 8b 0d 6c eb 27 00 4c 8d 1d 35 08 05 00 4f 63 0c 8b 4f 8d 0c 0b 41 ff e1 0f 0b 0f 1f 84 00 00 00 00 00 <f3> 0f 6f 44 16 f0 48 01 d6 4c 8d 44 17 f0 48 01 d7 48 89 f9 48 83
[405659.130978] RSP: 002b:00007f4c1b7fdaf8 EFLAGS: 00010283
[405659.137360] RAX: 00007f4c145aaf50 RBX: 00007f4c1b7fdb30 RCX: 0000000000000006
[405659.145870] RDX: 0000000000015f04 RSI: 00007f4d2c2b4000 RDI: 00007f4c145aaf50
[405659.154380] RBP: 00007f4c1b7fdb70 R08: 00007f4d7b88dda1 R09: 0000000000000000
[405659.162909] R10: 00007f4c140008d0 R11: 00007f4e6d017ec0 R12: 000055c0b94b2510
[405659.171439] R13: 00007f4e50616aa0 R14: 000055c0b99d44c0 R15: 00007f4c1b7fdbf0
[405659.179960] Modules linked in: nv_peer_mem(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) msr overlay nvidia_uvm(OE) nfsv3 nfs_acl nfs lockd grace fscache gdrdrv(POE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) esp6_offload esp6 esp4_offload esp4 xfrm_algo mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_en(OE) mlx4_ib(OE) ib_core(OE) mlx4_core(OE) amd64_edac_mod edac_mce_amd nvidia_drm(POE) nvidia_modeset(POE) kvm_amd kvm crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper ipmi_ssif nvidia(POE) nls_iso8859_1 mxm_wmi wmi ast video drm_vram_helper ttm drm_kms_helper cdc_ether fb_sys_fops syscopyarea usbnet input_leds joydev sysfillrect mii sysimgblt ccp k10temp ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel knem(OE) drm sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
[405659.180007] raid6_pq libcrc32c raid1 multipath linear ses enclosure raid0 hid_generic mlx5_core(OE) crc32_pclmul vfio_mdev mdev(OE) mpt3sas igb tls uas i2c_algo_bit usbhid raid_class mlxfw(OE) nvme dca usb_storage scsi_transport_sas mlx_compat(OE) hid i2c_piix4 nvme_core [last unloaded: nv_peer_mem]
[405659.313282] CR2: 00000000000000e8
[405659.317676] ---[ end trace ab9159a407597e5c ]---