[LU-4091] Kernel Panic MDS jbd2_journal_start+0x4f/0x110 [jbd2]

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.1
• Type: Bug
• Priority: Critical
• Reporter: Hussein N. Harake (Inactive)
• Assignee: Peter Jones
• Resolution: Cannot Reproduce

Environment

2 sockets, Intel CPU E5-2670 2.60GHz 32GB DDR3 memory1600 MHz
kernel 2.6.32-358.18.1.el6_lustre.x86_64 Scientific Linux 6.4

• Severity: 4

Description

At certain load of benchmarking the metadata performance, the MDS crashes kernel panic, the job is running on 16 MDTs and served by one MDS.

Oct 11 11:44:49 greina15 kernel: BUG: unable to handle kernel NULL pointer dereference at (null)
Oct 11 11:44:49 greina15 kernel: IP: [<ffffffffa049052f>] jbd2_journal_start+0x4f/0x110 [jbd2]
Oct 11 11:44:49 greina15 kernel: PGD 0
Oct 11 11:44:49 greina15 kernel: Oops: 0000 1 SMP
Oct 11 11:44:49 greina15 kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sda/uevent
Oct 11 11:44:49 greina15 kernel: CPU 23
Oct 11 11:44:49 greina15 kernel: Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) mmfs26(U) mmfslinux(U) tracedev(U) limic(U) ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad iw_cxgb4 iw_cxgb3 ib_core usb_storage microcode serio_raw rsxx(U) sb_edac edac_core sr_mod cdrom sg iTCO_wdt iTCO_vendor_support shpchp i2c_i801 i2c_core hpilo xfs exportfs qla3xxx cciss hpsa igb dca ptp pps_core megaraid_sas aacraid aic79xx aic7xxx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase arcmsr sata_nv sata_svw 3w_9xxx 3w_xxxx bnx2 forcedeth ext4 jbd2 ext3 jbd ext2 mbcache sata_sil tg3 e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc tun sd_mod crc_t10dif ml
Oct 11 11:44:49 greina15 kernel: x4_core ahci e1000e isci libsas scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: speedstep_lib]
Oct 11 11:44:49 greina15 kernel:

regards
Hussein

Comments

[Gabriele Paciucci (Inactive)]

Do you any idea of the load average of the system during your experiments?

[Hussein N. Harake (Inactive)]

During the load, an estimation of 60 to 85% of CPU load and 80% of memory.

[Hussein N. Harake (Inactive)]

I changed the MDS server with 64GB of memory instead of 32GB, the test passed for the second time without any crash.
If it's the memory, do we have any recommendation for much memory required per MDT?

[Gabriele Paciucci (Inactive)]

Hi Hussein,
is it possible for you to send us the /var/log/messages during the crash or any crash dump if available?

[Gabriele Paciucci (Inactive)]

The memory needed for the MDS depends on the number of clients, the size of the filesystem journal, the number of locks. If we assume that you have 16 clients locking 100k files each, 16 MDT with a 400MB journal each (default)

Operating system overhead = 2 GB
File system journal = 400 MB * 16 = 6.4 GB
16 interactive clients * 100,000 files * 2kB = 3.2GB

Additional RAM is used for caching file data for the working set, which is not actively in use by clients but should be kept "hot" for improved access times. Approximately 1.5 KB per file is needed to keep a file in cache without a lock.

[Gabriele Paciucci (Inactive)]

In our design best practice we suggest for Sandy/Ivy Bridge processor to complete all available memory slots (128GB). We also suggest in modern/high speed/low latency MDT (SSD based) to increase the journal to 4GB.

[Hussein N. Harake (Inactive)]

A snapshot of the /var/log/messages is already in the description, I don't have a crash dump.
I will try to reproduce the problem with a crash dump.

[Peter Jones]

As per CSCS this ticket is ok to close because it no longer occurs since extra memory was added to the affected system