[LU-4091] Kernel Panic MDS jbd2_journal_start+0x4f/0x110 [jbd2] Created: 11/Oct/13  Updated: 20/May/14  Resolved: 20/May/14

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.4.1
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: Hussein N. Harake (Inactive) Assignee: Peter Jones
Resolution: Cannot Reproduce Votes: 0
Labels: None
Environment:

2 sockets, Intel CPU E5-2670 2.60GHz 32GB DDR3 memory1600 MHz
kernel 2.6.32-358.18.1.el6_lustre.x86_64 Scientific Linux 6.4


Severity: 4
Rank (Obsolete): 10997

 Description   

At certain load of benchmarking the metadata performance, the MDS crashes kernel panic, the job is running on 16 MDTs and served by one MDS.

Oct 11 11:44:49 greina15 kernel: BUG: unable to handle kernel NULL pointer dereference at (null)
Oct 11 11:44:49 greina15 kernel: IP: [<ffffffffa049052f>] jbd2_journal_start+0x4f/0x110 [jbd2]
Oct 11 11:44:49 greina15 kernel: PGD 0
Oct 11 11:44:49 greina15 kernel: Oops: 0000 1 SMP
Oct 11 11:44:49 greina15 kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sda/uevent
Oct 11 11:44:49 greina15 kernel: CPU 23
Oct 11 11:44:49 greina15 kernel: Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) mmfs26(U) mmfslinux(U) tracedev(U) limic(U) ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr mlx4_ib ib_sa ib_mad iw_cxgb4 iw_cxgb3 ib_core usb_storage microcode serio_raw rsxx(U) sb_edac edac_core sr_mod cdrom sg iTCO_wdt iTCO_vendor_support shpchp i2c_i801 i2c_core hpilo xfs exportfs qla3xxx cciss hpsa igb dca ptp pps_core megaraid_sas aacraid aic79xx aic7xxx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase arcmsr sata_nv sata_svw 3w_9xxx 3w_xxxx bnx2 forcedeth ext4 jbd2 ext3 jbd ext2 mbcache sata_sil tg3 e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc tun sd_mod crc_t10dif ml
Oct 11 11:44:49 greina15 kernel: x4_core ahci e1000e isci libsas scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: speedstep_lib]
Oct 11 11:44:49 greina15 kernel:

regards
Hussein

 Comments   
Comment by Gabriele Paciucci (Inactive) [ 11/Oct/13 ]

Do you any idea of the load average of the system during your experiments?

Comment by Hussein N. Harake (Inactive) [ 11/Oct/13 ]

During the load, an estimation of 60 to 85% of CPU load and 80% of memory.

Comment by Hussein N. Harake (Inactive) [ 15/Oct/13 ]

I changed the MDS server with 64GB of memory instead of 32GB, the test passed for the second time without any crash.
If it's the memory, do we have any recommendation for much memory required per MDT?

Comment by Gabriele Paciucci (Inactive) [ 15/Oct/13 ]

Hi Hussein,
is it possible for you to send us the /var/log/messages during the crash or any crash dump if available?

Comment by Gabriele Paciucci (Inactive) [ 15/Oct/13 ]

The memory needed for the MDS depends on the number of clients, the size of the filesystem journal, the number of locks. If we assume that you have 16 clients locking 100k files each, 16 MDT with a 400MB journal each (default)

Operating system overhead = 2 GB
File system journal = 400 MB * 16 = 6.4 GB
16 interactive clients * 100,000 files * 2kB = 3.2GB

Additional RAM is used for caching file data for the working set, which is not actively in use by clients but should be kept "hot" for improved access times. Approximately 1.5 KB per file is needed to keep a file in cache without a lock.

Comment by Gabriele Paciucci (Inactive) [ 15/Oct/13 ]

In our design best practice we suggest for Sandy/Ivy Bridge processor to complete all available memory slots (128GB). We also suggest in modern/high speed/low latency MDT (SSD based) to increase the journal to 4GB.

Comment by Hussein N. Harake (Inactive) [ 15/Oct/13 ]

A snapshot of the /var/log/messages is already in the description, I don't have a crash dump.
I will try to reproduce the problem with a crash dump.

Comment by Peter Jones [ 20/May/14 ]

As per CSCS this ticket is ok to close because it no longer occurs since extra memory was added to the affected system

Generated at Sat Feb 10 06:37:33 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.