Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-4111] client kernel panic https://jira.whamcloud.com/browse/LU-4111 Lustre <p>While trying to mount a newly upgraded system ( from 1.8.8 to 2.4.1 ) my lustre client kernel panics.</p> ubuntu precise, with 2.6.32-jb23-358.18.1.el6-lustre-2.4.1 redhat kernel built from source. LU-4111

client kernel panic

Bug Major Resolved Not a Bug Bob Glossman James Beal Wed, 16 Oct 2013 09:04:02 +0000 Thu, 17 Oct 2013 22:18:18 +0000 Thu, 17 Oct 2013 22:18:17 +0000 Lustre 2.4.1 0 7 <p>ct 16 09:00:35 lus03-mds1-1g LustreError: 14385:0:(brw_test.c:334:brw_client_done_rpc()) BRW RPC to 12345-172.17.148.5@tcp failed with -4<br/> Oct 16 09:00:35 lus03-mds1-1g LustreError: 14385:0:(brw_test.c:334:brw_client_done_rpc()) Skipped 8 previous similar messages<br/> Oct 16 09:00:35 lus03-mds1-1g LustreError: 14385:0:(brw_test.c:334:brw_client_done_rpc()) BRW RPC to 12345-172.17.148.5@tcp failed with -4<br/> Oct 16 09:00:35 lus03-mds1-1g LustreError: 14385:0:(brw_test.c:334:brw_client_done_rpc()) Skipped 3 previous similar messages<br/> Oct 16 09:05:32 lus03-mds1-1g LustreError: 166-1: MGC172.17.148.4@tcp: Connection to MGS (at 172.17.148.4@tcp) was lost; in progress operations using this service will fail<br/> Oct 16 09:27:31 lus03-mds1-1g LustreError: 14384:0:(brw_test.c:334:brw_client_done_rpc()) BRW RPC to 12345-172.17.148.5@tcp failed with -4<br/> Oct 16 09:27:31 lus03-mds1-1g LustreError: 14384:0:(brw_test.c:334:brw_client_done_rpc()) Skipped 8 previous similar messages<br/> Oct 16 09:38:06 lus03-mds1-1g LustreError: 18334:0:(lmv_obd.c:1289:lmv_statfs()) can't stat MDS #0 (lus04-MDT0000-mdc-ffff8804a7607000), error -4<br/> Oct 16 09:39:33 lus03-mds1-1g LustreError: 3711:0:(lov_obd.c:937:lov_cleanup()) lov tgt 0 not cleaned! deathrow=0, lovrc=1<br/> Oct 16 09:39:33 lus03-mds1-1g LustreError: 18334:0:(obd_mount.c:1275:lustre_fill_super()) Unable to mount (-4)<br/> Oct 16 09:45:10 lus03-mds1-1g LustreError: 19644:0:(lmv_obd.c:1289:lmv_statfs()) can't stat MDS #0 (lus04-MDT0000-mdc-ffff8804b3657c00), error -4<br/> Oct 16 09:45:10 general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> <br/> Oct 16 09:45:10 SMP <br/> Oct 16 09:45:10 lus03-mds1-1g <br/> Oct 16 09:45:10 last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map<br/> Oct 16 09:45:10 CPU 1 <br/> Oct 16 09:45:10 lus03-mds1-1g <br/> Oct 16 09:45:10 Modules linked in:<br/> Oct 16 09:45:10 lus03-mds1-1g lmv<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g mgc<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g lustre<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g lov<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g osc<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g mdc<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g fid<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g fld<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g ksocklnd<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g ptlrpc<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g obdclass<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g lnet<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g lvfs<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g sha512_generic<br/> Oct 16 09:45:10 lus03-mds1-1g sha256_generic<br/> Oct 16 09:45:10 lus03-mds1-1g libcfs<br/> Oct 16 09:45:10 lus03-mds1-1g (U) <br/> Oct 16 09:45:10 lus03-mds1-1g netconsole<br/> Oct 16 09:45:10 lus03-mds1-1g configfs<br/> Oct 16 09:45:10 lus03-mds1-1g 8021q<br/> Oct 16 09:45:10 lus03-mds1-1g garp<br/> Oct 16 09:45:10 lus03-mds1-1g bridge<br/> Oct 16 09:45:10 lus03-mds1-1g stp<br/> Oct 16 09:45:10 lus03-mds1-1g llc<br/> Oct 16 09:45:10 lus03-mds1-1g acpi_cpufreq<br/> Oct 16 09:45:10 lus03-mds1-1g mperf<br/> Oct 16 09:45:10 lus03-mds1-1g cpufreq_powersave<br/> Oct 16 09:45:10 lus03-mds1-1g cpufreq_ondemand<br/> Oct 16 09:45:10 lus03-mds1-1g cpufreq_conservative<br/> Oct 16 09:45:10 lus03-mds1-1g cpufreq_stats<br/> Oct 16 09:45:10 lus03-mds1-1g freq_table<br/> Oct 16 09:45:10 lus03-mds1-1g autofs4<br/> Oct 16 09:45:10 lus03-mds1-1g dm_round_robin<br/> Oct 16 09:45:10 lus03-mds1-1g nfsd<br/> Oct 16 09:45:10 lus03-mds1-1g exportfs<br/> Oct 16 09:45:10 lus03-mds1-1g nfs<br/> Oct 16 09:45:10 lus03-mds1-1g lockd<br/> Oct 16 09:45:10 lus03-mds1-1g fscache<br/> Oct 16 09:45:10 lus03-mds1-1g auth_rpcgss<br/> Oct 16 09:45:10 lus03-mds1-1g nfs_acl<br/> Oct 16 09:45:10 lus03-mds1-1g sunrpc<br/> Oct 16 09:45:10 lus03-mds1-1g ipv6<br/> Oct 16 09:45:10 lus03-mds1-1g radeon<br/> Oct 16 09:45:10 lus03-mds1-1g ttm<br/> Oct 16 09:45:10 lus03-mds1-1g i5000_edac<br/> Oct 16 09:45:10 lus03-mds1-1g ibmpex<br/> Oct 16 09:45:10 lus03-mds1-1g drm_kms_helper<br/> Oct 16 09:45:10 lus03-mds1-1g edac_core<br/> Oct 16 09:45:10 lus03-mds1-1g ibmaem<br/> Oct 16 09:45:10 lus03-mds1-1g drm<br/> Oct 16 09:45:10 lus03-mds1-1g dm_multipath<br/> Oct 16 09:45:10 lus03-mds1-1g dm_mod<br/> Oct 16 09:45:10 lus03-mds1-1g ioatdma<br/> Oct 16 09:45:10 lus03-mds1-1g i5k_amb<br/> Oct 16 09:45:10 lus03-mds1-1g i2c_algo_bit<br/> Oct 16 09:45:10 lus03-mds1-1g ics932s401<br/> Oct 16 09:45:10 lus03-mds1-1g shpchp<br/> Oct 16 09:45:10 lus03-mds1-1g i2c_core<br/> Oct 16 09:45:10 lus03-mds1-1g dca<br/> Oct 16 09:45:10 lus03-mds1-1g serio_raw<br/> Oct 16 09:45:10 lus03-mds1-1g ext4<br/> Oct 16 09:45:10 lus03-mds1-1g jbd2<br/> Oct 16 09:45:10 lus03-mds1-1g mbcache<br/> Oct 16 09:45:10 lus03-mds1-1g sr_mod<br/> Oct 16 09:45:10 lus03-mds1-1g cdrom<br/> Oct 16 09:45:10 lus03-mds1-1g ata_generic<br/> Oct 16 09:45:10 lus03-mds1-1g ses<br/> Oct 16 09:45:10 lus03-mds1-1g pata_acpi<br/> Oct 16 09:45:10 lus03-mds1-1g sd_mod<br/> Oct 16 09:45:10 lus03-mds1-1g crc_t10dif<br/> Oct 16 09:45:10 lus03-mds1-1g enclosure<br/> Oct 16 09:45:10 lus03-mds1-1g qla2xxx<br/> Oct 16 09:45:10 lus03-mds1-1g scsi_transport_fc<br/> Oct 16 09:45:10 lus03-mds1-1g ata_piix<br/> Oct 16 09:45:10 lus03-mds1-1g bnx2<br/> Oct 16 09:45:10 lus03-mds1-1g scsi_tgt<br/> Oct 16 09:45:10 lus03-mds1-1g bnx2x<br/> Oct 16 09:45:10 lus03-mds1-1g libcrc32c<br/> Oct 16 09:45:10 lus03-mds1-1g mdio<br/> Oct 16 09:45:10 lus03-mds1-1g aacraid<br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[last unloaded: lnet_selftest]</span><br/> Oct 16 09:45:10 lus03-mds1-1g <br/> Oct 16 09:45:10 lus03-mds1-1g <br/> Oct 16 09:45:10 lus03-mds1-1g Pid: 19644, comm: mount.lustre Not tainted 2.6.32-jb23-358.18.1.el6-lustre-2.4.1 #1<br/> Oct 16 09:45:10 lus03-mds1-1g IBM IBM System x3650 <del><span class="error">[7979B9G]</span></del><br/> Oct 16 09:45:10 lus03-mds1-1g /System Planar<br/> Oct 16 09:45:10 lus03-mds1-1g <br/> Oct 16 09:45:10 lus03-mds1-1g RIP: 0010:<span class="error">[<ffffffff81274922>]</span> <br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff81274922>]</span> strlen+0x2/0x20<br/> Oct 16 09:45:10 lus03-mds1-1g RSP: 0018:ffff880465975c90 EFLAGS: 00010246<br/> Oct 16 09:45:10 lus03-mds1-1g RAX: 0000000000000000 RBX: ffff8804b3a6e6c0 RCX: 0000000000000008<br/> Oct 16 09:45:10 lus03-mds1-1g RDX: 0000000000000001 RSI: 0000000000000008 RDI: 5a5a5a5a5a5a5a5a<br/> Oct 16 09:45:10 lus03-mds1-1g RBP: ffff880465975ce8 R08: 0000000000000000 R09: 000000000000006f<br/> Oct 16 09:45:10 lus03-mds1-1g R10: 0000000000000001 R11: 0000000000000000 R12: ffff88045f0e4b80<br/> Oct 16 09:45:10 lus03-mds1-1g R13: 00000000fffffffc R14: ffff88045ee51f80 R15: ffff8804674b42c0<br/> Oct 16 09:45:10 lus03-mds1-1g FS: 00007fc50d598700(0000) GS:ffff880028240000(0000) knlGS:0000000000000000<br/> Oct 16 09:45:10 lus03-mds1-1g CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> Oct 16 09:45:10 lus03-mds1-1g CR2: 00007fff09fc6eb8 CR3: 000000045f4e6000 CR4: 00000000000407e0<br/> Oct 16 09:45:10 lus03-mds1-1g DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> Oct 16 09:45:10 lus03-mds1-1g DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Oct 16 09:45:10 Process mount.lustre (pid: 19644, threadinfo ffff880465974000, task ffff8804b4c93500)<br/> Oct 16 09:45:10 lus03-mds1-1g Stack: <br/> Oct 16 09:45:10 lus03-mds1-1g ffffffffa0c0f3e6<br/> Oct 16 09:45:10 lus03-mds1-1g ffff8804a4cb12e0<br/> Oct 16 09:45:10 lus03-mds1-1g ffff8804b3657c00<br/> Oct 16 09:45:10 lus03-mds1-1g ffff880467438000<br/> Oct 16 09:45:10 lus03-mds1-1g <br/> Oct 16 09:45:10 lus03-mds1-1g d> <br/> Oct 16 09:45:10 lus03-mds1-1g ffff880464494800<br/> Oct 16 09:45:10 lus03-mds1-1g ffff880465975ce8<br/> Oct 16 09:45:10 lus03-mds1-1g 0000000000000000<br/> Oct 16 09:45:10 lus03-mds1-1g ffff8804b3a6ed40<br/> Oct 16 09:45:10 lus03-mds1-1g <br/> Oct 16 09:45:10 lus03-mds1-1g d> <br/> Oct 16 09:45:10 lus03-mds1-1g ffff8804b51fa000<br/> Oct 16 09:45:10 lus03-mds1-1g ffff8804a4cb12e0<br/> Oct 16 09:45:10 lus03-mds1-1g ffff8804b3657c00<br/> Oct 16 09:45:10 lus03-mds1-1g ffff880465975d88<br/> Oct 16 09:45:10 lus03-mds1-1g <br/> Oct 16 09:45:10 Call Trace: <br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffffa0c0f3e6>]</span> ? ll_fill_super+0xd96/0x15b0 <span class="error">[lustre]</span><br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffffa06dec81>]</span> lustre_fill_super+0x771/0x24c0 <span class="error">[obdclass]</span><br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff8117b843>]</span> ? sget+0x3a3/0x410<br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffffa06de510>]</span> ? lustre_fill_super+0x0/0x24c0 <span class="error">[obdclass]</span><br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff8117b90f>]</span> get_sb_nodev+0x5f/0xa0<br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffffa06d7955>]</span> lustre_get_sb+0x25/0x30 <span class="error">[obdclass]</span><br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff8117b214>]</span> vfs_kern_mount+0x74/0x1c0<br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff8117b3d4>]</span> do_kern_mount+0x54/0x120<br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff8119a9eb>]</span> do_mount+0x1cb/0x930<br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff81158833>]</span> ? alloc_pages_current+0xa3/0x110<br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff8111b19e>]</span> ? __get_free_pages+0xe/0x50<br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff8119a69a>]</span> ? copy_mount_options+0x3a/0x170<br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff8119b450>]</span> sys_mount+0x90/0xe0<br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff8100b072>]</span> system_call_fastpath+0x16/0x1b<br/> Oct 16 09:45:10 lus03-mds1-1g Code: <br/> Oct 16 09:45:10 48 <br/> Oct 16 09:45:10 89 <br/> Oct 16 09:45:10 e5 <br/> Oct 16 09:45:10 f6 <br/> Oct 16 09:45:10 82 <br/> Oct 16 09:45:10 80 <br/> Oct 16 09:45:10 11 <br/> Oct 16 09:45:10 af <br/> Oct 16 09:45:10 81 <br/> Oct 16 09:45:10 20 <br/> Oct 16 09:45:10 74 <br/> Oct 16 09:45:10 15 <br/> Oct 16 09:45:10 0f <br/> Oct 16 09:45:10 1f <br/> Oct 16 09:45:10 44 <br/> Oct 16 09:45:10 00 <br/> Oct 16 09:45:10 00 <br/> Oct 16 09:45:10 48 <br/> Oct 16 09:45:10 83 <br/> Oct 16 09:45:10 c0 <br/> Oct 16 09:45:10 01 <br/> Oct 16 09:45:10 0f <br/> Oct 16 09:45:10 b6 <br/> Oct 16 09:45:10 10 <br/> Oct 16 09:45:10 f6 <br/> Oct 16 09:45:10 82 <br/> Oct 16 09:45:10 80 <br/> Oct 16 09:45:10 11 <br/> Oct 16 09:45:10 af <br/> Oct 16 09:45:10 81 <br/> Oct 16 09:45:10 20 <br/> Oct 16 09:45:10 75 <br/> Oct 16 09:45:10 f0 <br/> Oct 16 09:45:10 5d <br/> Oct 16 09:45:10 c3 <br/> Oct 16 09:45:10 66 <br/> Oct 16 09:45:10 0f <br/> Oct 16 09:45:10 1f <br/> Oct 16 09:45:10 44 <br/> Oct 16 09:45:10 00 <br/> Oct 16 09:45:10 00 <br/> Oct 16 09:45:10 31 <br/> Oct 16 09:45:10 c0 <br/> Oct 16 09:45:10 lus03-mds1-1g <br/> Oct 16 09:45:10 3f <br/> Oct 16 09:45:10 00 <br/> Oct 16 09:45:10 55 <br/> Oct 16 09:45:10 48 <br/> Oct 16 09:45:10 89 <br/> Oct 16 09:45:10 e5 <br/> Oct 16 09:45:10 74 <br/> Oct 16 09:45:10 11 <br/> Oct 16 09:45:10 48 <br/> Oct 16 09:45:10 89 <br/> Oct 16 09:45:10 f8 <br/> Oct 16 09:45:10 66 <br/> Oct 16 09:45:10 90 <br/> Oct 16 09:45:10 48 <br/> Oct 16 09:45:10 83 <br/> Oct 16 09:45:10 c0 <br/> Oct 16 09:45:10 01 <br/> Oct 16 09:45:10 80 <br/> Oct 16 09:45:10 38 <br/> Oct 16 09:45:10 00 <br/> Oct 16 09:45:10 lus03-mds1-1g <br/> Oct 16 09:45:10 RIP <br/> Oct 16 09:45:10 lus03-mds1-1g <span class="error">[<ffffffff81274922>]</span> strlen+0x2/0x20<br/> Oct 16 09:45:10 lus03-mds1-1g RSP <ffff880465975c90><br/> Oct 16 09:45:11 lus03-mds1-1g --<del>[ end trace 0a0429a1702b83ae ]</del>--<br/> Oct 16 09:45:11 Kernel panic - not syncing: Fatal exception<br/> Oct 16 09:45:11 lus03-mds1-1g Pid: 19644, comm: mount.lustre Tainted: G D --------------- 2.6.32-jb23-358.18.1.el6-lustre-2.4.1 #1<br/> Oct 16 09:45:11 Call Trace: <br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff81501d57>]</span> ? panic+0xa7/0x167<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8150d0c4>]</span> ? oops_end+0xe4/0x100<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8100f438>]</span> ? die+0x58/0x90<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8150c932>]</span> ? do_general_protection+0x152/0x160<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8150c3a5>]</span> ? general_protection+0x25/0x30<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff81274922>]</span> ? strlen+0x2/0x20<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffffa0c0f3e6>]</span> ? ll_fill_super+0xd96/0x15b0 <span class="error">[lustre]</span><br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffffa06dec81>]</span> ? lustre_fill_super+0x771/0x24c0 <span class="error">[obdclass]</span><br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8117b843>]</span> ? sget+0x3a3/0x410<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffffa06de510>]</span> ? lustre_fill_super+0x0/0x24c0 <span class="error">[obdclass]</span><br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8117b90f>]</span> ? get_sb_nodev+0x5f/0xa0<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffffa06d7955>]</span> ? lustre_get_sb+0x25/0x30 <span class="error">[obdclass]</span><br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8117b214>]</span> ? vfs_kern_mount+0x74/0x1c0<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8117b3d4>]</span> ? do_kern_mount+0x54/0x120<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8119a9eb>]</span> ? do_mount+0x1cb/0x930<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff81158833>]</span> ? alloc_pages_current+0xa3/0x110<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8111b19e>]</span> ? __get_free_pages+0xe/0x50<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8119a69a>]</span> ? copy_mount_options+0x3a/0x170<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8119b450>]</span> ? sys_mount+0x90/0xe0<br/> Oct 16 09:45:11 lus03-mds1-1g <span class="error">[<ffffffff8100b072>]</span> ? system_call_fastpath+0x16/0x1b<br/> Oct 16 09:45:11 panic occurred, switching back to text console<br/> root@lus03-mds2:~# uname -a<br/> Linux lus03-mds2 2.6.32-jb23-358.18.1.el6-lustre-2.4.1 #1 SMP Fri Sep 20 10:06:06 BST 2013 x86_64 x86_64 x86_64 GNU/Linux<br/> root@lus03-mds2:~# </p> <p>We have one lustre 2.4.1 server system running in production with 1.8.9wc1 clients. We are trying to use a previous system now as test as a lustre 2.4.1 client. </p> <p>This system can mount our working 2.4.1 server system however trying an ls always fail.</p> <p>Oct 16 13:12:32 lus03-mds1-1g BUG: unable to handle kernel <br/> Oct 16 13:12:32 paging request <br/> Oct 16 13:12:32 lus03-mds1-1g at 0000082d00000019<br/> Oct 16 13:12:32 lus03-mds1-1g IP: <br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffffa0c033ff>]</span> ll_show_options+0x2f/0x180 <span class="error">[lustre]</span><br/> Oct 16 13:12:32 PGD 0 <br/> Oct 16 13:12:32 lus03-mds1-1g <br/> Oct 16 13:12:32 lus03-mds1-1g Oops: 0000 <a href="#1" target="_blank" rel="noopener">1</a> <br/> Oct 16 13:12:32 SMP <br/> Oct 16 13:12:32 lus03-mds1-1g <br/> Oct 16 13:12:32 last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:1a:00.0/0000:1b:01.0/0000:24:00.0/host5/rport-5:0-2/target5:0:1/5:0:1:0/state<br/> Oct 16 13:12:32 CPU 5 <br/> Oct 16 13:12:32 lus03-mds1-1g <br/> Oct 16 13:12:32 Modules linked in:<br/> Oct 16 13:12:32 lus03-mds1-1g mgc<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g lustre<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g lov<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g osc<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g mdc<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g fid<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g fld<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g ksocklnd<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g ptlrpc<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g obdclass<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g lnet<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g lvfs<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g sha512_generic<br/> Oct 16 13:12:32 lus03-mds1-1g sha256_generic<br/> Oct 16 13:12:32 lus03-mds1-1g libcfs<br/> Oct 16 13:12:32 lus03-mds1-1g (U) <br/> Oct 16 13:12:32 lus03-mds1-1g netconsole<br/> Oct 16 13:12:32 lus03-mds1-1g configfs<br/> Oct 16 13:12:32 lus03-mds1-1g 8021q<br/> Oct 16 13:12:32 lus03-mds1-1g garp<br/> Oct 16 13:12:32 lus03-mds1-1g bridge<br/> Oct 16 13:12:32 lus03-mds1-1g stp<br/> Oct 16 13:12:32 lus03-mds1-1g llc<br/> Oct 16 13:12:32 lus03-mds1-1g acpi_cpufreq<br/> Oct 16 13:12:32 lus03-mds1-1g mperf<br/> Oct 16 13:12:32 lus03-mds1-1g cpufreq_powersave<br/> Oct 16 13:12:32 lus03-mds1-1g cpufreq_ondemand<br/> Oct 16 13:12:32 lus03-mds1-1g cpufreq_conservative<br/> Oct 16 13:12:32 lus03-mds1-1g cpufreq_stats<br/> Oct 16 13:12:32 lus03-mds1-1g freq_table<br/> Oct 16 13:12:32 lus03-mds1-1g autofs4<br/> Oct 16 13:12:32 lus03-mds1-1g nfsd<br/> Oct 16 13:12:32 lus03-mds1-1g exportfs<br/> Oct 16 13:12:32 lus03-mds1-1g nfs<br/> Oct 16 13:12:32 lus03-mds1-1g lockd<br/> Oct 16 13:12:32 lus03-mds1-1g fscache<br/> Oct 16 13:12:32 lus03-mds1-1g auth_rpcgss<br/> Oct 16 13:12:32 lus03-mds1-1g nfs_acl<br/> Oct 16 13:12:32 lus03-mds1-1g sunrpc<br/> Oct 16 13:12:32 lus03-mds1-1g ipv6<br/> Oct 16 13:12:32 lus03-mds1-1g dm_round_robin<br/> Oct 16 13:12:32 lus03-mds1-1g radeon<br/> Oct 16 13:12:32 lus03-mds1-1g ttm<br/> Oct 16 13:12:32 lus03-mds1-1g drm_kms_helper<br/> Oct 16 13:12:32 lus03-mds1-1g i5000_edac<br/> Oct 16 13:12:32 lus03-mds1-1g drm<br/> Oct 16 13:12:32 lus03-mds1-1g edac_core<br/> Oct 16 13:12:32 lus03-mds1-1g i2c_algo_bit<br/> Oct 16 13:12:32 lus03-mds1-1g ibmpex<br/> Oct 16 13:12:32 lus03-mds1-1g ibmaem<br/> Oct 16 13:12:32 lus03-mds1-1g dm_multipath<br/> Oct 16 13:12:32 lus03-mds1-1g ics932s401<br/> Oct 16 13:12:32 lus03-mds1-1g shpchp<br/> Oct 16 13:12:32 lus03-mds1-1g dm_mod<br/> Oct 16 13:12:32 lus03-mds1-1g ioatdma<br/> Oct 16 13:12:32 lus03-mds1-1g i5k_amb<br/> Oct 16 13:12:32 lus03-mds1-1g serio_raw<br/> Oct 16 13:12:32 lus03-mds1-1g i2c_core<br/> Oct 16 13:12:32 lus03-mds1-1g dca<br/> Oct 16 13:12:32 lus03-mds1-1g ext4<br/> Oct 16 13:12:32 lus03-mds1-1g jbd2<br/> Oct 16 13:12:32 lus03-mds1-1g mbcache<br/> Oct 16 13:12:32 lus03-mds1-1g sr_mod<br/> Oct 16 13:12:32 lus03-mds1-1g cdrom<br/> Oct 16 13:12:32 lus03-mds1-1g ata_generic<br/> Oct 16 13:12:32 lus03-mds1-1g pata_acpi<br/> Oct 16 13:12:32 lus03-mds1-1g ses<br/> Oct 16 13:12:32 lus03-mds1-1g sd_mod<br/> Oct 16 13:12:32 lus03-mds1-1g enclosure<br/> Oct 16 13:12:32 lus03-mds1-1g crc_t10dif<br/> Oct 16 13:12:32 lus03-mds1-1g ata_piix<br/> Oct 16 13:12:32 lus03-mds1-1g bnx2x<br/> Oct 16 13:12:32 lus03-mds1-1g qla2xxx<br/> Oct 16 13:12:32 lus03-mds1-1g scsi_transport_fc<br/> Oct 16 13:12:32 lus03-mds1-1g libcrc32c<br/> Oct 16 13:12:32 lus03-mds1-1g bnx2<br/> Oct 16 13:12:32 lus03-mds1-1g mdio<br/> Oct 16 13:12:32 lus03-mds1-1g scsi_tgt<br/> Oct 16 13:12:32 lus03-mds1-1g aacraid<br/> Oct 16 13:12:32 lus03-mds1-1g <br/> Oct 16 13:12:32 lus03-mds1-1g <br/> Oct 16 13:12:32 lus03-mds1-1g Pid: 17711, comm: ls Not tainted 2.6.32-jb23-358.18.1.el6-lustre-2.4.1 #1<br/> Oct 16 13:12:32 lus03-mds1-1g IBM IBM System x3650 <del><span class="error">[7979B9G]</span></del><br/> Oct 16 13:12:32 lus03-mds1-1g /System Planar<br/> Oct 16 13:12:32 lus03-mds1-1g <br/> Oct 16 13:12:32 lus03-mds1-1g RIP: 0010:<span class="error">[<ffffffffa0c033ff>]</span> <br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffffa0c033ff>]</span> ll_show_options+0x2f/0x180 <span class="error">[lustre]</span><br/> Oct 16 13:12:32 lus03-mds1-1g RSP: 0018:ffff8804b5375e18 EFLAGS: 00010282<br/> Oct 16 13:12:32 lus03-mds1-1g RAX: 0000082d00000001 RBX: ffff88048e0b4380 RCX: 0000000000000000<br/> Oct 16 13:12:32 lus03-mds1-1g RDX: 0000000000000000 RSI: ffff8804b65da8c0 RDI: ffff88048e0b4380<br/> Oct 16 13:12:32 lus03-mds1-1g RBP: ffff8804b5375e28 R08: ffff8804b5375d78 R09: ffffffff8179faa9<br/> Oct 16 13:12:32 lus03-mds1-1g R10: 6e2c38363732333d R11: 0000000000000246 R12: ffff8804b65da928<br/> Oct 16 13:12:32 lus03-mds1-1g R13: ffff8804b65da8c0 R14: 0000000000000000 R15: 000000000000009c<br/> Oct 16 13:12:32 lus03-mds1-1g FS: 00007feefeaaa7c0(0000) GS:ffff880028340000(0000) knlGS:0000000000000000<br/> Oct 16 13:12:32 lus03-mds1-1g CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br/> Oct 16 13:12:32 lus03-mds1-1g CR2: 0000082d00000019 CR3: 00000004ac991000 CR4: 00000000000407e0<br/> Oct 16 13:12:32 lus03-mds1-1g DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> Oct 16 13:12:32 lus03-mds1-1g DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Oct 16 13:12:32 Process ls (pid: 17711, threadinfo ffff8804b5374000, task ffff8804b3e36040)<br/> Oct 16 13:12:32 lus03-mds1-1g Stack: <br/> Oct 16 13:12:32 lus03-mds1-1g ffff88048e0b4380<br/> Oct 16 13:12:32 lus03-mds1-1g ffff8804b65da928<br/> Oct 16 13:12:32 lus03-mds1-1g ffff8804b5375e68<br/> Oct 16 13:12:32 lus03-mds1-1g ffffffff81198965<br/> Oct 16 13:12:32 lus03-mds1-1g <br/> Oct 16 13:12:32 lus03-mds1-1g d> <br/> Oct 16 13:12:32 lus03-mds1-1g ffff8804b65da8c0<br/> Oct 16 13:12:32 lus03-mds1-1g ffff8804a9398e00<br/> Oct 16 13:12:32 lus03-mds1-1g ffff8804b3a85200<br/> Oct 16 13:12:32 lus03-mds1-1g ffff88048e0b4380<br/> Oct 16 13:12:32 lus03-mds1-1g <br/> Oct 16 13:12:32 lus03-mds1-1g d> <br/> Oct 16 13:12:32 lus03-mds1-1g ffff8804b65da928<br/> Oct 16 13:12:32 lus03-mds1-1g 00007feefeac739d<br/> Oct 16 13:12:32 lus03-mds1-1g ffff8804b5375ee8<br/> Oct 16 13:12:32 lus03-mds1-1g ffffffff8119c1ab<br/> Oct 16 13:12:32 lus03-mds1-1g <br/> Oct 16 13:12:32 Call Trace: <br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff81198965>]</span> show_vfsmnt+0x105/0x120<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff8119c1ab>]</span> seq_read+0x12b/0x3b0<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff81212136>]</span> ? security_file_permission+0x16/0x20<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff811792d5>]</span> vfs_read+0xa5/0x180<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff811793fa>]</span> sys_read+0x4a/0x90<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff8100b072>]</span> system_call_fastpath+0x16/0x1b<br/> Oct 16 13:12:32 lus03-mds1-1g Code: <br/> Oct 16 13:12:32 41 <br/> Oct 16 13:12:32 54 <br/> Oct 16 13:12:32 53 <br/> Oct 16 13:12:32 0f <br/> Oct 16 13:12:32 1f <br/> Oct 16 13:12:32 44 <br/> Oct 16 13:12:32 00 <br/> Oct 16 13:12:32 00 <br/> Oct 16 13:12:32 48 <br/> Oct 16 13:12:32 85 <br/> Oct 16 13:12:32 f6 <br/> Oct 16 13:12:32 48 <br/> Oct 16 13:12:32 89 <br/> Oct 16 13:12:32 fb <br/> Oct 16 13:12:32 0f <br/> Oct 16 13:12:32 84 <br/> Oct 16 13:12:32 2e <br/> Oct 16 13:12:32 01 <br/> Oct 16 13:12:32 00 <br/> Oct 16 13:12:32 00 <br/> Oct 16 13:12:32 48 <br/> Oct 16 13:12:32 85 <br/> Oct 16 13:12:32 ff <br/> Oct 16 13:12:32 0f <br/> Oct 16 13:12:32 84 <br/> Oct 16 13:12:32 25 <br/> Oct 16 13:12:32 01 <br/> Oct 16 13:12:32 00 <br/> Oct 16 13:12:32 00 <br/> Oct 16 13:12:32 48 <br/> Oct 16 13:12:32 8b <br/> Oct 16 13:12:32 86 <br/> Oct 16 13:12:32 90 <br/> Oct 16 13:12:32 00 <br/> Oct 16 13:12:32 00 last message repeated 2 times<br/> Oct 16 13:12:32 48 <br/> Oct 16 13:12:32 8b <br/> Oct 16 13:12:32 80 <br/> Oct 16 13:12:32 90 <br/> Oct 16 13:12:32 02 <br/> Oct 16 13:12:32 00 <br/> Oct 16 13:12:32 00 <br/> Oct 16 13:12:32 lus03-mds1-1g c> <br/> Oct 16 13:12:32 8b <br/> Oct 16 13:12:32 60 <br/> Oct 16 13:12:32 18 <br/> Oct 16 13:12:32 41 <br/> Oct 16 13:12:32 8b <br/> Oct 16 13:12:32 44 <br/> Oct 16 13:12:32 24 <br/> Oct 16 13:12:32 70 <br/> Oct 16 13:12:32 a8 <br/> Oct 16 13:12:32 01 <br/> Oct 16 13:12:32 0f <br/> Oct 16 13:12:32 85 <br/> Oct 16 13:12:32 f0 <br/> Oct 16 13:12:32 00 <br/> Oct 16 13:12:32 00 last message repeated 2 times<br/> Oct 16 13:12:32 a8 <br/> Oct 16 13:12:32 04 <br/> Oct 16 13:12:32 0f <br/> Oct 16 13:12:32 85 <br/> Oct 16 13:12:32 lus03-mds1-1g <br/> Oct 16 13:12:32 RIP <br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffffa0c033ff>]</span> ll_show_options+0x2f/0x180 <span class="error">[lustre]</span><br/> Oct 16 13:12:32 lus03-mds1-1g RSP <ffff8804b5375e18><br/> Oct 16 13:12:32 lus03-mds1-1g CR2: 0000082d00000019<br/> Oct 16 13:12:32 lus03-mds1-1g --<del>[ end trace a3fb76990639f714 ]</del>--<br/> Oct 16 13:12:32 Kernel panic - not syncing: Fatal exception<br/> Oct 16 13:12:32 lus03-mds1-1g Pid: 17711, comm: ls Tainted: G D --------------- 2.6.32-jb23-358.18.1.el6-lustre-2.4.1 #1<br/> Oct 16 13:12:32 Call Trace: <br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff81501d57>]</span> ? panic+0xa7/0x167<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff8150d0c4>]</span> ? oops_end+0xe4/0x100<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff81501655>]</span> ? no_context+0x209/0x218<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff811ded50>]</span> ? proc_delete_inode+0x0/0x60<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff815017e5>]</span> ? __bad_area_nosemaphore+0x181/0x1a4<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff8111463e>]</span> ? find_get_page+0x1e/0xa0<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff81501862>]</span> ? bad_area+0x45/0x4e<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff81044be5>]</span> ? __do_page_fault+0x345/0x430<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff81275dfd>]</span> ? string.isra.3+0x3d/0xf0<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff8150f0eb>]</span> ? do_page_fault+0x3b/0xa0<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff8150c3d5>]</span> ? page_fault+0x25/0x30<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffffa0c033ff>]</span> ? ll_show_options+0x2f/0x180 <span class="error">[lustre]</span><br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff81198965>]</span> ? show_vfsmnt+0x105/0x120<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff8119c1ab>]</span> ? seq_read+0x12b/0x3b0<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff81212136>]</span> ? security_file_permission+0x16/0x20<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff811792d5>]</span> ? vfs_read+0xa5/0x180<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff811793fa>]</span> ? sys_read+0x4a/0x90<br/> Oct 16 13:12:32 lus03-mds1-1g <span class="error">[<ffffffff8100b072>]</span> ? system_call_fastpath+0x16/0x1b<br/> root@lus03-mds2:~# </p> <p>In the first crash it looks like you were running lnet_selftest on the client before it had mount troubles? Is there a problem mounting if you don't run this first?</p> <p>In the second crash, it looks like it crashes in ll_show_options() where sbi is set, but it may be slightly different since you compiled your own Lustre code:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(gdb) list *(ll_show_options + 0x2f) 0x2ee0f is in ll_show_options (/usr/src/lustre-head/lustre/llite/llite_lib.c:2371). 2366 struct ll_sb_info *sbi; 2367 2368 #ifdef HAVE_SUPEROPS_USE_DENTRY 2369 LASSERT((seq != NULL) && (dentry != NULL)); 2370 sbi = ll_s2sbi(dentry->d_sb); 2371 #else 2372 LASSERT((seq != NULL) && (vfs != NULL)); 2373 sbi = ll_s2sbi(vfs->mnt_sb); 2374 #endif </pre> </div></div> <p>It isn't at all clear why this would be crashing, since we've of course tested 2.4.1 clients+servers together. Is this a newly formatted filesystem, or is the server also upgraded? Did you try writeconf to rebuild the config logs for 2.4?</p> <p>Bob, can you please take a look at this?</p> <p>"In the first crash it looks like you were running lnet_selftest on the client before it had mount troubles? Is there a problem mounting if you don't run this first?"</p> <p>No, the problem is independent of running a lnet_selftest, We just had a system which we were having real problems with and we were proving the network was not an issue. ( We are moving towards thinking that the problem was we had done a writeconf and not specified an --mgs in the tunefs.lustre ).</p> <p>"It isn't at all clear why this would be crashing, since we've of course tested 2.4.1 clients+servers together. Is this a newly formatted filesystem, or is the server also upgraded? Did you try writeconf to rebuild the config logs for 2.4?"</p> <p>Note this was a client system that hadn't either an MGS,MDT or OST mounted on it before being used as a client. We had gone though the loop of using writeconf multiple times on the servers serving the file system we were mounting.</p> <p>We do have a in production 2.4.1 filesystem however while I can mount this from a 2.4.1 client ( either a system installed but not used as a server or from one of our dkms builds) both clients fail if I try and do an ls, I suspect that should be a separate issue but I have been trying to rule out most of my stupidity.</p> <p>Could you please attach the .config from your kernel build and the config.h from your lustre build? I'd like to look for anomalies or departures from expected values there.</p> <p>Kernel config file</p> <p>config.h</p> <p>I think I've spotted at least one anomaly in your lustre build. Your config.h has</p> <p>#define HAVE_SUPEROPS_USE_DENTRY 1</p> <p>Normally building against rhel/centos 2.6.32-358.18.1 kernel source this is #undef'ed. As far as I know this is controlled by the definition of struct super_operations.show_options in include/linux/fs.h of the kernel source. In standard red hat kernel source of this version the 2nd argument of that method is a struct vfsmount *, not a struct dentry *. The fact that your config doesn't match the includes in proper kernel source suggest that something is wrong in your configure or build of lustre.</p> <p>Normally we only build lustre clients native on ubuntu. We don't build server code there. I'm wondering if there's something about building rhel kernel or lustre server code in a ubuntu environment that is just not working right. Or maybe you have bad options in the configure cmd of your lustre build.</p> <p>My notes on building the kernel on ubuntu precise are...</p> <p>Edit arch/x86/vdso/Makefile replace</p> <p>#VDSO_LDFLAGS_vdso.lds = -m elf_x86_64 -Wl,-soname=linux-vdso.so.1 \<br/> -Wl,-z,max-page-size=4096 -Wl,-z,common-page-size=4096<br/> VDSO_LDFLAGS_vdso.lds = -m64 -Wl,-soname=linux-vdso.so.1 \<br/> -Wl,-z,max-page-size=4096 -Wl,-z,common-page-size=4096</p> <p>and</p> <ol> <li>VDSO_LDFLAGS_vdso32.lds = -m elf_x86 -Wl,-soname=linux-gate.so.1<br/> VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-soname=linux-gate.so.1</li> </ol> <p>( Because of <a href="http://stackoverflow.com/questions/10425761/kernel-compile-error-gcc-error-elf-i386-no-such-file-or-directory" class="external-link" target="_blank" rel="nofollow noopener">http://stackoverflow.com/questions/10425761/kernel-compile-error-gcc-error-elf-i386-no-such-file-or-directory</a> )</p> <p>Edit Makefile replace, as gcc 4.6 does not parse....</p> <p>CPP_MINOR := $(shell $(CPP) -dumpversion 2>&1 | cut -d'.' -f2)<br/> CPP_PATCH := $(shell $(CPP) -dumpversion 2>&1 | cut -d'.' -f3)<br/> with<br/> CPP_MINOR := $(shell $(CPP) -dumpversion 2>&1 | cut -d'.' -f2)<br/> CPP_PATCH := 0</p> <p>gcc -v<br/> Using built-in specs.<br/> COLLECT_GCC=gcc<br/> COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.6/lto-wrapper<br/> Target: x86_64-linux-gnu<br/> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.3-1ubuntu5' --with-bugurl=<a href="file:///usr/share/doc/gcc-4.6/README.Bugs" class="external-link" target="_blank" rel="nofollow noopener">file:///usr/share/doc/gcc-4.6/README.Bugs</a> --enable-languages=c,c+<ins>,fortran,objc,obj-c</ins>+ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu<br/> Thread model: posix<br/> gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) </p> <p>cpp -dumpversion<br/> 4.6</p> <p>And finally</p> <p>Compilation failures....<br/> CONFIG_SCSI_PMCRAID=n<br/> CC <span class="error">[M]</span> drivers/scsi/pmcraid.o<br/> In file included from drivers/scsi/pmcraid.c:57:0:<br/> drivers/scsi/pmcraid.h:611:8: error: duplicate member ‘sense_buffer’</p> <p>cd <kernel-source><br/> cp <lustre-source>/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.conf ./.config<br/> make oldconfig <br/> make kpkg-clean<br/> CONCURRENCY_LEVEL=9 make-kpkg kernel-image kernel-headers kernel-source --append-to-version="-jb23-358.18.1.el6-lustre-2.4.1" --revision=20130920:1 --rootcmd fakeroot --initrd</p> <p>"Normally building against rhel/centos 2.6.32-358.18.1 kernel source this is #undef'ed. As far as I know this is controlled by the definition of struct super_operations.show_options in include/linux/fs.h of the kernel source. In standard red hat kernel source of this version the 2nd argument of that method is a struct vfsmount *, not a struct dentry *. The fact that your config doesn't match the includes in proper kernel source suggest that something is wrong in your configure or build of lustre."</p> <p>The kernel we get seems to be working as a server. However it seems to fail as a client <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Taking the stupid approach of looking show_options in fs.h we get</p> <p>root@isg-dev4-1g:/local/lustre_2.4.1/kernel_source/kernel-2.6.32-358.18.1.el6/linux-2.6.32-358.18.1.el6.x86_64/include/linux# grep -n show_options fs.h <br/> 1484: * generic_show_options()<br/> 1789: int (*show_options)(struct seq_file *, struct vfsmount *);<br/> 2657:extern int generic_show_options(struct seq_file *m, struct vfsmount *mnt);</p> <p>md5sum fs.h <br/> 4068f8d7a42f6e796049b36ddd076a60 fs.h</p> <p>I more strongly suspect the lustre build, not the kernel build. I note that in the kernel #includes for the 3.8 kernel in the Ubuntu distro the 2nd arg of super_operations.show_options is in fact a struct dentry *. If you are somehow seeing the ubuntu native #includes instead of the rhel kernel #includes during the lustre build that would be a complete explanation of what went wrong.</p> <p>which ubuntu release is 'precise'? I think the only ubuntu I have on hand to look at is a bit newer than that.</p> <p>Precise is Ubuntu 12.04.2 LTS ( <a href="http://releases.ubuntu.com/precise/" class="external-link" target="_blank" rel="nofollow noopener">http://releases.ubuntu.com/precise/</a> ).</p> <p>"I note that in the kernel #includes for the 3.8 kernel in the Ubuntu distro the 2nd arg of super_operations.show_options is in fact a struct dentry *. If you are somehow seeing the ubuntu native #includes instead of the rhel kernel #includes during the lustre build that would be a complete explanation of what went wrong."</p> <p>I note that I did the lustre build while running 3.8.0-31-generic, its perfectly possible that I have picked up the native includes..., my config stanza looks like:</p> <p>It was created by Lustre configure LUSTRE_VERSION, which was<br/> generated by GNU Autoconf 2.68. Invocation command line was</p> <p> $ ./configure --enable-mpitests=no --with-o2ib=/usr/src/ofa-kernel-headers-2.6.32-jb23-358.18.1.el6-lustre-2.4.1/ --with-linux=/local/lustre_2.4.1/kernel_source/kernel-2.6.32-358.<br/> 18.1.el6/linux-2.6.32-358.18.1.el6.x86_64/</p> <p>Which seems reasonable..., however looking at the config.log I am not so sure, I will attached it.</p> <p>config.log</p> <p>It seems to be a build environment or Makefile thing. In centos builds native on Centos all the autoconf compile tests have -Werror in the command lines captured in config.log. In your config.log none of the compile test lines have -Werror in them. This leads to compile tests that should fail due to warnings about mismatched types being promoted to hard errors succeeding instead. Can't tell if this is due to some subtle version skew in autoconf or automake tools on ubuntu or due to some differences in Makefiles for kernel or lustre or what. All I know is it is different in your config.log than in builds on Centos or SLES or RHEL.</p> <p>Can you describe how you obtained & installed rhel kernel source in ubuntu? My ubuntu environment doesn't have rpm tools, for example.</p> <p>Would like to follow along in your footsteps and see if I can cause similar build errors.</p> <p>I find if I do a client build in ubuntu against the native kernel all the autoconf compile lines in config.log do have -Werror in them as they probably should. Don't know what is different about the builds you are doing.</p> <p>The way we get the kernel is to get a centos vm...</p> <p>rpm -ivh <kernel-package><br/> cd ~/rpmbuild<br/> rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec</p> <p>This is our latest method for getting the lustre source where you can see we explicitly turn off -Werror ...</p> <p>cd lustre-release<br/> git checkout <tag></p> <ol> <li>git tag -l will show the available tags</li> <li>Continue to do this on the same machine, do not move the git checkout....</li> <li>tidy up the source ready for building.</li> <li>Remove -Werror from the config<br/> cd <lustre-source><br/> sed -i <b>/autoconf/</b>.m4 -e 's/-Werror//g' -e 's/Werror-implicit-function-declaration//'<br/> sh ./autogen.sh<br/> ./configure --disable-modules<br/> make dist</li> </ol> <p>This will generate a lustre-X.tar.gz file. This is the source tarball you should for the subsequent stages in the build. I will discuss with my colleagues in the morning the reason why we started ensuring that Werror is not set as I can't remember right now....</p> <p>I strongly suspect taking out -Werror is the source of your build problems. Much of our autoconf test infrastructure depends on having that option. I'm pretty sure the HAVE_SUPEROPS_USE_DENTRY is not the only config variable that's wrong due to stripping it out, it's probably only the tip of the iceberg.</p> <p>Removing -Werror was the problem; </p> <p>Putting it back give us a client that appears to work. I used </p> <p>#pragma GCC diagnostic</p> <p>to silence the compiler warnings generated by by gcc 4.6 that were causing the build to fail.</p> <p>I'm curious why you are working so hard to create and run a rhel kernel on ubuntu? Why not just use a centos or vanilla rhel server instead? We don't normally build for or support ubuntu servers, only clients.</p> <p>By the way we have a number of small changes that went in after the 2.4 release and aren't in the b2_4 branch of our code that fix problems with errors seen in late gcc versions like 4.6 and 4.7. Not normally seen during 2.4 builds as the native gcc version in rhel/centos 6.4 is 4.4</p> <p>The theory was that is was easier to port the rhel kernel -> ubuntu than to fold new centos servers into our server mgmt and provisioning infrastructure (which currently handles ubuntu & SLES, but not RHEL.)</p> <p>That assumption is under active re-consideration <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>If it's any consolation we have started building for and supporting SLES11 servers again. Both SLES11 SP2 and SP3 server builds are in the 2.5 release. The vast majority of our installed base use Centos servers though, as far as I know.</p> <p>If you insist on using rhel kernels on Ubuntu one approach might be to download and install one set of our prebuilt Centos kernel and lustre rpms on Ubuntu using "alien". Could avoid all kinds of problems in building your own. Just a suggestion.</p> <p>Maybe just the kernel rpm would be safer. Might need to build lustre anyway to make sure all the userspace pieces were the right Ubuntu resident versions.</p> <p>"If you insist on using rhel kernels on Ubuntu one approach might be to download and install one set of our prebuilt Centos kernel and lustre rpms on Ubuntu using "alien". Could avoid all kinds of problems in building your own. Just a suggestion."</p> <p>We did take that approach about 5 or 6 years ago if I remember correctly. However our current method is easier than the pain we had with that. Given your notes Guy has built new kernel modules and userspace while I have been fighting with trying to upgrade a filesystem today. This new client seems to work. I suspect we would be happy for this ticket to be closed. ( I will let Guy confirm tomorrow ).</p> <p>Problem was pilot error. Must not strip -Werror from autoconf scripts.</p> Development Rank 1|hzw5tr: Rank (Obsolete) 11067 Severity