[root@iu-34 tests]# sh auster -r -f sarah-quota racer Started at Thu May 17 22:13:03 PDT 2012 Lustre is not mounted, trying to do setup ... Stopping clients: iu-34 /mnt/lustre (opts:) Stopping clients: iu-34 /mnt/lustre2 (opts:) Loading modules from /usr/lib64/lustre WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. [31663.932127] LNet: 9497:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. [31663.947918] LNet: 9498:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. subsystem_debug=0xffb7efff ../lnet/lnet/lnet options: 'networks="tcp0(eth0)" accept_port=7988 accept=all' WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. [31664.165788] Lustre: Lustre: Build Version: 2.0.59.0-g7058185-CHANGED-../lustre/scripts WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. [31664.328378] LNet: Added LNI 172.29.1.34@tcp [8/256/0/180] [31664.333898] LNet: Accept all, port 7988 gss/krb5 is not supported WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. Formatting mgs, mds, osts Format mds1: /dev/sda3 Format ost1: /dev/sda4 Checking servers environments Checking clients iu-34 environments Loading modules from /usr/lib64/lustre [31776.092183] LNet: 9926:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. debug=0xb3f0405 subsystem_debug=0xffb7efff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o user_xattr,acl /dev/sda3 /mnt/mds1 Started lustre-MDT0000 Starting ost1: /dev/sda4 /mnt/ost1 Started lustre-OST0000 Starting client: iu-34.lab.whamcloud.com: -o flock,user_xattr,acl iu-33@tcp:/lustre /mnt/lustre [31779.217106] LustreError: 152-6: Ignoring deprecated mount option 'acl'. [31779.218660] Lustre: MGC172.29.1.33@tcp: Reactivating import [31779.248971] LustreError: 11-0: an error occurred while communicating with 172.29.1.33@tcp. The mds_connect operation failed with -11 [31784.340476] Lustre: Mounted lustre-client [31784.354587] LNet: 10108:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. [31784.369185] LNet: 10108:0:(debug.c:324:libcfs_debug_str2mask()) Skipped 1 previous similar message Starting client iu-34: -o flock,user_xattr,acl iu-33@tcp:/lustre /mnt/lustre Started clients iu-34: iu-33@tcp:/lustre on /mnt/lustre type lustre (rw,flock,user_xattr,acl) Using TIMEOUT=20 [31786.145984] Lustre: DEBUG MARKER: Using TIMEOUT=20 enable quota as required [HOST:iu-34.lab.whamcloud.com] [old_quota_type:3] [new_quota_type:] iu-33: error: conf_param: No such device iu-33: error: conf_param: No such device Total disk size: 79701520 block-softlimit: 79702544 block-hardlimit: 83687671 inode-softlimit: 5000040 inode-hardlimit: 5250042 Logging to local directory: /tmp/test_logs/2012-05-17/221303 running: racer run_suite racer /usr/lib64/lustre/tests/racer.sh [31792.718600] Lustre: DEBUG MARKER: -----============= acceptance-small: racer ============----- Thu May 17 22:15:13 PDT 2012 [31792.987044] Lustre: DEBUG MARKER: excepting tests: [31793.132917] LustreError: 152-6: Ignoring deprecated mount option 'acl'. [31793.259476] Lustre: Mounted lustre-client [31793.556366] LNet: 11388:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. [31793.571018] LNet: 11388:0:(debug.c:324:libcfs_debug_str2mask()) Skipped 3 previous similar messages [31795.202149] Lustre: DEBUG MARKER: Using TIMEOUT=20 [31797.270394] Lustre: DEBUG MARKER: == racer test 1: racer on clients: iu-34 DURATION=900 == 22:15:17 (1337318117) [31810.307520] LustreError: 25110:0:(file.c:2279:ll_inode_revalidate_fini()) failure -2 inode 144115205272503660 [31810.378966] LustreError: 25041:0:(file.c:2279:ll_inode_revalidate_fini()) failure -2 inode 144115205272503660 [31811.720011] LustreError: 26074:0:(file.c:2279:ll_inode_revalidate_fini()) failure -2 inode 144115205272503660 [31811.729918] LustreError: 26074:0:(file.c:2279:ll_inode_revalidate_fini()) Skipped 3 previous similar messages [31812.737916] LustreError: 27663:0:(file.c:2279:ll_inode_revalidate_fini()) failure -2 inode 144115205272503660 [31812.747823] LustreError: 27663:0:(file.c:2279:ll_inode_revalidate_fini()) Skipped 9 previous similar messages [31815.601374] LustreError: 30125:0:(file.c:2279:ll_inode_revalidate_fini()) failure -2 inode 144115205272503660 [31815.611281] LustreError: 30125:0:(file.c:2279:ll_inode_revalidate_fini()) Skipped 13 previous similar messages [31844.085119] LustreError: 24091:0:(file.c:2279:ll_inode_revalidate_fini()) failure -2 inode 144115205255730715 [31844.095018] LustreError: 24091:0:(file.c:2279:ll_inode_revalidate_fini()) Skipped 17 previous similar messages [31862.619506] LustreError: 11358:0:(file.c:2279:ll_inode_revalidate_fini()) failure -2 inode 144115205272510100 [31862.629439] LustreError: 11358:0:(file.c:2279:ll_inode_revalidate_fini()) Skipped 1 previous similar message [31885.237515] LustreError: 2675:0:(file.c:2279:ll_inode_revalidate_fini()) failure -2 inode 144115205255736310 [31885.247336] LustreError: 2675:0:(file.c:2279:ll_inode_revalidate_fini()) Skipped 2 previous similar messages [31933.901744] LustreError: 18583:0:(file.c:2279:ll_inode_revalidate_fini()) failure -2 inode 144115205272519454 [31933.911652] LustreError: 18583:0:(file.c:2279:ll_inode_revalidate_fini()) Skipped 4 previous similar messages [32010.903565] LustreError: 1176:0:(file.c:2279:ll_inode_revalidate_fini()) failure -116 inode 144115205272530474 [32010.913578] LustreError: 1176:0:(file.c:2279:ll_inode_revalidate_fini()) Skipped 73 previous similar messages [32163.212697] LustreError: 24874:0:(file.c:2279:ll_inode_revalidate_fini()) failure -2 inode 144115205255775367 [32163.222613] LustreError: 24874:0:(file.c:2279:ll_inode_revalidate_fini()) Skipped 82 previous similar messages [32401.641110] LustreError: 11-0: an error occurred while communicating with 172.29.1.33@tcp. The ldlm_enqueue operation failed with -107 [32401.653195] Lustre: lustre-MDT0000-mdc-ffff88035c1d4000: Connection to lustre-MDT0000 (at 172.29.1.33@tcp) was lost; in progress operations using this service will wait for recovery to complete [32401.705451] LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [32401.724557] LustreError: 18179:0:(mdc_locks.c:725:mdc_enqueue()) ldlm_cli_enqueue: -4 [32401.724659] LustreError: 17938:0:(client.c:1040:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff88060c25a000 x1402279544836327/t0(0) o101->lustre-MDT0000-mdc-ffff88035c1d4000@172.29.1.33@tcp:12/10 lens 544/1104 e 0 to 0 dl 0 ref 2 fl Rpc:P/0/ffffffff rc 0/-1 [32401.725743] LustreError: 17932:0:(mdc_locks.c:725:mdc_enqueue()) ldlm_cli_enqueue: -108 [32401.725747] LustreError: 17932:0:(mdc_locks.c:725:mdc_enqueue()) Skipped 8 previous similar messages [32401.725796] LustreError: 17939:0:(client.c:1040:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff88033f72f800 x1402279544876204/t0(0) o101->lustre-MDT0000-mdc-ffff88035c1d4000@172.29.1.33@tcp:12/10 lens 544/1104 e 0 to 0 dl 0 ref 2 fl Rpc:P/0/ffffffff rc 0/-1 [32401.725803] LustreError: 17939:0:(client.c:1040:ptlrpc_import_delay_req()) Skipped 8 previous similar messages [32401.733412] LustreError: 18561:0:(ldlm_resource.c:757:ldlm_resource_complain()) Namespace lustre-MDT0000-mdc-ffff88035c1d4000 resource refcount nonzero (1) after lock cleanup; forcing cleanup. [32401.733417] LustreError: 18561:0:(ldlm_resource.c:763:ldlm_resource_complain()) Resource: ffff880620472c80 (8589935616/62145/0/0) (rc: 1) [32401.733744] Lustre: lustre-MDT0000-mdc-ffff88035c1d4000: Connection restored to lustre-MDT0000 (at 172.29.1.33@tcp) [32401.845455] LustreError: 18179:0:(mdc_locks.c:725:mdc_enqueue()) Skipped 13 previous similar messages [32483.048923] BUG: soft lockup - CPU#0 stuck for 67s! [ls:17939] [32483.054744] Modules linked in: lustre mgc lov osc mdc lmv fid fld lquota ptlrpc obdclass lvfs ksocklnd lnet libcfs nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 g450_pll matroxfb_misc cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core igb dca [last unloaded: libcfs] [32483.106005] CPU 0 [32483.107843] Modules linked in: lustre mgc lov osc mdc lmv fid fld lquota ptlrpc obdclass lvfs ksocklnd lnet libcfs nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 g450_pll matroxfb_misc cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core igb dca [last unloaded: libcfs] [32483.159291] [32483.160785] Pid: 17939, comm: ls Tainted: G W 2.6.38.6-rc1 #1 IBM IBM System X iDataPlex dx360 M3 Server -[6391AC1]-/69Y4782 [32483.173278] RIP: 0010:[] [] _raw_spin_lock+0x1e/0x25 [32483.181454] RSP: 0018:ffff880612919bb8 EFLAGS: 00000297 [32483.186757] RAX: 000000000000eccb RBX: 0000000000000000 RCX: ffff8803643f5580 [32483.193873] RDX: 000000000000ecca RSI: 0000000000000000 RDI: ffff88035f040c5c [32483.200989] RBP: ffff880612919bb8 R08: ffff880612919b68 R09: 0000000000000054 [32483.208107] R10: 5a5a5a5a5a5a5a5a R11: 0000000000000001 R12: ffffffff8100b5ce [32483.215223] R13: ffff88035c1d4000 R14: ffff880612919b68 R15: ffffffff81a022c0 [32483.222339] FS: 00007f99bdd137a0(0000) GS:ffff88007cc00000(0000) knlGS:0000000000000000 [32483.230411] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [32483.236142] CR2: 0000003d0d8ab0b0 CR3: 0000000617029000 CR4: 00000000000006f0 [32483.243259] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32483.250375] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [32483.257492] Process ls (pid: 17939, threadinfo ffff880612918000, task ffff8806129d4560) [32483.265473] Stack: [32483.267485] ffff880612919c48 ffffffff8113eb55 [32483.270554] BUG: soft lockup - CPU#3 stuck for 67s! [ls:17931] [32483.270555] Modules linked in: lustre mgc lov osc mdc lmv fid fld lquota ptlrpc obdclass lvfs ksocklnd lnet libcfs nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 g450_pll matroxfb_misc cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core igb dca [last unloaded: libcfs] [32483.270586] CPU 3 [32483.270586] Modules linked in: lustre mgc lov osc mdc lmv fid fld lquota ptlrpc obdclass lvfs ksocklnd lnet libcfs nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 g450_pll matroxfb_misc cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core igb dca [last unloaded: libcfs] [32483.270606] [32483.270608] Pid: 17931, comm: ls Tainted: G W 2.6.38.6-rc1 #1 IBM IBM System X iDataPlex dx360 M3 Server -[6391AC1]-/69Y4782 [32483.270611] RIP: 0010:[] [] dentry_kill+0x4a/0x11e [32483.270617] RSP: 0018:ffff8806189b3ac8 EFLAGS: 00000286 [32483.270618] RAX: 0000000000000001 RBX: 5a5a5a5a5a5a5a5a RCX: ffff8803643f5580 [32483.270620] RDX: 0000000025432501 RSI: 0000000000000000 RDI: ffff88035f040c5c [32483.270622] RBP: ffff8806189b3ae8 R08: ffff8806189b3b68 R09: 0000000000000054 [32483.270623] R10: 5a5a5a5a5a5a5a5a R11: 0000000000000001 R12: ffffffff8100b5ce [32483.270625] R13: ffff8806189b3ab8 R14: ffffffff8100b5ce R15: ffff8806189b3a78 [32483.270627] FS: 00007f94a145b7a0(0000) GS:ffff88007cc60000(0000) knlGS:0000000000000000 [32483.270629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32483.270631] CR2: 0000000001753ee0 CR3: 000000032aa43000 CR4: 00000000000006e0 [32483.270632] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32483.270634] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [32483.270636] Process ls (pid: 17931, threadinfo ffff8806189b2000, task ffff88060b43ae40) [32483.270637] Stack: [32483.270638] ffff8803643f5500 ffff8806189b3b68 ffff8806189b3b10 ffff8803643f5580 [32483.270641] ffff8806189b3b38 ffffffff8113e7b5 ffff88035c1d40e0 0000000000000001 [32483.270644] ffffffff81a022c0 ffff8803643f5580 ffff88035c1d4000 ffff8803643f5500 [32483.270647] Call Trace: [32483.270651] [] shrink_dentry_list+0x67/0xbb [32483.270653] [] __shrink_dcache_sb+0x157/0x16f [32483.270657] [] ? _raw_spin_lock+0x1e/0x25 [32483.270659] [] shrink_dcache_parent+0x51/0x217 [32483.270662] [] d_invalidate+0x3c/0x80 [32483.270665] [] do_revalidate+0x34/0x4a [32483.270667] [] do_lookup+0x1ad/0x28c [32483.270671] [] ? security_inode_exec_permission+0x2a/0x2c [32483.270674] [] link_path_walk+0x149/0x39a [32483.270677] [] do_path_lookup+0x50/0x10e [32483.270679] [] do_filp_open+0x12a/0x65a [32483.270683] [] ? handle_mm_fault+0x229/0x240 [32483.270686] [] ? slab_pre_alloc_hook+0x2d/0x31 [32483.270690] [] ? might_fault+0x21/0x23 [32483.270694] [] ? alloc_fd+0x7b/0x124 [32483.270697] [] do_sys_open+0x60/0xf2 [32483.270699] [] sys_open+0x20/0x22 [32483.270704] [] system_call_fastpath+0x16/0x1b [32483.270706] Code: 49 8d 7d 20 e8 98 d1 34 00 85 c0 75 0b 66 ff 43 5c f3 90 e9 db 00 00 00 4c 8b 63 18 4c 39 e3 74 1f 4d 85 e4 74 1d 49 8d 7c 24 5c 71 d1 34 00 85 c0 75 0f 4d 85 ed 74 d4 66 41 ff 45 20 eb cd [32483.270720] Call Trace: [32483.270722] [] shrink_dentry_list+0x67/0xbb [32483.270724] [] __shrink_dcache_sb+0x157/0x16f [32483.270726] [] ? _raw_spin_lock+0x1e/0x25 [32483.270728] [] shrink_dcache_parent+0x51/0x217 [32483.270731] [] d_invalidate+0x3c/0x80 [32483.270733] [] do_revalidate+0x34/0x4a [32483.270735] [] do_lookup+0x1ad/0x28c [32483.270737] [] ? security_inode_exec_permission+0x2a/0x2c [32483.270740] [] link_path_walk+0x149/0x39a [32483.270742] [] do_path_lookup+0x50/0x10e [32483.270744] [] do_filp_open+0x12a/0x65a [32483.270746] [] ? handle_mm_fault+0x229/0x240 [32483.270749] [] ? slab_pre_alloc_hook+0x2d/0x31 [32483.270751] [] ? might_fault+0x21/0x23 [32483.270753] [] ? alloc_fd+0x7b/0x124 [32483.270755] [] do_sys_open+0x60/0xf2 [32483.270757] [] sys_open+0x20/0x22 [32483.270760] [] system_call_fastpath+0x16/0x1b [32483.569050] BUG: soft lockup - CPU#7 stuck for 67s! [ls:17938] [32483.569053] Modules linked in: lustre mgc lov osc mdc lmv fid fld lquota ptlrpc obdclass lvfs ksocklnd lnet libcfs nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 g450_pll matroxfb_misc cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core igb dca [last unloaded: libcfs] [32483.569082] CPU 7 [32483.569083] Modules linked in: lustre mgc lov osc mdc lmv fid fld lquota ptlrpc obdclass lvfs ksocklnd lnet libcfs nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 g450_pll matroxfb_misc cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core igb dca [last unloaded: libcfs] [32483.569103] [32483.569105] Pid: 17938, comm: ls Tainted: G W 2.6.38.6-rc1 #1 IBM IBM System X iDataPlex dx360 M3 Server -[6391AC1]-/69Y4782 [32483.569109] RIP: 0010:[] [] _raw_spin_lock+0x1e/0x25 [32483.569115] RSP: 0018:ffff880639aa3bb8 EFLAGS: 00000293 [32483.569117] RAX: 000000000000ecce RBX: 0000000000000000 RCX: ffff8803643f5580 [32483.569118] RDX: 000000000000eccb RSI: 0000000000000000 RDI: ffff88035f040c5c [32483.569120] RBP: ffff880639aa3bb8 R08: ffff880639aa3b68 R09: 0000000000000054 [32483.569121] R10: 5a5a5a5a5a5a5a5a R11: 0000000000000001 R12: ffffffff8100b5ce [32483.569123] R13: ffff88035c1d4000 R14: ffff880639aa3b68 R15: ffff8803643f555c [32483.569125] FS: 00007f70128087a0(0000) GS:ffff88067fc20000(0000) knlGS:0000000000000000 [32483.569127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32483.569129] CR2: 0000000000aa390c CR3: 00000003288c3000 CR4: 00000000000006e0 [32483.569130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32483.569132] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [32483.569134] Process ls (pid: 17938, threadinfo ffff880639aa2000, task ffff88060c235c80) [32483.569136] Stack: [32483.569137] ffff880639aa3c48 ffffffff8113eb55 0000000000000010 ffff88035c1d4000 [32483.569140] ffff8803643f5580 ffff8803643f5500 ffff880639aa2010 ffff88035c1d4000 [32483.569143] ffff88035f040c5c ffff880631b8d710 0000000000000000 0000000000000069 [32483.569146] Call Trace: [32483.569152] [] shrink_dcache_parent+0x71/0x217 [32483.569155] [] d_invalidate+0x3c/0x80 [32483.569160] [] do_revalidate+0x34/0x4a [32483.569162] [] do_lookup+0x1ad/0x28c [32483.569167] [] ? security_inode_exec_permission+0x2a/0x2c [32483.569170] [] link_path_walk+0x149/0x39a [32483.569173] [] do_path_lookup+0x50/0x10e [32483.569175] [] do_filp_open+0x12a/0x65a [32483.569180] [] ? handle_mm_fault+0x229/0x240 [32483.569184] [] ? slab_pre_alloc_hook+0x2d/0x31 [32483.569188] [] ? might_fault+0x21/0x23 [32483.569192] [] ? alloc_fd+0x7b/0x124 [32483.569196] [] do_sys_open+0x60/0xf2 [32483.569198] [] sys_open+0x20/0x22 [32483.569204] [] system_call_fastpath+0x16/0x1b [32483.569206] Code: 0f b6 c2 85 c0 c9 0f 95 c0 0f b6 c0 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c9 c3 55 48 89 e5 0f 1f 44 00 00 9c 58 0f 1f 44 [32483.569220] Call Trace: [32483.569222] [] shrink_dcache_parent+0x71/0x217 [32483.569224] [] d_invalidate+0x3c/0x80 [32483.569227] [] do_revalidate+0x34/0x4a [32483.569229] [] do_lookup+0x1ad/0x28c [32483.569231] [] ? security_inode_exec_permission+0x2a/0x2c [32483.569234] [] link_path_walk+0x149/0x39a [32483.569236] [] do_path_lookup+0x50/0x10e [32483.569238] [] do_filp_open+0x12a/0x65a [32483.569241] [] ? handle_mm_fault+0x229/0x240 [32483.569243] [] ? slab_pre_alloc_hook+0x2d/0x31 [32483.569245] [] ? might_fault+0x21/0x23 [32483.569247] [] ? alloc_fd+0x7b/0x124 [32483.569249] [] do_sys_open+0x60/0xf2 [32483.569251] [] sys_open+0x20/0x22 [32483.569254] [] system_call_fastpath+0x16/0x1b [32483.790676] BUG: soft lockup - CPU#10 stuck for 67s! [ls:17932] [32483.790678] Modules linked in: lustre mgc lov osc mdc lmv fid fld lquota ptlrpc obdclass lvfs ksocklnd lnet libcfs nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 g450_pll matroxfb_misc cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core igb dca [last unloaded: libcfs] [32483.790702] CPU 10 [32483.790703] Modules linked in: lustre mgc lov osc mdc lmv fid fld lquota ptlrpc obdclass lvfs ksocklnd lnet libcfs nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 g450_pll matroxfb_misc cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core igb dca [last unloaded: libcfs] [32483.790723] [32483.790725] Pid: 17932, comm: ls Tainted: G W 2.6.38.6-rc1 #1 IBM IBM System X iDataPlex dx360 M3 Server -[6391AC1]-/69Y4782 [32483.790728] RIP: 0010:[] [] _raw_spin_lock+0x1e/0x25 [32483.790733] RSP: 0018:ffff88061322bbb8 EFLAGS: 00000293 [32483.790734] RAX: 000000000000eccd RBX: 0000000000000000 RCX: ffff8803643f5580 [32483.790736] RDX: 000000000000eccb RSI: 0000000000000000 RDI: ffff88035f040c5c [32483.790738] RBP: ffff88061322bbb8 R08: ffff88061322bb68 R09: 0000000000000054 [32483.790739] R10: 5a5a5a5a5a5a5a5a R11: 0000000000000001 R12: ffffffff8100b5ce [32483.790741] R13: ffff88035c1d4000 R14: ffff88061322bb68 R15: ffffffff81a022c0 [32483.790743] FS: 00007fefa5c937a0(0000) GS:ffff88067fc80000(0000) knlGS:0000000000000000 [32483.790745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [32483.790747] CR2: 0000000001fa1430 CR3: 000000061584b000 CR4: 00000000000006e0 [32483.790748] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32483.790750] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [32483.790752] Process ls (pid: 17932, threadinfo ffff88061322a000, task ffff880615a30000) [32483.790753] Stack: [32483.790754] ffff88061322bc48 ffffffff8113eb55 0000000000000010 ffff88035c1d4000 [32483.790758] ffff8803643f5580 ffff8803643f5500 ffff88061322a010 ffff88035c1d4000 [32483.790760] ffff88035f040c5c ffff880631b8d710 0000000000000000 0000000000000069 [32483.790763] Call Trace: [32483.790767] [] shrink_dcache_parent+0x71/0x217 [32483.790770] [] d_invalidate+0x3c/0x80 [32483.790772] [] do_revalidate+0x34/0x4a [32483.790774] [] do_lookup+0x1ad/0x28c [32483.790777] [] ? security_inode_exec_permission+0x2a/0x2c [32483.790780] [] link_path_walk+0x149/0x39a [32483.790783] [] do_path_lookup+0x50/0x10e [32483.790785] [] do_filp_open+0x12a/0x65a [32483.790788] [] ? handle_mm_fault+0x229/0x240 [32483.790791] [] ? slab_pre_alloc_hook+0x2d/0x31 [32483.790794] [] ? might_fault+0x21/0x23 [32483.790796] [] ? alloc_fd+0x7b/0x124 [32483.790799] [] do_sys_open+0x60/0xf2 [32483.790801] [] sys_open+0x20/0x22 [32483.790805] [] system_call_fastpath+0x16/0x1b [32483.790806] Code: 0f b6 c2 85 c0 c9 0f 95 c0 0f b6 c0 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c9 c3 55 48 89 e5 0f 1f 44 00 00 9c 58 0f 1f 44 [32483.790820] Call Trace: [32483.790822] [] shrink_dcache_parent+0x71/0x217 [32483.790825] [] d_invalidate+0x3c/0x80 [32483.790827] [] do_revalidate+0x34/0x4a [32483.790829] [] do_lookup+0x1ad/0x28c [32483.790831] [] ? security_inode_exec_permission+0x2a/0x2c [32483.790834] [] link_path_walk+0x149/0x39a [32483.790837] [] do_path_lookup+0x50/0x10e [32483.790839] [] do_filp_open+0x12a/0x65a [32483.790841] [] ? handle_mm_fault+0x229/0x240 [32483.790843] [] ? slab_pre_alloc_hook+0x2d/0x31 [32483.790846] [] ? might_fault+0x21/0x23 [32483.790848] [] ? alloc_fd+0x7b/0x124 [32483.790850] [] do_sys_open+0x60/0xf2 [32483.790852] [] sys_open+0x20/0x22 [32483.790854] [] system_call_fastpath+0x16/0x1b [32483.864552] BUG: soft lockup - CPU#11 stuck for 66s! [ls:17966] [32483.864553] Modules linked in: lustre mgc lov osc mdc lmv fid fld lquota ptlrpc obdclass lvfs ksocklnd lnet libcfs nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 g450_pll matroxfb_misc cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core igb dca [last unloaded: libcfs] [32483.864576] CPU 11 [32483.864577] Modules linked in: lustre mgc lov osc mdc lmv fid fld lquota ptlrpc obdclass lvfs ksocklnd lnet libcfs nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 g450_pll matroxfb_misc cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core igb dca [last unloaded: libcfs] [32483.864597] [32483.864598] Pid: 17966, comm: ls Tainted: G W 2.6.38.6-rc1 #1 IBM IBM System X iDataPlex dx360 M3 Server -[6391AC1]-/69Y4782 [32483.864602] RIP: 0010:[] [] _raw_spin_lock+0x21/0x25 [32483.864605] RSP: 0018:ffff88061a07fbb8 EFLAGS: 00000297 [32483.864607] RAX: 000000000000eccc RBX: 0000000000000000 RCX: ffff8803643f5580 [32483.864608] RDX: 000000000000eccb RSI: 0000000000000000 RDI: ffff88035f040c5c [32483.864610] RBP: ffff88061a07fbb8 R08: ffff88061a07fb68 R09: 0000000000000054 [32483.864612] R10: 5a5a5a5a5a5a5a5a R11: 0000000000000001 R12: ffffffff8100b5ce [32483.864613] R13: ffff88035c1d4000 R14: ffff88061a07fb68 R15: ffffffff8148b830 [32483.864615] FS: 00007fea18df27a0(0000) GS:ffff88067fca0000(0000) knlGS:0000000000000000 [32483.864617] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [32483.864619] CR2: 00000000015de960 CR3: 00000006170b4000 CR4: 00000000000006e0 [32483.864620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [32483.864622] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [32483.864624] Process ls (pid: 17966, threadinfo ffff88061a07e000, task ffff88060c26dc80) [32483.864625] Stack: [32483.864626] ffff88061a07fc48 ffffffff8113eb55 0000000000000010 ffff88035c1d4000 [32483.864630] ffff8803643f5580 ffff8803643f5500 ffff88061a07e010 ffff88035c1d4000 [32483.864632] ffff88035f040c5c ffff880631b8d710 0000000000000000 0000000000000069 [32483.864635] Call Trace: [32483.864638] [] shrink_dcache_parent+0x71/0x217 [32483.864641] [] d_invalidate+0x3c/0x80 [32483.864644] [] do_revalidate+0x34/0x4a [32483.864646] [] do_lookup+0x1ad/0x28c [32483.864649] [] ? security_inode_exec_permission+0x2a/0x2c [32483.864652] [] link_path_walk+0x149/0x39a [32483.864654] [] do_path_lookup+0x50/0x10e [32483.864657] [] do_filp_open+0x12a/0x65a [32483.864659] [] ? handle_mm_fault+0x229/0x240 [32483.864662] [] ? slab_pre_alloc_hook+0x2d/0x31 [32483.864665] [] ? might_fault+0x21/0x23 [32483.864667] [] ? alloc_fd+0x7b/0x124 [32483.864670] [] do_sys_open+0x60/0xf2 [32483.864672] [] sys_open+0x20/0x22 [32483.864675] [] system_call_fastpath+0x16/0x1b [32483.864676] Code: 85 c0 c9 0f 95 c0 0f b6 c0 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 17 f5 c9 c3 55 48 89 e5 0f 1f 44 00 00 9c 58 0f 1f 44 00 00 48 [32483.864690] Call Trace: [32483.864692] [] shrink_dcache_parent+0x71/0x217 [32483.864695] [] d_invalidate+0x3c/0x80 [32483.864697] [] do_revalidate+0x34/0x4a [32483.864699] [] do_lookup+0x1ad/0x28c [32483.864702] [] ? security_inode_exec_permission+0x2a/0x2c [32483.864704] [] link_path_walk+0x149/0x39a [32483.864707] [] do_path_lookup+0x50/0x10e [32483.864709] [] do_filp_open+0x12a/0x65a [32483.864711] [] ? handle_mm_fault+0x229/0x240 [32483.864714] [] ? slab_pre_alloc_hook+0x2d/0x31 [32483.864716] [] ? might_fault+0x21/0x23 [32483.864718] [] ? alloc_fd+0x7b/0x124 [32483.864720] [] do_sys_open+0x60/0xf2 [32483.864722] [] sys_open+0x20/0x22 [32483.864725] [] system_call_fastpath+0x16/0x1b [32485.029229] 0000000000000010 ffff88035c1d4000 [32485.033701] ffff8803643f5580 ffff8803643f5500 ffff880612918010 ffff88035c1d4000 [32485.041154] ffff88035f040c5c ffff880631b8d710 0000000000000000 0000000000000069 [32485.048609] Call Trace: [32485.051056] [] shrink_dcache_parent+0x71/0x217 [32485.057135] [] d_invalidate+0x3c/0x80 [32485.062440] [] do_revalidate+0x34/0x4a [32485.067826] [] do_lookup+0x1ad/0x28c [32485.073042] [] ? security_inode_exec_permission+0x2a/0x2c [32485.080076] [] link_path_walk+0x149/0x39a [32485.085726] [] do_path_lookup+0x50/0x10e [32485.091285] [] do_filp_open+0x12a/0x65a [32485.096761] [] ? handle_mm_fault+0x229/0x240 [32485.102668] [] ? slab_pre_alloc_hook+0x2d/0x31 [32485.108747] [] ? might_fault+0x21/0x23 [32485.114135] [] ? alloc_fd+0x7b/0x124 [32485.119348] [] do_sys_open+0x60/0xf2 [32485.124563] [] sys_open+0x20/0x22 [32485.129521] [] system_call_fastpath+0x16/0x1b [32485.135513] Code: 0f b6 c2 85 c0 c9 0f 95 c0 0f b6 c0 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c9 c3 55 48 89 e5 0f 1f 44 00 00 9c 58 0f 1f 44 [32485.155615] Call Trace: [32485.158061] [] shrink_dcache_parent+0x71/0x217 [32485.164142] [] d_invalidate+0x3c/0x80 [32485.169447] [] do_revalidate+0x34/0x4a [32485.174835] [] do_lookup+0x1ad/0x28c [32485.180051] [] ? security_inode_exec_permission+0x2a/0x2c [32485.187086] [] link_path_walk+0x149/0x39a [32485.192734] [] do_path_lookup+0x50/0x10e [32485.198294] [] do_filp_open+0x12a/0x65a [32485.203771] [] ? handle_mm_fault+0x229/0x240 [32485.209677] [] ? slab_pre_alloc_hook+0x2d/0x31 [32485.215757] [] ? might_fault+0x21/0x23 [32485.221145] [] ? alloc_fd+0x7b/0x124 [32485.226360] [] do_sys_open+0x60/0xf2 [32485.231574] [] sys_open+0x20/0x22 [32485.236532] [] system_call_fastpath+0x16/0x1b