[LU-6831] The ticket for tracking all DNE2 bugs Created: 09/Jul/15  Updated: 25/Feb/20

Status: Reopened
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.8.0, Lustre 2.9.0
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Di Wang Assignee: Di Wang
Resolution: Unresolved Votes: 0
Labels: dne2, llnl

Attachments: Text File kern-09-10-2015.log    
Issue Links:
Related
is related to LU-5222 Some operation should never happened ... Open
is related to LU-6297 Move rename is_subdir check from MDD ... Open
is related to LU-6819 LBUG ASSERTION( tdtd->tdtd_last_updat... Resolved
is related to LU-6834 idx_array is overwritten in lod_prep_... Resolved
is related to LU-6837 MDS panic during 24 hours failover test. Resolved
is related to LU-6840 update memory reply data in DNE updat... Resolved
is related to LU-6880 recovery timeout during 24 hours fail... Resolved
is related to LU-7039 llog_osd.c:778:llog_osd_next_block())... Resolved
is related to LU-7285 ASSERTION in check_for_next_transno Resolved
is related to LU-7318 OUT: dynamic reply buffer Resolved
is related to LU-5951 sanity test_39k: mtime is lost on close Resolved
is related to LU-6475 race between open and migration Resolved
is related to LU-6602 ASSERTION( rec->lrh_len <= 8192 ) failed Resolved
is related to LU-6741 OSP should use inline data for small ... Resolved
is related to LU-6896 update llog object is missing during ... Resolved
is related to LU-6916 unable to handle kernel NULL pointer ... Resolved
is related to LU-6977 lod_striped_it_next()) ASSERTION( nex... Resolved
is related to LU-6981 obd_last_committed is not updated in ... Resolved
is related to LU-6998 Do not always lock stripe0 object in ... Resolved
is related to LU-6999 Disable xattr cache for remote mdt ob... Resolved
is related to LU-7091 refresh link after update linkea in m... Resolved
is related to LU-5344 ldlm/ifind deadlock for striped direc... Resolved
is related to LU-6586 "lctl conf_param testfs-MDT0001.mdc.a... Resolved
is related to LU-6838 update llog become too big before it ... Resolved
is related to LU-6846 dt_record_write()) ASSERTION( dt->do_... Resolved
is related to LU-6852 MDS is evicted during 24-24 hours fai... Resolved
is related to LU-6859 osd_declare_object_destroy()) ASSERTI... Resolved
is related to LU-6904 linkea prepare in mdt_reint_rename ca... Resolved
is related to LU-6924 remote regular file are missing after... Resolved
is related to LU-6928 Version mismatch during DNE replay Resolved
is related to LU-6997 migration should enqueue nonblock loc... Resolved
is related to LU-7005 conf-sanity test_50i: lustre-MDT0001-... Resolved
is related to LU-7008 osd_index_declare_ea_delete()) ASSERT... Resolved
is related to LU-6637 sanity-scrub test_9: Got speed 158, e... Resolved
is related to LU-6638 sanity-hsm test_37:request on 0x20000... Resolved
is related to LU-6639 sanity-lfsck test_2e: cpu softlock Resolved
is related to LU-6888 osp_update_request_create() should ch... Resolved
is related to LU-6906 During 24 hours DNE test, one of MDS ... Resolved
is related to LU-6968 Update the whole header in llog_cance... Resolved
is related to LU-7009 Testing DNE on ZFS Resolved
is related to LU-7050 llog_skip_over skip the record by too... Resolved
is related to LU-7074 skip invalid linkEA entry in migration Resolved
is related to LU-7213 replay-single test_70d: mkdir: create... Resolved
is related to LU-6362 DNE2: add dt object and thandle check... Closed
is related to LU-6640 sanity-hsm test_21:Copytool failed to... Closed
is related to LU-3538 commit on share for cross-MDT operation. Resolved
is related to LU-6773 DNE2 Failover and recovery soak testing Closed
is related to LU-7827 DNE3: automatically select MDT for lf... Resolved
is related to LU-6826 sanity test_71: No space left on device Resolved
is related to LU-6984 Failure to delete over a million file... Resolved
is related to LU-7107 lfs_migrate() may fail in striped dir... Resolved
is related to LU-7013 replay-single test_110c: FAIL: create... Resolved
is related to LU-6284 FLD read is not swabbed correctly Resolved
is related to LU-7293 DNE2 perfomance analysis Resolved
is related to LU-7357 Add layout lock for striped directories. Resolved
Severity: 3
Rank (Obsolete): 9223372036854775807

 Description   

This ticket is for tracking all of DNE2 bugs.

 Comments   
Comment by James A Simmons [ 13/Jul/15 ]

In my latest testing I'm running into this bug:

mdtest-1.8.3 was launched with 1 total task(s) on 1 nodes
Command line used: /lustre/sultan/stf008/scratch/jsimmons/mdtest -I 10000 -i 5 -d /lustre/sultan/stf008/scratch/jsimm
ons/dne2_2_mds_md_test/shared_10k_1
Path: /lustre/sultan/stf008/scratch/jsimmons/dne2_2_mds_md_test
FS: 21.8 TiB Used FS: 0.2% Inodes: 82.7 Mi Used Inodes: 0.0%
07/13/2015 16:12:12: Process 0(): FAILED in main, Unable to create test directory path: Object is remote
Rank 0 [Mon Jul 13 16:12:12 2015] [c0-0c0s1n2] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
_pmiu_daemon(SIGCHLD): [NID 00028] [c0-0c0s1n2] [Mon Jul 13 16:12:12 2015] PE RANK 0 exit signal Aborted
Application 2640 exit codes: 134
Application 2640 resources: utime ~0s, stime ~0s, Rss ~3856, inblocks ~5254, outblocks ~13678

It is very easy to reproduce. What I did was create a stripe directory of count 2 at a index of 1. The reason is I'm avoiding using MDS0 which has the smallest MDT.

Comment by Di Wang [ 13/Jul/15 ]

James: I can run this on my node

[root@testnode mdtest]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_testnode-lv_root
                      27228028  10704408  15133848  42% /
tmpfs                  4014860         0   4014860   0% /dev/shm
/dev/sda1               487652     48320    413732  11% /boot
192.168.167.1:/Users/wangdi/work
                     243358976 188184448  54918528  78% /work
/dev/loop2              133560      1904    121932   2% /mnt/mds3
/dev/loop3              133560      1908    121928   2% /mnt/mds4
/dev/loop4              358552     13900    324420   5% /mnt/ost1
/dev/loop5              358552     13904    324416   5% /mnt/ost2
/dev/loop0              133560      2192    121644   2% /mnt/mds1
/dev/loop1              133560      2040    121796   2% /mnt/mds2
testnode@tcp:/lustre    717104     27804    648836   5% /mnt/lustre
[root@testnode mdtest]# cd /work/lustre_release_work/lustre-release_new/lustre/tests/
[root@testnode tests]# pwd
/work/lustre_release_work/lustre-release_new/lustre/tests
[root@testnode tests]# dne2_2_mds_md_test^C
[root@testnode tests]# ../utils/lfs mkdir -i1 -c2 /mnt/lustre/test
[root@testnode tests]# /work/mdtest/
COPYRIGHT    Makefile     mdtest       mdtest.1     README       RELEASE_LOG  scripts/     
[root@testnode tests]# /work/mdtest/mdtest -I 10000 -i 5 -d /mnt/lustre/test/shared_10k_1
-- started at 07/12/2015 00:10:25 --

mdtest-1.9.3 was launched with 1 total task(s) on 1 node(s)
Command line used: /work/mdtest/mdtest -I 10000 -i 5 -d /mnt/lustre/test/shared_10k_1
Path: /mnt/lustre/test
FS: 0.7 GiB   Used FS: 3.9%   Inodes: 0.2 Mi   Used Inodes: 0.5%

1 tasks, 10000 files/directories


SUMMARY: (of 5 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:       6005.343       5420.750       5707.103        218.750
   Directory stat    :       5607.789       5152.637       5289.370        166.391
   Directory removal :       5475.998       5276.844       5368.197         78.798
   File creation     :       3563.760       3035.905       3204.669        194.226
   File stat         :       2883.332       2761.046       2820.534         41.375
   File read         :       3320.368       2823.954       3061.123        194.269
   File removal      :       5018.647       4533.453       4712.663        186.385
   Tree creation     :       4696.869       2688.656       3332.096        703.002
   Tree removal      :       2096.104       1976.581       2042.187         48.555

-- finished at 07/12/2015 00:11:53 --

Could you please check if these values has been set correctly? Thanks

[root@testnode lustre-release_new]# ./lustre/utils/lctl get_param mdt.*.enable_remote_dir
mdt.lustre-MDT0000.enable_remote_dir=1
mdt.lustre-MDT0001.enable_remote_dir=1
mdt.lustre-MDT0002.enable_remote_dir=1
mdt.lustre-MDT0003.enable_remote_dir=1
Comment by James A Simmons [ 13/Jul/15 ]

Doh. I assume remote_dir was enabled by default. I just set it. Seems to work now. The strange thing is even with remote_dir=0 everywhere when I set the directory MDS stripe to > 1 as long as the index was zero it appeared to work.

Comment by James A Simmons [ 14/Jul/15 ]

Yes as root it works but not as a regular user. I created a directory with the count = 2 and index = 1 then I did the -D flags so any directories created under my DNE2 directory would inherit the properties. As my self (non-root) I tried a mkdir and it failed.

Comment by Di Wang [ 14/Jul/15 ]

James: you need set these value to -1 to make sure all users can create remote and striped dir

[root@mds01 ~]# lctl get_param mdt.*.enable_remote_dir_gid
mdt.lustre-MDT0000.enable_remote_dir_gid=0
mdt.lustre-MDT0004.enable_remote_dir_gid=0
Comment by James A Simmons [ 15/Jul/15 ]

Nope. Creation of the directories appear to only work part of the time. Some times it does work and some times it does not work. Running mdtest never can run for me.

Comment by Di Wang [ 15/Jul/15 ]

If you run the test as non-root user, then you probably need this patch

http://review.whamcloud.com/#/c/13990/

And also please set enable_remote_dir_gid=-1 on all of MDTs.

Comment by James A Simmons [ 15/Jul/15 ]

Patch 13990 did the trick. Now I can create DNE2 striped directories as myself. Thanks.

Comment by James A Simmons [ 20/Jul/15 ]

Updated to your latest patches and I lost the ability to create remote directories. Now I get the following errors:

[ 1016.185382] Lustre: 19975:0:(lmv_obd.c:297:lmv_init_ea_size()) sultan-clilmv-ffff88080ac8ec00: NULL export for 1
[ 1016.195740] Lustre: 19975:0:(lmv_obd.c:297:lmv_init_ea_size()) Skipped 26 previous similar messages
[ 1016.498646] Lustre: Mounted sultan-client
[ 1022.254520] LustreError: 19996:0:(lmv_obd.c:1332:lmv_fid_alloc()) Can't alloc new fid, rc -19

Comment by Di Wang [ 20/Jul/15 ]

which patches? could you please list your patches here? thanks

Comment by James A Simmons [ 21/Jul/15 ]

http://review.whamcloud.com/#/c/13990
http://review.whamcloud.com/#/c/15572
http://review.whamcloud.com/#/c/15576

Comment by Di Wang [ 21/Jul/15 ]

Strange, does your build include this patch http://review.whamcloud.com/#/c/15269/ ? If it does, please remove this one. and retry? thanks

Comment by James A Simmons [ 21/Jul/15 ]

Nope. I found the source of the problems. It was the patch from LU-6341. Once I removed it the problems went away.

Comment by James A Simmons [ 22/Jul/15 ]

Doing more testing I found that the patch from LU-6341 was not the source of the bug but it exposed the issue reported in LU-6713. Once I applied patch 15269 the problem went away.

Comment by James A Simmons [ 23/Jul/15 ]

Now I'm seeing clients get evicted during heavy meta data operations. Di Wang have you seen this behavior and does a patch exist to address this?

Comment by Di Wang [ 23/Jul/15 ]

James: What test did you run? Do you have the trace? I am not sure if there are such fixes. Thanks.

Comment by James A Simmons [ 24/Jul/15 ]

I see what is triggering the client evictions. I'm getting these errors on the clients:

LustreError: 10306:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 1 [0x2800013ba:0x84ad:0x0]:[0x2400013c8:0x84ad:0x0]
LustreError: 10306:0:(file.c:3104:ll_inode_revalidate_fini()) sultan: revalidate FID [0x2400013ce:0x3:0x0] error: rc = -5
LustreError: 10316:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 0 [0x2400013b3:0x14262:0x0]:[0x2400013b3:0x14262:0x0]
LustreError: 10863:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 0 [0x2400013b3:0x14262:0x0]:[0x2400013b3:0x14262:0x0]
LustreError: 10894:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 1 [0x2800013ab:0x14262:0x0]:[0x2400013b3:0x14262:0x0]
LustreError: 10894:0:(file.c:3104:ll_inode_revalidate_fini()) sultan: revalidate FID [0x2400013a4:0x5:0x0] error: rc = -5

Comment by Gerrit Updater [ 24/Jul/15 ]

wangdi (di.wang@intel.com) uploaded a new patch: http://review.whamcloud.com/15720
Subject: LU-6831 lmv: revalidate the dentry for striped dir
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 26133ac386693647ef3ee4480a6ce0cbb2773adb

Comment by Di Wang [ 24/Jul/15 ]

James: Please try this patch to see if it works? thanks. Unfortunately, I can not reproduce this problem locally.

Comment by James A Simmons [ 29/Jul/15 ]

Yes LU-6831 helped with the revalidate FID bug.

Comment by James A Simmons [ 29/Jul/15 ]

For my DNE2 testing here is the list of patches I running against:

http://review.whamcloud.com/#/c/14346
http://review.whamcloud.com/#/c/14747
http://review.whamcloud.com/#/c/15594
http://review.whamcloud.com/#/c/15720
http://review.whamcloud.com/#/c/15576
http://review.whamcloud.com/#/c/15730
http://review.whamcloud.com/#/c/15692
http://review.whamcloud.com/#/c/15691
http://review.whamcloud.com/#/c/15682
http://review.whamcloud.com/#/c/15690
http://review.whamcloud.com/#/c/15721
http://review.whamcloud.com/#/c/15724
http://review.whamcloud.com/#/c/15728
http://review.whamcloud.com/#/c/15770

Comment by Jessica A. Popp (Inactive) [ 30/Jul/15 ]

Translating James' list to ticket numbers for tracking purposes:
LU-6427
LU-6586
LU-6819
LU-6831
LU-6840
LU-6846
LU-6874
LU-6875
LU-6880
LU-6881
LU-6896
LU-6904
LU-6906
LU-6916

Comment by James A Simmons [ 03/Aug/15 ]

The patch for this ticket landed but I like to see this kept open to handle any further bug reports.

Comment by Di Wang [ 03/Aug/15 ]

Sorry, it might be a mistakes, even the patch on this ticket is not landed.

Comment by James A Simmons [ 03/Aug/15 ]

An update in my latest testing. I'm still seeing problems when creating 1 million+ files per directory. Clearing out the debug logs I see the problem is only on the client side. When running a application I see:

command line used: /lustre/sultan/stf008/scratch/jsimmons/mdtest -I 100000 -i 5 -d /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test/shared_1000k_10
Path: /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test
FS: 21.8 TiB Used FS: 0.2% Inodes: 58.7 Mi Used Inodes: 4.6%

10 tasks, 1000000 files/directories
aprun: Apid 3172: Caught signal Window changed, sending to application
08/03/2015 10:34:45: Process 0(nid00028): FAILED in create_remove_directory_tree, Unable to remove directory: No such file or directory
Rank 0 [Mon Aug 3 10:34:45 2015] [c0-0c0s1n2] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
_pmiu_daemon(SIGCHLD): [NID 00028] [c0-0c0s1n2] [Mon Aug 3 10:34:45 2015] PE RANK 0 exit signal Aborted
aprun: Apid 3172: Caught signal Interrupt, sending to application
_pmiu_daemon(SIGCHLD): [NID 00012] [c0-0c0s6n0] [Mon Aug 3 10:50:50 2015] PE RANK 7 exit signal Interrupt
_pmiu_daemon(SIGCHLD): [NID 00018] [c0-0c0s6n2] [Mon Aug 3 10:50:50 2015] PE RANK 9 exit signal Interrupt
_pmiu_daemon(SIGCHLD): [NID 00013] [c0-0c0s6n1] [Mon Aug 3 10:50:50 2015] PE RANK 8 exit signal Interrupt

After the test failed any attempt to remove the files create by these test fail. When I attempt to remove the files I see the following errors in dmesg.

LustreError: 5430:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2
LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2
LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) Skipped 7 previous similar messages
LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2

DiWang have you seen these errors during your testing?

Comment by Di Wang [ 03/Aug/15 ]

James: no, I did not see these errors? Could you please collect -1 debug log on client side, when you remove one of these files? thanks

Comment by Di Wang [ 06/Aug/15 ]

James: Any news for this -2 problem? Thanks

Comment by James A Simmons [ 06/Aug/15 ]

Testing to see if the problem exist on directory striped across 8 MDS servers. Waiting for the results. I will push some log data soon for you.

Comment by Gerrit Updater [ 07/Aug/15 ]

Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/15720/
Subject: LU-6831 lmv: revalidate the dentry for striped dir
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a17909a92da74cb26fb9bf2824f968b2adf0897e

Comment by James A Simmons [ 11/Aug/15 ]

I attached my client logs to LU-6984.

Comment by James A Simmons [ 26/Aug/15 ]

Due to the lose of some of my MDS servers I attempted to create new striped directories today but instead I get this error every time.

lfs setdirstripe -c 4 /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test
error on LL_IOC_LMV_SETSTRIPE '/lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test' (3): Invalid argument
error: setdirstripe: create stripe dir '/lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test' failed

This happens even when I'm root.

Comment by Di Wang [ 26/Aug/15 ]

could you please get the debug log(-1 level) on MDT0? I assume jsimmons is on MDT0 ? Thanks.

Comment by James A Simmons [ 10/Sep/15 ]

Here is the full log from the node that was crashing this morning

Just to let you know the IOC_LMV_SETSTRIPE is no longer a issue.

Comment by James A Simmons [ 18/Dec/15 ]

Updated my software stack and I'm seeing a lot of these on the OSS servers:

[94725.339746] Lustre: sultan-OST0004: already connected client sultan-MDT0000-mdtlov_UUID (at 10.37.248.155@o2ib1) with handle 0xb4b2e32f66f3ee41. Rejecting client with the same UUID trying to reconnect with handle 0x157ffaac64917bbd

Its seems to be only MDS1 having this. On that MDS the error message is:

95881.016995] LustreError: 137-5: sultan-MDT0001_UUID: not available for connect from 10.37.248.130@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server.

Comment by James A Simmons [ 23/Dec/15 ]

a soft lockup happing on a spin lock -

Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.147894] BUG: soft lockup - CPU#0 stuck for 67s! [osp_up7-0:20904]
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.152894] BUG: soft lockup - CPU#1 stuck for 67s! [osp_up4-0:20901]

Dec 22 10:54:26 feral17.ccs.ornl.gItsov kernel: [ 793.152993] Pid: 20901, comm: osp_up4-0 Tainted: P --------------- 2.6.32
504.30.3.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.152996] RIP: 0010:[<ffffffff8152da2e>] [<ffffffff8152da2e>] _spin_lock+0x1e/0x30
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153004] RSP: 0018:ffff8817d793dda0 EFLAGS: 00000202
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153005] RAX: 0000000000000002 RBX: ffff8817d793dda0 RCX: 0000000000000000
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153007] RDX: 0000000000000003 RSI: ffff8817d793dea8 RDI: ffff880bcd9e8d50
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153009] RBP: ffffffff8100bc0e R08: ffff8817d793c000 R09: 00000000ffffffff
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153010] R10: 000000a7ad891c7a R11: 0000000000000001 R12: 0000000000000000
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153012] R13: 0000000000000000 R14: 0000000000015900 R15: 0000000000000000
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153014] FS: 0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153016] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153017] CR2: 0000003d64205380 CR3: 0000000001a85000 CR4: 00000000000007e0
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153021] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153023] Process osp_up4-0 (pid: 20901, threadinfo ffff8817d793c000, task ffff881828342ab0)
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153024] Stack:
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153025] ffff8817d793ddf0 ffffffffa13ab4f9 0000000000000246 ffff8817d793dea8
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153027] <d> ffff8817d793ddf0 ffff881828342ab0 ffff880bcd9e8d40 ffff880bcd8d0800
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153030] <d> ffff8817d793de40 ffff8817d793dea8 ffff8817d793dee0 ffffffffa13b0074
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153032] Call Trace:
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153055] [<ffffffffa13ab4f9>] ? osp_get_next_request+0x29/0x1a0 [osp]
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153066] [<ffffffffa13b0074>] ? osp_send_update_thread+0x2f4/0x5b0 [osp]
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153071] [<ffffffff81064d00>] ? default_wake_function+0x0/0x20

Comment by Di Wang [ 28/Dec/15 ]

James: I just updated the patch http://review.whamcloud.com/#/c/16969/ please retry, thanks.

Comment by James A Simmons [ 29/Dec/15 ]

Yep. I'm testing it right now.

Generated at Sat Feb 10 02:03:37 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.