Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-6831] The ticket for tracking all DNE2 bugs https://jira.whamcloud.com/browse/LU-6831 Lustre <p>This ticket is for tracking all of DNE2 bugs.</p> LU-6831

The ticket for tracking all DNE2 bugs

Bug Minor Reopened Unresolved Di Wang Di Wang dne2 llnl Thu, 9 Jul 2015 18:35:56 +0000 Tue, 25 Feb 2020 18:23:26 +0000 Lustre 2.8.0 Lustre 2.9.0 0 14 <p>In my latest testing I'm running into this bug:</p> <p>mdtest-1.8.3 was launched with 1 total task(s) on 1 nodes<br/> Command line used: /lustre/sultan/stf008/scratch/jsimmons/mdtest -I 10000 -i 5 -d /lustre/sultan/stf008/scratch/jsimm<br/> ons/dne2_2_mds_md_test/shared_10k_1<br/> Path: /lustre/sultan/stf008/scratch/jsimmons/dne2_2_mds_md_test<br/> FS: 21.8 TiB Used FS: 0.2% Inodes: 82.7 Mi Used Inodes: 0.0%<br/> 07/13/2015 16:12:12: Process 0(): FAILED in main, Unable to create test directory path: Object is remote<br/> Rank 0 <span class="error">[Mon Jul 13 16:12:12 2015]</span> <span class="error">[c0-0c0s1n2]</span> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0<br/> _pmiu_daemon(SIGCHLD): <span class="error">[NID 00028]</span> <span class="error">[c0-0c0s1n2]</span> <span class="error">[Mon Jul 13 16:12:12 2015]</span> PE RANK 0 exit signal Aborted<br/> Application 2640 exit codes: 134<br/> Application 2640 resources: utime ~0s, stime ~0s, Rss ~3856, inblocks ~5254, outblocks ~13678</p> <p>It is very easy to reproduce. What I did was create a stripe directory of count 2 at a index of 1. The reason is I'm avoiding using MDS0 which has the smallest MDT.</p> <p>James: I can run this on my node</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@testnode mdtest]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_testnode-lv_root 27228028 10704408 15133848 42% / tmpfs 4014860 0 4014860 0% /dev/shm /dev/sda1 487652 48320 413732 11% /boot 192.168.167.1:/Users/wangdi/work 243358976 188184448 54918528 78% /work /dev/loop2 133560 1904 121932 2% /mnt/mds3 /dev/loop3 133560 1908 121928 2% /mnt/mds4 /dev/loop4 358552 13900 324420 5% /mnt/ost1 /dev/loop5 358552 13904 324416 5% /mnt/ost2 /dev/loop0 133560 2192 121644 2% /mnt/mds1 /dev/loop1 133560 2040 121796 2% /mnt/mds2 testnode@tcp:/lustre 717104 27804 648836 5% /mnt/lustre [root@testnode mdtest]# cd /work/lustre_release_work/lustre-release_new/lustre/tests/ [root@testnode tests]# pwd /work/lustre_release_work/lustre-release_new/lustre/tests [root@testnode tests]# dne2_2_mds_md_test^C [root@testnode tests]# ../utils/lfs mkdir -i1 -c2 /mnt/lustre/test [root@testnode tests]# /work/mdtest/ COPYRIGHT Makefile mdtest mdtest.1 README RELEASE_LOG scripts/ [root@testnode tests]# /work/mdtest/mdtest -I 10000 -i 5 -d /mnt/lustre/test/shared_10k_1 -- started at 07/12/2015 00:10:25 -- mdtest-1.9.3 was launched with 1 total task(s) on 1 node(s) Command line used: /work/mdtest/mdtest -I 10000 -i 5 -d /mnt/lustre/test/shared_10k_1 Path: /mnt/lustre/test FS: 0.7 GiB Used FS: 3.9% Inodes: 0.2 Mi Used Inodes: 0.5% 1 tasks, 10000 files/directories SUMMARY: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 6005.343 5420.750 5707.103 218.750 Directory stat : 5607.789 5152.637 5289.370 166.391 Directory removal : 5475.998 5276.844 5368.197 78.798 File creation : 3563.760 3035.905 3204.669 194.226 File stat : 2883.332 2761.046 2820.534 41.375 File read : 3320.368 2823.954 3061.123 194.269 File removal : 5018.647 4533.453 4712.663 186.385 Tree creation : 4696.869 2688.656 3332.096 703.002 Tree removal : 2096.104 1976.581 2042.187 48.555 -- finished at 07/12/2015 00:11:53 -- </pre> </div></div> <p>Could you please check if these values has been set correctly? Thanks</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@testnode lustre-release_new]# ./lustre/utils/lctl get_param mdt.*.enable_remote_dir mdt.lustre-MDT0000.enable_remote_dir=1 mdt.lustre-MDT0001.enable_remote_dir=1 mdt.lustre-MDT0002.enable_remote_dir=1 mdt.lustre-MDT0003.enable_remote_dir=1 </pre> </div></div> <p>Doh. I assume remote_dir was enabled by default. I just set it. Seems to work now. The strange thing is even with remote_dir=0 everywhere when I set the directory MDS stripe to > 1 as long as the index was zero it appeared to work.</p> <p>Yes as root it works but not as a regular user. I created a directory with the count = 2 and index = 1 then I did the -D flags so any directories created under my DNE2 directory would inherit the properties. As my self (non-root) I tried a mkdir and it failed.</p> <p>James: you need set these value to -1 to make sure all users can create remote and striped dir</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@mds01 ~]# lctl get_param mdt.*.enable_remote_dir_gid mdt.lustre-MDT0000.enable_remote_dir_gid=0 mdt.lustre-MDT0004.enable_remote_dir_gid=0 </pre> </div></div> <p>Nope. Creation of the directories appear to only work part of the time. Some times it does work and some times it does not work. Running mdtest never can run for me.</p> <p>If you run the test as non-root user, then you probably need this patch </p> <p><a href="http://review.whamcloud.com/#/c/13990/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13990/</a> </p> <p>And also please set enable_remote_dir_gid=-1 on all of MDTs.</p> <p>Patch 13990 did the trick. Now I can create DNE2 striped directories as myself. Thanks.</p> <p>Updated to your latest patches and I lost the ability to create remote directories. Now I get the following errors:</p> <p>[ 1016.185382] Lustre: 19975:0:(lmv_obd.c:297:lmv_init_ea_size()) sultan-clilmv-ffff88080ac8ec00: NULL export for 1<br/> [ 1016.195740] Lustre: 19975:0:(lmv_obd.c:297:lmv_init_ea_size()) Skipped 26 previous similar messages<br/> [ 1016.498646] Lustre: Mounted sultan-client<br/> [ 1022.254520] LustreError: 19996:0:(lmv_obd.c:1332:lmv_fid_alloc()) Can't alloc new fid, rc -19</p> <p>which patches? could you please list your patches here? thanks</p> <p><a href="http://review.whamcloud.com/#/c/13990" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13990</a><br/> <a href="http://review.whamcloud.com/#/c/15572" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15572</a><br/> <a href="http://review.whamcloud.com/#/c/15576" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15576</a></p> <p>Strange, does your build include this patch <a href="http://review.whamcloud.com/#/c/15269/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15269/</a> ? If it does, please remove this one. and retry? thanks</p> <p>Nope. I found the source of the problems. It was the patch from <a href="https://jira.whamcloud.com/browse/LU-6341" title="Do not check security when accessing default stripeEA" class="issue-link" data-issue-key="LU-6341"><del>LU-6341</del></a>. Once I removed it the problems went away.</p> <p>Doing more testing I found that the patch from <a href="https://jira.whamcloud.com/browse/LU-6341" title="Do not check security when accessing default stripeEA" class="issue-link" data-issue-key="LU-6341"><del>LU-6341</del></a> was not the source of the bug but it exposed the issue reported in <a href="https://jira.whamcloud.com/browse/LU-6713" title="Noisy error messages on client while creating DNE filesystem" class="issue-link" data-issue-key="LU-6713"><del>LU-6713</del></a>. Once I applied patch 15269 the problem went away.</p> <p>Now I'm seeing clients get evicted during heavy meta data operations. Di Wang have you seen this behavior and does a patch exist to address this?</p> <p>James: What test did you run? Do you have the trace? I am not sure if there are such fixes. Thanks.</p> <p>I see what is triggering the client evictions. I'm getting these errors on the clients:</p> <p>LustreError: 10306:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 1 <span class="error">[0x2800013ba:0x84ad:0x0]</span>:<span class="error">[0x2400013c8:0x84ad:0x0]</span><br/> LustreError: 10306:0:(file.c:3104:ll_inode_revalidate_fini()) sultan: revalidate FID <span class="error">[0x2400013ce:0x3:0x0]</span> error: rc = -5<br/> LustreError: 10316:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 0 <span class="error">[0x2400013b3:0x14262:0x0]</span>:<span class="error">[0x2400013b3:0x14262:0x0]</span><br/> LustreError: 10863:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 0 <span class="error">[0x2400013b3:0x14262:0x0]</span>:<span class="error">[0x2400013b3:0x14262:0x0]</span><br/> LustreError: 10894:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 1 <span class="error">[0x2800013ab:0x14262:0x0]</span>:<span class="error">[0x2400013b3:0x14262:0x0]</span><br/> LustreError: 10894:0:(file.c:3104:ll_inode_revalidate_fini()) sultan: revalidate FID <span class="error">[0x2400013a4:0x5:0x0]</span> error: rc = -5</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15720" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15720</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6831" title="The ticket for tracking all DNE2 bugs" class="issue-link" data-issue-key="LU-6831">LU-6831</a> lmv: revalidate the dentry for striped dir<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 26133ac386693647ef3ee4480a6ce0cbb2773adb</p> <p>James: Please try this patch to see if it works? thanks. Unfortunately, I can not reproduce this problem locally.</p> <p>Yes <a href="https://jira.whamcloud.com/browse/LU-6831" title="The ticket for tracking all DNE2 bugs" class="issue-link" data-issue-key="LU-6831">LU-6831</a> helped with the revalidate FID bug.</p> <p>For my DNE2 testing here is the list of patches I running against:</p> <p><a href="http://review.whamcloud.com/#/c/14346" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14346</a><br/> <a href="http://review.whamcloud.com/#/c/14747" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14747</a><br/> <a href="http://review.whamcloud.com/#/c/15594" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15594</a><br/> <a href="http://review.whamcloud.com/#/c/15720" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15720</a><br/> <a href="http://review.whamcloud.com/#/c/15576" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15576</a><br/> <a href="http://review.whamcloud.com/#/c/15730" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15730</a><br/> <a href="http://review.whamcloud.com/#/c/15692" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15692</a><br/> <a href="http://review.whamcloud.com/#/c/15691" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15691</a><br/> <a href="http://review.whamcloud.com/#/c/15682" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15682</a><br/> <a href="http://review.whamcloud.com/#/c/15690" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15690</a><br/> <a href="http://review.whamcloud.com/#/c/15721" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15721</a><br/> <a href="http://review.whamcloud.com/#/c/15724" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15724</a><br/> <a href="http://review.whamcloud.com/#/c/15728" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15728</a><br/> <a href="http://review.whamcloud.com/#/c/15770" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15770</a></p> <p>Translating James' list to ticket numbers for tracking purposes:<br/> <a href="https://jira.whamcloud.com/browse/LU-6427" title="(osd_index.c:168:osd_index_it_init()) LBUG" class="issue-link" data-issue-key="LU-6427"><del>LU-6427</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6586" title=""lctl conf_param testfs-MDT0001.mdc.active=0" does not deactivate MDC permanently" class="issue-link" data-issue-key="LU-6586"><del>LU-6586</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6819" title="LBUG ASSERTION( tdtd->tdtd_last_update_transno <= transno ) failed" class="issue-link" data-issue-key="LU-6819"><del>LU-6819</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6831" title="The ticket for tracking all DNE2 bugs" class="issue-link" data-issue-key="LU-6831">LU-6831</a><br/> <a href="https://jira.whamcloud.com/browse/LU-6840" title="update memory reply data in DNE update replay " class="issue-link" data-issue-key="LU-6840"><del>LU-6840</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6846" title="dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed: " class="issue-link" data-issue-key="LU-6846"><del>LU-6846</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6874" title="Potential buffer overflow in out_handler_find" class="issue-link" data-issue-key="LU-6874"><del>LU-6874</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6875" title="thandle_get_sub_by_dt dereferences ERR_PTR pointer on error" class="issue-link" data-issue-key="LU-6875"><del>LU-6875</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6880" title="recovery timeout during 24 hours failover test" class="issue-link" data-issue-key="LU-6880"><del>LU-6880</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6881" title="sub_trans_commit_cb() is racy" class="issue-link" data-issue-key="LU-6881"><del>LU-6881</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6896" title="update llog object is missing during recovery." class="issue-link" data-issue-key="LU-6896"><del>LU-6896</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6904" title="linkea prepare in mdt_reint_rename cause deadlock in 24 hours failover" class="issue-link" data-issue-key="LU-6904"><del>LU-6904</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6906" title="During 24 hours DNE test, one of MDS can not be mounted after restarts." class="issue-link" data-issue-key="LU-6906"><del>LU-6906</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-6916" title="unable to handle kernel NULL pointer dereference at (null) in tgt_client_free()" class="issue-link" data-issue-key="LU-6916"><del>LU-6916</del></a></p> <p>The patch for this ticket landed but I like to see this kept open to handle any further bug reports.</p> <p>Sorry, it might be a mistakes, even the patch on this ticket is not landed.</p> <p>An update in my latest testing. I'm still seeing problems when creating 1 million+ files per directory. Clearing out the debug logs I see the problem is only on the client side. When running a application I see:</p> <p>command line used: /lustre/sultan/stf008/scratch/jsimmons/mdtest -I 100000 -i 5 -d /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test/shared_1000k_10<br/> Path: /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test<br/> FS: 21.8 TiB Used FS: 0.2% Inodes: 58.7 Mi Used Inodes: 4.6%</p> <p>10 tasks, 1000000 files/directories<br/> aprun: Apid 3172: Caught signal Window changed, sending to application<br/> 08/03/2015 10:34:45: Process 0(nid00028): FAILED in create_remove_directory_tree, Unable to remove directory: No such file or directory<br/> Rank 0 <span class="error">[Mon Aug 3 10:34:45 2015]</span> <span class="error">[c0-0c0s1n2]</span> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0<br/> _pmiu_daemon(SIGCHLD): <span class="error">[NID 00028]</span> <span class="error">[c0-0c0s1n2]</span> <span class="error">[Mon Aug 3 10:34:45 2015]</span> PE RANK 0 exit signal Aborted<br/> aprun: Apid 3172: Caught signal Interrupt, sending to application<br/> _pmiu_daemon(SIGCHLD): <span class="error">[NID 00012]</span> <span class="error">[c0-0c0s6n0]</span> <span class="error">[Mon Aug 3 10:50:50 2015]</span> PE RANK 7 exit signal Interrupt<br/> _pmiu_daemon(SIGCHLD): <span class="error">[NID 00018]</span> <span class="error">[c0-0c0s6n2]</span> <span class="error">[Mon Aug 3 10:50:50 2015]</span> PE RANK 9 exit signal Interrupt<br/> _pmiu_daemon(SIGCHLD): <span class="error">[NID 00013]</span> <span class="error">[c0-0c0s6n1]</span> <span class="error">[Mon Aug 3 10:50:50 2015]</span> PE RANK 8 exit signal Interrupt</p> <p>After the test failed any attempt to remove the files create by these test fail. When I attempt to remove the files I see the following errors in dmesg.</p> <p>LustreError: 5430:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2<br/> LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2<br/> LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) Skipped 7 previous similar messages<br/> LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2</p> <p>DiWang have you seen these errors during your testing?</p> <p>James: no, I did not see these errors? Could you please collect -1 debug log on client side, when you remove one of these files? thanks</p> <p>James: Any news for this -2 problem? Thanks</p> <p>Testing to see if the problem exist on directory striped across 8 MDS servers. Waiting for the results. I will push some log data soon for you.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15720/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15720/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6831" title="The ticket for tracking all DNE2 bugs" class="issue-link" data-issue-key="LU-6831">LU-6831</a> lmv: revalidate the dentry for striped dir<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a17909a92da74cb26fb9bf2824f968b2adf0897e</p> <p>I attached my client logs to <a href="https://jira.whamcloud.com/browse/LU-6984" title="Failure to delete over a million files in a DNE2 directory." class="issue-link" data-issue-key="LU-6984"><del>LU-6984</del></a>.</p> <p>Due to the lose of some of my MDS servers I attempted to create new striped directories today but instead I get this error every time.</p> <p>lfs setdirstripe -c 4 /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test<br/> error on LL_IOC_LMV_SETSTRIPE '/lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test' (3): Invalid argument<br/> error: setdirstripe: create stripe dir '/lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test' failed</p> <p>This happens even when I'm root.</p> <p>could you please get the debug log(-1 level) on MDT0? I assume jsimmons is on MDT0 ? Thanks.</p> <p>Here is the full log from the node that was crashing this morning</p> <p>Just to let you know the IOC_LMV_SETSTRIPE is no longer a issue.</p> <p>Updated my software stack and I'm seeing a lot of these on the OSS servers:</p> <p><span class="error">[94725.339746]</span> Lustre: sultan-OST0004: already connected client sultan-MDT0000-mdtlov_UUID (at 10.37.248.155@o2ib1) with handle 0xb4b2e32f66f3ee41. Rejecting client with the same UUID trying to reconnect with handle 0x157ffaac64917bbd</p> <p>Its seems to be only MDS1 having this. On that MDS the error message is:</p> <p>95881.016995] LustreError: 137-5: sultan-MDT0001_UUID: not available for connect from 10.37.248.130@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server.</p> <p>a soft lockup happing on a spin lock -</p> <p>Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.147894] BUG: soft lockup - CPU#0 stuck for 67s! <span class="error">[osp_up7-0:20904]</span><br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.152894] BUG: soft lockup - CPU#1 stuck for 67s! <span class="error">[osp_up4-0:20901]</span></p> <p>Dec 22 10:54:26 feral17.ccs.ornl.gItsov kernel: [ 793.152993] Pid: 20901, comm: osp_up4-0 Tainted: P --------------- 2.6.32<br/> 504.30.3.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.152996] RIP: 0010:<span class="error">[<ffffffff8152da2e>]</span> <span class="error">[<ffffffff8152da2e>]</span> _spin_lock+0x1e/0x30<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153004] RSP: 0018:ffff8817d793dda0 EFLAGS: 00000202<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153005] RAX: 0000000000000002 RBX: ffff8817d793dda0 RCX: 0000000000000000<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153007] RDX: 0000000000000003 RSI: ffff8817d793dea8 RDI: ffff880bcd9e8d50<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153009] RBP: ffffffff8100bc0e R08: ffff8817d793c000 R09: 00000000ffffffff<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153010] R10: 000000a7ad891c7a R11: 0000000000000001 R12: 0000000000000000<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153012] R13: 0000000000000000 R14: 0000000000015900 R15: 0000000000000000<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153014] FS: 0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153016] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153017] CR2: 0000003d64205380 CR3: 0000000001a85000 CR4: 00000000000007e0<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153021] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153023] Process osp_up4-0 (pid: 20901, threadinfo ffff8817d793c000, task ffff881828342ab0)<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153024] Stack:<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153025] ffff8817d793ddf0 ffffffffa13ab4f9 0000000000000246 ffff8817d793dea8<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153027] <d> ffff8817d793ddf0 ffff881828342ab0 ffff880bcd9e8d40 ffff880bcd8d0800<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153030] <d> ffff8817d793de40 ffff8817d793dea8 ffff8817d793dee0 ffffffffa13b0074<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153032] Call Trace:<br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153055] <span class="error">[<ffffffffa13ab4f9>]</span> ? osp_get_next_request+0x29/0x1a0 <span class="error">[osp]</span><br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153066] <span class="error">[<ffffffffa13b0074>]</span> ? osp_send_update_thread+0x2f4/0x5b0 <span class="error">[osp]</span><br/> Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153071] <span class="error">[<ffffffff81064d00>]</span> ? default_wake_function+0x0/0x20</p> <p>James: I just updated the patch <a href="http://review.whamcloud.com/#/c/16969/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16969/</a> please retry, thanks.</p> <p>Yep. I'm testing it right now.</p> Related LU-5222 LU-6297 LU-6819 LU-6834 LU-6837 LU-6840 LU-6880 LU-7039 LU-7285 LU-7318 LU-5951 LU-6475 LU-6602 LU-6741 LU-6896 LU-6916 LU-6977 LU-6981 LU-6998 LU-6999 LU-7091 LU-5344 LU-6586 LU-6838 LU-6846 LU-6852 LU-6859 LU-6904 LU-6924 LU-6928 LU-6997 LU-7005 LU-7008 LU-6637 LU-6638 LU-6639 LU-6888 LU-6906 LU-6968 LU-7009 LU-7050 LU-7074 LU-7213 LU-6362 LU-6640 LU-3538 LU-6773 LU-7827 LU-6826 LU-6984 LU-7107 LU-7013 LU-6284 LU-7293 LU-7357 Development Rank 1|hzxhvr: Rank (Obsolete) 9223372036854775807 Severity