https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3577 Lustre <p>We have a Lustre 2.1.5 system with two MDSes (active / standby), and two OSSes (active / active). Each OSS has 6 OSTs.</p> <p>We filled the file system to 100%. To remove the files, one Lustre client ran the following script:</p> <p>rm -rf /mnt/hss45/ost/ost-00/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-01/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-02/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-03/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-04/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-05/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-06/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-07/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-08/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-09/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-10/* &amp;<br/> rm -rf /mnt/hss45/ost/ost-11/* &amp;</p> <p>One OSS crashed with this error: <br/> BUG: soft lockup - CPU#25 stuck for 67s! <span class="error">&#91;jbd2/dm-8-8:8966&#93;</span><br/> . . .<br/> Kernel panic - not syncing: softlockup: hung tasks</p> <p>The OSS was STONITH'ed.</p> <p>Shortly thereafter, the second OSS got the same error:</p> <p>BUG: soft lockup - CPU#17 stuck for 67s! <span class="error">&#91;jbd2/dm-6-8:21440&#93;</span><br/> Kernel panic - not syncing: softlockup: hung tasks</p> <p>I have attached the full console output. There was nothing in /var/log/messages.</p> Kernel: 2.6.32-279.19.1.el6_lustre.2.1.5_1.0.3<br/> Distro: CentOS release 6.4 (Final)<br/> CPUs: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz<br/> 32 Cores<br/> 128GB RAM on MDSes; 64GB RAM on OSSes<br/> Active/Standby MDS<br/> Two OSSes<br/> 12 OSTs<br/> Each OST is 22T LU-3577

BUG: soft lockup - CPU#25 stuck for 67s! [jbd2/dm-8-8:8966]; Kernel panic - not syncing: softlockup: hung tasks

Bug Major Resolved Won't Fix Peter Jones Roger Spellman Thu, 11 Jul 2013 17:40:45 +0000 Wed, 21 Mar 2018 15:07:15 +0000 Wed, 21 Mar 2018 15:07:15 +0000 Lustre 2.1.5 0 2 <p>Thanks for the report Roger. Given that you are running RHEL 6.4 is there any reason you chose 2.1.5 over 2.1.6? Other than rebuilding for RHEL6.4, are there any other changes you made from a standard 2.1.5?</p> <p>Peter,</p> <p>We are using 2.1.5 because we started this project a couple of months ago (before 2.1.6 was release), and we have promised a 2.1.x release to a customer pretty soon. We are pretty far into our QA cycle, so switching Lustre versions right now would set us back a bit.</p> <p>We will go to 2.1.6 very soon. But, if you say that this is a known bug in 2.1.5 that is fixed in 2.1.6, that will push us to 2.1.6 even sooner.</p> <p>We make changes to configure scripts and Makefiles, so that we can build on our build machine. </p> <p>We make some minor functional changes to the code (we made them some time ago, in earlier releases). Here are the patches that are functional changes.</p> <p>diff -rcN -x '<b>~' -x '</b>.orig' /build/lustre/lustre-2.1.5/lustre/ldlm/ldlm_pool.c 2.1.5/trunk/lustre-working_lustre.patch/lustre/ldlm/ldlm_pool.c</p> <ul> <li> <ul> <li> <ul> <li>/build/lustre/lustre-2.1.5/lustre/ldlm/ldlm_pool.c Tue Apr 30 10:34:06 2013</li> </ul> <ul class="alternate" type="square"> <li>2.1.5/trunk/lustre-working_lustre.patch/lustre/ldlm/ldlm_pool.c Tue Jun 18 15:23:14 2013<br/> ***************</li> </ul> <ul> <li>143,149 ****<br/> /*</li> </ul> </li> </ul> </li> <li>Max age for locks on clients.<br/> */<br/> ! #define LDLM_POOL_MAX_AGE (36000)</li> </ul> <p> /*</p> <ul> <li>The granularity of SLV calculation. <ul class="alternate" type="square"> <li> <ul class="alternate" type="square"> <li>143,157 ----<br/> /*</li> </ul> </li> </ul> </li> <li>Max age for locks on clients.<br/> */<br/> ! //#define LDLM_POOL_MAX_AGE (36000)<br/> ! /*<br/> ! * Max age for locks on clients.<br/> ! * Terascala: Set to default 2 minute max age<br/> ! * Units are seconds.<br/> ! * This actually kicks in lru eviction after 7 minutes at this setting.<br/> ! */<br/> ! static u_int32_t ldlm_pool_max_age=120;<br/> !</li> </ul> <p> /*</p> <ul> <li>The granularity of SLV calculation.<br/> *************** <ul> <li> <ul> <li>162,171 ****<br/> static inline _<em>u64 ldlm_pool_slv_max(</em>_u32 L) { /* ! * Allow to have all locks for 1 client for 10 hrs. ! * Formula is the following: limit * 10h / 1 client. */ ! __u64 lim = (__u64)L * LDLM_POOL_MAX_AGE / 1; return lim; }</li> </ul> </li> </ul> </li> </ul> <p>&#8212; 170,179 ----<br/> static inline _<em>u64 ldlm_pool_slv_max(</em>_u32 L)</p> { /* ! * Allow to have all locks for 1 client for 10 minutes. ! * Formula is the following: limit * 2 min / 1 client. */ ! __u64 lim = (__u64)L * ldlm_pool_max_age / 1; /* Terascala */ return lim; } <p>***************</p> <ul> <li> <ul> <li> <ul> <li>805,810 ****</li> </ul> <ul class="alternate" type="square"> <li>813,825 ----<br/> pool_vars<span class="error">&#91;0&#93;</span>.write_fptr = lprocfs_wr_atomic; <br/> lprocfs_add_vars(pl-&gt;pl_proc_dir, pool_vars, 0);</li> </ul> </li> </ul> </li> </ul> <p>+ /* Terascala */<br/> + snprintf(var_name, MAX_STRING_SIZE, "ldlm_pool_max_age");<br/> + pool_vars<span class="error">&#91;0&#93;</span>.data = &amp;ldlm_pool_max_age;<br/> + pool_vars<span class="error">&#91;0&#93;</span>.read_fptr = lprocfs_rd_uint;<br/> + pool_vars<span class="error">&#91;0&#93;</span>.write_fptr = lprocfs_wr_uint;<br/> + lprocfs_add_vars(pl-&gt;pl_proc_dir, pool_vars, 0);<br/> +<br/> snprintf(var_name, MAX_STRING_SIZE, "state");<br/> pool_vars<span class="error">&#91;0&#93;</span>.data = pl;<br/> pool_vars<span class="error">&#91;0&#93;</span>.read_fptr = lprocfs_rd_pool_state;<br/> diff -rcN -x '<b>~' -x '</b>.orig' /build/lustre/lustre-2.1.5/lustre/liblustre/super.c 2.1.5/trunk/lustre-working_lustre.patch/lustre/liblustre/super.c</p> <ul> <li> <ul> <li> <ul> <li>/build/lustre/lustre-2.1.5/lustre/liblustre/super.c Tue Apr 30 10:34:12 2013</li> </ul> <ul class="alternate" type="square"> <li>2.1.5/trunk/lustre-working_lustre.patch/lustre/liblustre/super.c Tue Jun 18 15:23:14 2013<br/> ***************</li> </ul> <ul> <li>141,146 ****</li> </ul> <ul class="alternate" type="square"> <li>141,147 ----<br/> struct mdt_body *body = md-&gt;body;<br/> struct lov_stripe_md *lsm = md-&gt;lsm;<br/> struct intnl_stat *st = llu_i2stat(inode);<br/> + struct ll_sb_info *sbi = ll_i2sbi(inode);</li> </ul> </li> </ul> </li> </ul> <p> LASSERT ((lsm != NULL) == ((body-&gt;valid &amp; OBD_MD_FLEASIZE) != 0));</p> <p>***************</p> <ul> <li> <ul> <li> <ul> <li>181,187 ****<br/> lli-&gt;lli_lvb.lvb_ctime = body-&gt;ctime;<br/> }<br/> if (S_ISREG(st-&gt;st_mode))<br/> ! st-&gt;st_blksize = min(2UL * PTLRPC_MAX_BRW_SIZE, LL_MAX_BLKSIZE);<br/> else<br/> st-&gt;st_blksize = 4096;<br/> if (body-&gt;valid &amp; OBD_MD_FLUID)</li> </ul> <ul class="alternate" type="square"> <li>182,188 ----<br/> lli-&gt;lli_lvb.lvb_ctime = body-&gt;ctime;<br/> }<br/> if (S_ISREG(st-&gt;st_mode))<br/> ! st-&gt;st_blksize = min(2UL * PTLRPC_MAX_BRW_SIZE, 1UL&lt;&lt; sbi-&gt;ll_max_blksize_bits);<br/> else<br/> st-&gt;st_blksize = 4096;<br/> if (body-&gt;valid &amp; OBD_MD_FLUID)</li> </ul> </li> </ul> </li> </ul> <p>Hope this helps.</p> Development End date Thu, 11 Jul 2013 17:40:45 +0000 Rank 1|hzvv6f: Rank (Obsolete) 9055 Severity Start date Thu, 11 Jul 2013 17:40:45 +0000