https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4971 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run:<br/> <a href="http://maloo.whamcloud.com/test_sets/598caeaa-cd2c-11e3-b548-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/598caeaa-cd2c-11e3-b548-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/fc94ece0-a552-11e3-9fee-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fc94ece0-a552-11e3-9fee-52540035b04c</a></p> <p>The sub-test test_2 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: sanity-scrub 2</p> LU-4971

sanity-scrub test_2: ldlm_lock2desc()) ASSERTION( lock->l_policy_data.l_inodebits.bits == (MDS_INODELOCK_LOOKUP | MDS_INODELOCK_UPDATE | MDS_INODELOCK_LAYOUT)failed

Bug Critical Resolved Fixed nasf Maloo Mon, 28 Apr 2014 18:52:45 +0000 Wed, 8 Feb 2017 16:54:53 +0000 Thu, 10 Jul 2014 23:52:41 +0000 Lustre 2.6.0 Lustre 2.5.4 Lustre 2.6.0 0 9 <p>Some log on the client side:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">23:56:52:LustreError: 23554:0:(ldlm_lock.c:671:ldlm_lock2desc()) ASSERTION( lock-&gt;l_policy_data.l_inodebits.bits == (MDS_INODELOCK_LOOKUP | MDS_INODELOCK_UPDATE | MDS_INODELOCK_LAYOUT) ) failed: Inappropriate inode lock bits during conversion 3 23:56:52:LustreError: 23554:0:(ldlm_lock.c:671:ldlm_lock2desc()) LBUG 23:56:52:Pid: 23554, comm: diff 23:56:52: 23:56:52:Call Trace: 23:56:52: [&lt;ffffffffa03d0895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 23:56:52: [&lt;ffffffffa03d0e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 23:56:52: [&lt;ffffffffa08dc239&gt;] ldlm_lock2desc+0x179/0x180 [ptlrpc] 23:56:52: [&lt;ffffffffa08eed90&gt;] ldlm_cli_enqueue+0x1f0/0x790 [ptlrpc] 23:56:52: [&lt;ffffffffa0914c0a&gt;] ? ptlrpc_request_set_replen+0x3a/0x60 [ptlrpc] 23:56:52: [&lt;ffffffffa08f3bd0&gt;] ? ldlm_completion_ast+0x0/0x930 [ptlrpc] 23:56:52: [&lt;ffffffffa13434d0&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] 23:56:52: [&lt;ffffffffa0a3d88e&gt;] mdc_enqueue+0x2be/0x1b00 [mdc] 23:56:52: [&lt;ffffffffa08e25e5&gt;] ? ldlm_lock_match+0x215/0x8b0 [ptlrpc] 23:56:52: [&lt;ffffffffa0a3f2ce&gt;] mdc_intent_lock+0x1fe/0x63f [mdc] 23:56:52: [&lt;ffffffffa13434d0&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] 23:56:52: [&lt;ffffffffa08f3bd0&gt;] ? ldlm_completion_ast+0x0/0x930 [ptlrpc] 23:56:52: [&lt;ffffffffa053ba28&gt;] ? lprocfs_counter_add+0x1a8/0x1c0 [obdclass] 23:56:52: [&lt;ffffffffa082eae1&gt;] lmv_intent_remote+0x481/0xa90 [lmv] 23:56:52: [&lt;ffffffffa13434d0&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] 23:56:52: [&lt;ffffffffa082fd49&gt;] lmv_intent_lookup+0xc59/0xd00 [lmv] 23:56:52: [&lt;ffffffffa13434d0&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] 23:56:52: [&lt;ffffffffa03d027b&gt;] ? cfs_set_ptldebug_header+0x2b/0xc0 [libcfs] 23:56:52: [&lt;ffffffffa0830ada&gt;] lmv_intent_lock+0x32a/0x380 [lmv] 23:56:52: [&lt;ffffffffa13434d0&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] 23:56:52: [&lt;ffffffffa1325ed8&gt;] ? ll_prep_md_op_data+0x1a8/0x490 [lustre] 23:56:52: [&lt;ffffffffa1344d79&gt;] ll_lookup_it+0x299/0xb50 [lustre] 23:56:52: [&lt;ffffffffa13434d0&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] 23:56:52: [&lt;ffffffffa134589f&gt;] ll_lookup_nd+0x26f/0x3e0 [lustre] 23:56:52: [&lt;ffffffff811a394e&gt;] ? d_alloc+0x13e/0x1b0 23:56:52: [&lt;ffffffff81198a25&gt;] do_lookup+0x1a5/0x230 23:56:52: [&lt;ffffffff81198db0&gt;] __link_path_walk+0x200/0xff0 23:56:52: [&lt;ffffffff8114a647&gt;] ? handle_pte_fault+0xf7/0xb00 23:56:52: [&lt;ffffffff81199e5a&gt;] path_walk+0x6a/0xe0 23:56:52: [&lt;ffffffff8119a06b&gt;] filename_lookup+0x6b/0xc0 23:56:52: [&lt;ffffffff81196996&gt;] ? final_putname+0x26/0x50 23:56:52: [&lt;ffffffff8119b197&gt;] user_path_at+0x57/0xa0 23:56:52: [&lt;ffffffff8104a98c&gt;] ? __do_page_fault+0x1ec/0x480 23:56:52: [&lt;ffffffff812826f5&gt;] ? _atomic_dec_and_lock+0x55/0x80 23:56:52: [&lt;ffffffff811aaa10&gt;] ? mntput_no_expire+0x30/0x110 23:56:52: [&lt;ffffffff8118e7a4&gt;] ? cp_new_stat+0xe4/0x100 23:56:52: [&lt;ffffffff8118e9e0&gt;] vfs_fstatat+0x50/0xa0 23:56:52: [&lt;ffffffff8118eb5b&gt;] vfs_stat+0x1b/0x20 23:56:52: [&lt;ffffffff8118eb84&gt;] sys_newstat+0x24/0x50 23:56:52: [&lt;ffffffff810e2057&gt;] ? audit_syscall_entry+0x1d7/0x200 23:56:52: [&lt;ffffffff810e1e4e&gt;] ? __audit_syscall_exit+0x25e/0x290 23:56:52: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>So it appears tha client side exp_connect_flags is somehow corrupted because we had no servers not supporting inode bits for many years now. Not since 1.4.x I suspect.</p> <p>It is strange that the LASSERT() is on the client, but LFSCK is running on the server (only some parts of the test scripts are setting up the environment on the client).</p> <p>Oleg notes that this problem indicates that the client is trying to use an IBITS lock (as it should) but it thinks the server does not support this (due to missing connect feature flags). That might indicate some problem or corruption with the connection state.</p> <p>Yes, I also suspected that it should not related with OI scrub, instead, might because of some connection race cases. I remembered that we have met similar connection flags issues about client_is_remote() before at LLNL site. But I cannot remember the detail...</p> <p><a href="https://testing.hpdd.intel.com/test_sets/099fd572-fd95-11e3-b84d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/099fd572-fd95-11e3-b84d-5254006e85c2</a></p> <p>sanity-scrub test 4 is displaying this error and crashing the client node.</p> <p>I'm not sure why so few logs were produced, but here are the logs from the client node: <a href="https://testing.hpdd.intel.com/test_sessions/77df2c0e-0092-11e4-a42b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/77df2c0e-0092-11e4-a42b-5254006e85c2</a></p> <p>I've run sanity-scrub four times on the OpenSFS cluster and sanity-scrub fails with this error each time in test 2, 4 or 5. </p> <p>I've run with one, two and four clients with two MDSs each with two MDTs and four OSSs with two OSTs each with Lustre 2.5.60 build # 2535.</p> <p>A debug patch to check the export status when assertion:</p> <p><a href="http://review.whamcloud.com/#/c/10958/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10958/</a></p> <p>I've run sanity-scrub twice with the debug patch and only hit this error one of the runs. Here is the output from the debug message for a single client run that failed in sanity-scrub test 2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... restore data restore EA /usr/lib64/lustre/tests remove recovery logs removed `/lustre/brpt/CATALOGS' starting MDTs without disabling OI scrub Starting client: c17: -o user_xattr,flock mds01@o2ib:/scratch /lustre/scratch Message from syslogd@c17 at Jul 3 11:36:24 ... kernel:LustreError: 22841:0:(ldlm_lock.c:672:ldlm_lock2desc()) ASSERTION( lock-&gt;l_policy_data.l_inodebits.bits == (MDS_INODELOCK_LOOKUP | MDS_INODELOCK_UPDATE | MDS_INODELOCK_LAYOUT) ) failed: Inappropriate inode lock bits during conversion, bits 0x3, exp ffff880827d6dc00, flags 0x0 Message from syslogd@c17 at Jul 3 11:36:24 ... kernel:LustreError: 22841:0:(ldlm_lock.c:672:ldlm_lock2desc()) LBUG </pre> </div></div> <p>Hmm, it has bits 0x3 set, so it is missing the LAYOUT flag. That would be possible if the file is being migrated, or maybe if it is a remote MDT object? I suspect that the LASSERT is wrong here, but I'm not positive yet as I haven't looked closely at the surrounding code. </p> <p>The bits may be right, because the former code cannot find OBD_CONNECT_IBITS in the connection_flags, so the layout ibits will be dropped automatically.</p> <p>Here we get the connection_flags as "0x0", that is impossible for any connection. So there should be data corruption.</p> <p>Since this only happens during sanity-scrub, I expect there is some kind of memory corruption or use-after-free problem in the code? It might be useful to run sanity-scrub on a kernel with CONFIG_DEBUG_SLAB and other memory debugging features (CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_OBJECTS, CONFIG_DEBUG_KMEMLEAK, CONFIG_DEBUG_MUTEXES, CONFIG_DEBUG_KOBJECT) enabled to see if this triggers a problem.</p> <p>I have run related tests locally for hundreds of times, but cannot reproduce the failure by myself. So I have to go though all related code. Finally, I found that: it may be not related with data corruption, but more seems race condition between the asynchronous connection and synchronous lock_enqueue operation. Consider the following scenario:</p> <p>1) There are two MDTs in the system, when the client mount, it will trigger MDS_CONNECT request to each of the MDT. But the connect request will be handled asynchronously. The mount processing will go ahead when the connection between the client and MDT0 is established, but will not wait for the connection between the client and the MDT1 to be established.</p> <p>2) After the client mount returned, the application tries to "stat $MOUNT/a/b", the directory "a" resides on the MDT0, the "b" resides on the MDT1. So the client will send lock_enqueue RPC to the MDT0 firstly.</p> <p>3) The RPC handler on the MDT0 finds that the target object "b" resides on the remote MDT1, then it replies the client to try to talk with the MDT1.</p> <p>4) The client gets the MDT0's reply, then calls lmv_intent_remote() to forward the request to the MDT1 via MDC1, but at that time, the connection between the client and the MDT1 has not been established yet.</p> <p>5) For most of other RPCs, above case is not a problem, because the client-side RPC sender (in ptlrpc) will handle kinds of connection issues. But for the ldlm_enqueue, it is some different, because before giving the ldlm_enqueue RPC to the sender (or to the ptlrpc), the sponsor will check the export connect flags that will not be set until the connection between the client and the MDT has been established. So at such time point, the sponsor of the ldlm_enqueue request will get "0" flags from the export connection flags, then trigger the ASSERTION as described in this ticket.</p> <p>So it is not a special bug for OI scrub, but more general issue for the whole Lustre framework. Because the sanity-scrub will access some cross-MDTs objects immediately after the client mount (without any wait), then it is more easy to reproduce the failure.</p> <p>About the fixing, I do not want to change the general connection/ldlm framework, because I am afraid of introducing some potential bugs. Instead, the patch will make the ldlm_request request to wait the connection to be established for such race case.</p> <p>James, would you please to verify the new patch? Thanks!<br/> <a href="http://review.whamcloud.com/#/c/10958/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10958/</a></p> <p>Hmm, I think this is the problem for single MDT FS too, i.e. ldlm_cli_enqueue should not use exp connection flag before it make sure the connection is established?<br/> This inodebit connect flag is being added even before 1.6 I guess? At least this inodebit interop code is added in 2007</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 113303973ec9f8484eb2355a1a6ef3c4c7fd6a56 Author: nathan &lt;nathan&gt; Date: Sat Feb 10 06:33:41 2007 +0000 land b1_5 onto HEAD </pre> </div></div> <p>Which I thought it use exp connection flag in a wrong way. So we probably just remove these code to save some trouble here. Since I do not think we plan to let 2.6 lustre client to connect some non-inodebit support server (1.4 ?) </p> <p>No, for single MDT case, there is no such race condition. Because the mount process will call md_getstatus() on the ROOT object firstly, it is NOT ldlm_enqueue RPC. Then when the mount return to user-space, it can guarantee that the connection between the client and the MDT0 has been established successfully.</p> <p>Ah, I see. Hmm, I still think remove those incompatible inodebit handling is a better choice, or can we do sth like this</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@testnode lustre-release]# git diff lustre/ldlm/ diff --git a/lustre/ldlm/ldlm_request.c b/lustre/ldlm/ldlm_request.c index 76f329d..2c0d05d 100644 --- a/lustre/ldlm/ldlm_request.c +++ b/lustre/ldlm/ldlm_request.c @@ -909,7 +909,7 @@ int ldlm_cli_enqueue(struct obd_export *exp, struct ptlrpc_request **reqp, OBD_CONNECT_IBITS)) lock-&gt;l_policy_data.l_inodebits.bits = MDS_INODELOCK_LOOKUP | - MDS_INODELOCK_UPDATE; + MDS_INODELOCK_UPDATE | MDS_INODELOCK_LAYOUT; else lock-&gt;l_policy_data = *policy; } </pre> </div></div> <p>Wangdi, I have verified the race conditions locally. But I am not sure whether removing the incompatible bits is a suitable solution or not. Because the key issue is that we are checking un-iniitialized export flags.</p> <blockquote> <p>Because the key issue is that we are checking un-iniitialized export flags.</p></blockquote> <p>Yes, but that is for checking if the server support inodebit lock. But since 2.6 client will be only connected to inodebit enable server, so why do we need this check? TBH, I really do not like add "connection wait logic" above ptlrpc, which might bring us troubles or even worse. </p> <p>Even though we drop the incompatible ibits, we still needs to prevent the user to use the non-initialised export, otherwise, there may be other potential bugs.</p> <p>I've tested the patch at <a href="http://review.whamcloud.com/#/c/10958/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10958/</a> (version 2) and, after running sanity-scrub three times, I have not seen this assertion.</p> <p>The patch fixes the crash/error for the configuration I've been testing with.</p> <p>Thanks James!</p> <p>"Even though we drop the incompatible ibits, we still needs to prevent the user to use the non-initialised export, otherwise, there may be other potential bugs."</p> <p>That is something should never happen, IMHO.</p> <p>Here is the patch to drop the redundant ibits lock interoperability check:</p> <p><a href="http://review.whamcloud.com/11004" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11004</a></p> <p>The patch has been landed to master.</p> <p>While verifying patch <a href="http://review.whamcloud.com/12606" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12606</a> on Lustre b2_5 branch, the same failure occurred:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 19407:0:(ldlm_lock.c:669:ldlm_lock2desc()) ASSERTION( lock-&gt;l_policy_data.l_inodebits.bits == (MDS_INODELOCK_LOOKUP | MDS_INODELOCK_UPDATE | MDS_INODELOCK_LAYOUT) ) failed: Inappropriate inode lock bits during conversion 3 LustreError: 19407:0:(ldlm_lock.c:669:ldlm_lock2desc()) LBUG Pid: 19407, comm: stat Call Trace: [&lt;ffffffffa03d0895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa03d0e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa067a239&gt;] ldlm_lock2desc+0x179/0x180 [ptlrpc] [&lt;ffffffffa068cb90&gt;] ldlm_cli_enqueue+0x1f0/0x790 [ptlrpc] [&lt;ffffffffa06b29ea&gt;] ? ptlrpc_request_set_replen+0x3a/0x60 [ptlrpc] [&lt;ffffffffa06919d0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [&lt;ffffffffa0a825c0&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] [&lt;ffffffffa08ded8e&gt;] mdc_enqueue+0x2be/0x1a10 [mdc] [&lt;ffffffffa01bc294&gt;] ? fld_client_rpc+0x864/0xed0 [fld] [&lt;ffffffffa08e06dd&gt;] mdc_intent_lock+0x1fd/0x64a [mdc] [&lt;ffffffffa0a825c0&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] [&lt;ffffffffa06919d0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [&lt;ffffffffa00e58b8&gt;] ? lprocfs_counter_add+0x1a8/0x1d6 [lvfs] [&lt;ffffffffa08a846e&gt;] lmv_intent_remote+0x47e/0xa80 [lmv] [&lt;ffffffffa0a825c0&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] [&lt;ffffffffa08a9137&gt;] lmv_intent_lookup+0x6c7/0x700 [lmv] [&lt;ffffffffa0a825c0&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] [&lt;ffffffffa08a9d6a&gt;] lmv_intent_lock+0x32a/0x380 [lmv] [&lt;ffffffffa0a825c0&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] [&lt;ffffffffa0a81d0e&gt;] ? ll_i2gids+0x2e/0xd0 [lustre] [&lt;ffffffffa0a68c0d&gt;] ? ll_prep_md_op_data+0x10d/0x3b0 [lustre] [&lt;ffffffffa0a8491f&gt;] ll_lookup_it+0x33f/0xb00 [lustre] [&lt;ffffffffa0a825c0&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] [&lt;ffffffffa0a50c51&gt;] ? __ll_inode_revalidate_it+0x1e1/0xc30 [lustre] [&lt;ffffffffa0a8535f&gt;] ll_lookup_nd+0x27f/0x3f0 [lustre] [&lt;ffffffff811a42fe&gt;] ? d_alloc+0x13e/0x1b0 [&lt;ffffffff81198a35&gt;] do_lookup+0x1a5/0x230 [&lt;ffffffff81199100&gt;] __link_path_walk+0x200/0x1000 [&lt;ffffffff8114a3d7&gt;] ? handle_pte_fault+0xf7/0xb00 [&lt;ffffffff8119a1ba&gt;] path_walk+0x6a/0xe0 [&lt;ffffffff8119a3cb&gt;] filename_lookup+0x6b/0xc0 [&lt;ffffffff8119b4f7&gt;] user_path_at+0x57/0xa0 [&lt;ffffffff8104a98c&gt;] ? __do_page_fault+0x1ec/0x480 [&lt;ffffffff8118e990&gt;] vfs_fstatat+0x50/0xa0 [&lt;ffffffff811515a5&gt;] ? do_mmap_pgoff+0x335/0x380 [&lt;ffffffff8118ea4e&gt;] vfs_lstat+0x1e/0x20 [&lt;ffffffff8118ea74&gt;] sys_newlstat+0x24/0x50 [&lt;ffffffff810e1e07&gt;] ? audit_syscall_entry+0x1d7/0x200 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b Kernel panic - not syncing: LBUG </pre> </div></div> <p>Maloo report: <a href="https://testing.hpdd.intel.com/test_sets/f4332d04-7af0-11e4-956d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f4332d04-7af0-11e4-956d-5254006e85c2</a></p> <p>Hi Nasf, could you please take a look whether this is a regression introduced by the patch <a href="http://review.whamcloud.com/12606" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12606</a> or not? If not, then this is an issue on Lustre b2_5 branch and I'll back-port your patch to fix it. Thank you!</p> <p>Yujian,</p> <p>I do not think it is <a href="http://review.whamcloud.com/#/c/12606/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12606/</a> caused the trouble. You need to back-port patch(es) to fix it.</p> <p>Thank you, Nasf. Will do.</p> <p>One more instance on Lustre b2_5 branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/c800bb44-5e7f-11e4-9843-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c800bb44-5e7f-11e4-9843-5254006e85c2</a></p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12960" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12960</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4971" title="sanity-scrub test_2: ldlm_lock2desc()) ASSERTION( lock-&gt;l_policy_data.l_inodebits.bits == (MDS_INODELOCK_LOOKUP | MDS_INODELOCK_UPDATE | MDS_INODELOCK_LAYOUT)failed" class="issue-link" data-issue-key="LU-4971"><del>LU-4971</del></a> ldlm: drop redundant ibits lock interoperability check<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 2b44f94c2bfbefd9b0f85a52218979b851d7af58</p> Duplicate LU-5273 LU-5272 Related LU-7975 Development Rank 1|hzwla7: Rank (Obsolete) 13763 Severity