https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6757 Lustre <p>This issue was created by maloo for sarah_lw &lt;wei3.liu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/74ee0e8e-13d8-11e5-b4b0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/74ee0e8e-13d8-11e5-b4b0-5254006e85c2</a>.</p> <p>The sub-test test_61 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trusted.big different after saving </pre> </div></div> <p>MDS dmesg</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10669.620927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save large xattr trusted.big on \/mnt\/lustre\/f61.conf-sanity [10669.768885] Lustre: DEBUG MARKER: save large xattr trusted.big on /mnt/lustre/f61.conf-sanity [10669.796354] ------------[ cut here ]------------ [10669.801490] WARNING: at /var/lib/jenkins/workspace/lustre-master/arch/x86_64/build_type/server/distro/el7/ib_stack/inkernel/BUILD/BUILD/lustre-2.7.55/ldiskfs/ldiskfs.h:2631 ldiskfs_xattr_set_entry+0x7c4/0x7f0 [ldiskfs]() [10669.809597] Modules linked in: osp(OF) mdd(OF) lod(OF) mdt(OF) lfsck(OF) mgs(OF) mgc(OF) osd_ldiskfs(OF) lquota(OF) fid(OF) fld(OF) ksocklnd(OF) ptlrpc(OF) obdclass(OF) lnet(OF) libcfs(OF) ldiskfs(OF) sha512_generic dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd fscache xprtrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ppdev parport_pc virtio_balloon serio_raw pcspkr parport i2c_piix4 ib_core ib_addr ext4 mbcache jbd2 ata_generic pata_acpi virtio_blk cirrus syscopyarea sysfillrect ata_piix sysimgblt drm_kms_helper virtio_pci virtio_ring virtio ttm 8139too drm 8139cp i2c_core mii libata floppy [last unloaded: libcfs] [10669.827111] CPU: 0 PID: 7739 Comm: mdt00_002 Tainted: GF O-------------- 3.10.0-229.4.2.el7_lustre.x86_64 #1 [10669.829588] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [10669.831727] 0000000000000000 00000000510e8244 ffff88006cd67710 ffffffff816050da [10669.834182] ffff88006cd67748 ffffffff8106e34b ffff8800655e9e88 ffff88007ac63cf0 [10669.836691] ffff88006cd67898 0000000000010000 ffff88006571c068 ffff88006cd67758 [10669.839122] Call Trace: [10669.841102] [&lt;ffffffff816050da&gt;] dump_stack+0x19/0x1b [10669.843216] [&lt;ffffffff8106e34b&gt;] warn_slowpath_common+0x6b/0xb0 [10669.845424] [&lt;ffffffff8106e49a&gt;] warn_slowpath_null+0x1a/0x20 [10669.847584] [&lt;ffffffffa05db334&gt;] ldiskfs_xattr_set_entry+0x7c4/0x7f0 [ldiskfs] [10669.849899] [&lt;ffffffffa05dbdd5&gt;] ldiskfs_xattr_ibody_set+0x35/0x90 [ldiskfs] [10669.852135] [&lt;ffffffffa05dc1c4&gt;] ldiskfs_xattr_set_handle+0x1a4/0x500 [ldiskfs] [10669.854442] [&lt;ffffffffa05dc666&gt;] ldiskfs_xattr_set+0x146/0x1c0 [ldiskfs] [10669.856627] [&lt;ffffffffa0607f0d&gt;] ldiskfs_xattr_trusted_set+0x2d/0x30 [ldiskfs] [10669.858871] [&lt;ffffffff811eb4b2&gt;] generic_setxattr+0x62/0x80 [10669.860919] [&lt;ffffffffa0ba62cc&gt;] osd_xattr_set+0x3ec/0x700 [osd_ldiskfs] [10669.863083] [&lt;ffffffffa0e29187&gt;] lod_sub_object_xattr_set+0x227/0x460 [lod] [10669.865182] [&lt;ffffffffa0e150d9&gt;] lod_xattr_set_internal+0xb9/0x2d0 [lod] [10669.867329] [&lt;ffffffffa0e20fb6&gt;] lod_xattr_set+0x136/0x3f0 [lod] [10669.869422] [&lt;ffffffffa0e83673&gt;] mdd_xattr_set+0x483/0x720 [mdd] [10669.872412] [&lt;ffffffffa0d453c7&gt;] ? mdt_version_save+0x67/0x120 [mdt] [10669.875322] [&lt;ffffffffa0d51b57&gt;] mdt_reint_setxattr+0x967/0xf90 [mdt] [10669.878297] [&lt;ffffffffa0d4f1d0&gt;] mdt_reint_rec+0x80/0x210 [mdt] [10669.880537] [&lt;ffffffffa0d32e6c&gt;] mdt_reint_internal+0x58c/0x780 [mdt] [10669.882681] [&lt;ffffffffa0d3c167&gt;] mdt_reint+0x67/0x140 [mdt] [10669.884691] [&lt;ffffffffa0a047e5&gt;] tgt_request_handle+0x6d5/0x1060 [ptlrpc] [10669.886831] [&lt;ffffffffa09b472b&gt;] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] [10669.888956] [&lt;ffffffffa09b22a8&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [10669.891045] [&lt;ffffffffa0561967&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [10669.893039] [&lt;ffffffffa09b8918&gt;] ptlrpc_main+0xaf8/0x1ea0 [ptlrpc] [10669.895022] [&lt;ffffffff810ad8b6&gt;] ? __dequeue_entity+0x26/0x40 [10669.896880] [&lt;ffffffffa09b7e20&gt;] ? ptlrpc_register_service+0xf00/0xf00 [ptlrpc] [10669.898876] [&lt;ffffffff8109739f&gt;] kthread+0xcf/0xe0 [10669.900601] [&lt;ffffffff810972d0&gt;] ? kthread_create_on_node+0x140/0x140 [10669.902499] [&lt;ffffffff81614f7c&gt;] ret_from_fork+0x7c/0xb0 [10669.904251] [&lt;ffffffff810972d0&gt;] ? kthread_create_on_node+0x140/0x140 [10669.906128] ---[ end trace 28efa5c9df746bee ]--- [10669.911949] LustreError: 7739:0:(mdt_handler.c:891:mdt_getattr_internal()) lustre-MDT0000: getattr error for [0x200000401:0x1:0x0]: rc = -5 [10670.039079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_61: @@@@@@ FAIL: trusted.big different after saving [10670.184684] Lustre: DEBUG MARKER: conf-sanity test_61: @@@@@@ FAIL: trusted.big different after saving </pre> </div></div> server: lustre-master build # 3071 EL7 ldiskfs<br/> client: lustre-master build #3071 EL6 LU-6757

conf-sanity test_61: trusted.big different after saving

Bug Critical Resolved Fixed Jian Yu Maloo Tue, 23 Jun 2015 22:00:40 +0000 Thu, 25 Feb 2016 22:13:52 +0000 Wed, 26 Aug 2015 17:39:03 +0000 Lustre 2.8.0 Lustre 2.8.0 0 10 <p>Jian</p> <p>Could you please look into this issue?</p> <p>Thanks</p> <p>Peter</p> <p>On client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>getfattr: /mnt/lustre/f61.conf-sanity: Input/output error </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000002:00000001:0.0:1439363436.401810:0:5372:0:(mdc_locks.c:571:mdc_finish_enqueue()) Process entered 00000002:00100000:0.0:1439363436.401812:0:5372:0:(mdc_locks.c:625:mdc_finish_enqueue()) @@@ op: 8 disposition: b, status: -5 req@ffff8800796ed800 x1509281914421388/t0(0) o101-&gt;lustre-MDT0000-mdc-ffff880078fbf000@10.100.4.86@tcp:12/10 lens 584/536 e 0 to 0 dl 1439363443 ref 1 fl Complete:R/0/0 rc 301/301 &lt;~snip~&gt; 00000002:00000001:0.0:1439363436.401820:0:5372:0:(mdc_locks.c:955:mdc_finish_intent_lock()) Process entered 00000002:00000001:0.0:1439363436.401821:0:5372:0:(mdc_locks.c:979:mdc_finish_intent_lock()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) 00000002:00000001:0.0:1439363436.401822:0:5372:0:(mdc_locks.c:1182:mdc_intent_lock()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) 00800000:00000001:0.0:1439363436.401823:0:5372:0:(obd_class.h:1535:md_intent_lock()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) 00800000:00000001:0.0:1439363436.401824:0:5372:0:(lmv_intent.c:410:lmv_intent_lookup()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) 00800000:00000001:0.0:1439363436.401825:0:5372:0:(lmv_intent.c:530:lmv_intent_lock()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) 00000080:00000001:0.0:1439363436.401826:0:5372:0:(obd_class.h:1535:md_intent_lock()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) 00000080:00000001:0.0:1439363436.401827:0:5372:0:(namei.c:590:ll_lookup_it()) Process leaving via out (rc=18446744073709551611 : -5 : 0xfffffffffffffffb) </pre> </div></div> <p>On MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000001:0.0:1439363436.853113:0:26612:0:(lod_object.c:1355:lod_xattr_get()) Process entered 00000004:00000001:0.0:1439363436.853119:0:26612:0:(lod_object.c:1395:lod_xattr_get()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) 00000004:00000001:0.0:1439363436.853121:0:26612:0:(mdd_object.c:244:mdd_xattr_get()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) 00000004:00000001:0.0:1439363436.853122:0:26612:0:(mdt_handler.c:802:mdt_attr_get_complex()) Process leaving via out (rc=18446744073709551611 : -5 : 0xfffffffffffffffb) 00000004:00000002:0.0:1439363436.853123:0:26612:0:(mdt_handler.c:861:mdt_attr_get_complex()) after getattr rc = -5, ma_valid = 0x1 ma_lmm=ffff88006982a340 00000004:00000001:0.0:1439363436.853124:0:26612:0:(mdt_handler.c:862:mdt_attr_get_complex()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) 00000004:00020000:0.0:1439363436.853125:0:26612:0:(mdt_handler.c:951:mdt_getattr_internal()) lustre-MDT0000: getattr error for [0x200000400:0x1:0x0]: rc = -5 00000004:00000001:0.0:1439363436.856913:0:26612:0:(mdt_handler.c:952:mdt_getattr_internal()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) </pre> </div></div> <p>Still investigate.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># Get the maximum xattr size supported by the filesystem. max_xattr_size() { local size <span class="code-keyword">if</span> large_xattr_enabled; then # include/linux/limits.h: #define XATTR_SIZE_MAX 65536 size=65536 <span class="code-keyword">else</span> local mds_dev=$(mdsdevname ${SINGLEMDS<span class="code-comment">//mds/}) </span> local block_size=$(get_block_size $SINGLEMDS $mds_dev) # maximum xattr size = size of block - size of header - # size of 1 entry - 4 <span class="code-keyword">null</span> bytes size=$((block_size - 32 - 32 - 4)) fi echo $size } </pre> </div></div> <p>Testing showed that after large_xattr was enabled, the test would fail on RHEL 7.1 while the maximum xattr size exceeded $((block_size - 32 - 32 - 4)). So, large xattr did not work.</p> <p>I'm looking into ldiskfs_xattr_set_entry().</p> <p>The problematic source code is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># gdb "$(modinfo -n ldiskfs)" &lt;~snip~&gt; Reading symbols from /usr/lib/modules/3.10.0-229.4.2.el7_lustre.x86_64/extra/kernel/fs/lustre/ldiskfs.ko...Reading symbols from /usr/lib/debug/lib/modules/3.10.0-229.4.2.el7_lustre.x86_64/extra/kernel/fs/lustre/ldiskfs.ko.debug...done. done. (gdb) list *(ldiskfs_xattr_set_entry+0x7c4) 0x1e244 is in ldiskfs_xattr_set_entry (/usr/src/debug/lustre-2.7.0/ldiskfs/ldiskfs.h:2637). 2632 #endif 2633 2634 /* Update i_disksize. Requires i_mutex to avoid races with truncate */ 2635 static inline void ldiskfs_update_i_disksize(struct inode *inode, loff_t newsize) 2636 { 2637 WARN_ON_ONCE(S_ISREG(inode-&gt;i_mode) &amp;&amp; 2638 !mutex_is_locked(&amp;inode-&gt;i_mutex)); 2639 down_write(&amp;LDISKFS_I(inode)-&gt;i_data_sem); 2640 if (newsize &gt; LDISKFS_I(inode)-&gt;i_disksize) 2641 LDISKFS_I(inode)-&gt;i_disksize = newsize; </pre> </div></div> <p>The i_mutex lock needs to be acquired before calling ext4_update_i_disksize() in xattr.c.<br/> I'm creating the patch.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16012" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16012</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6757" title="conf-sanity test_61: trusted.big different after saving" class="issue-link" data-issue-key="LU-6757"><del>LU-6757</del></a> ldiskfs: acquire i_mutex lock for ext4_update_i_disksize<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2dd614101cc3e7c5d0b8c5cc6d88e1c07fa55141</p> <p>The root cause of lod_xattr_get() returning "-5" (-EIO) error is related to the following changes in ext4_xattr_check_names() for the RHEL 7.1 kernel version 3.10.0:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>author Darrick J. Wong &lt;darrick.wong@oracle.com&gt; 2014-09-16 18:34:59 (GMT) committer Theodore Ts'o &lt;tytso@mit.edu&gt; 2014-09-16 18:34:59 (GMT) commit a0626e75954078cfacddb00a4545dde821170bc5 (patch) tree 61aeb6747c5fc7016ac54cc704fb6d9b1833674d parent 684de5748660e16e185754697ac0afa9e18297f6 (diff) ext4: check EA value offset when loading When loading extended attributes, check each entry's value offset to make sure it doesn't collide with the entries. Without this check it is easy to crash the kernel by mounting a malicious FS containing a file with an EA wherein e_value_offs = 0 and e_value_size &gt; 0 and then deleting the EA, which corrupts the name list. (See the f_ea_value_crash test's FS image in e2fsprogs for an example.) Signed-off-by: Darrick J. Wong &lt;darrick.wong@oracle.com&gt; Signed-off-by: Theodore Ts'o &lt;tytso@mit.edu&gt; Cc: stable@vger.kernel.org </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c index da4df70..42823ab 100644 --- a/fs/ext4/xattr.c +++ b/fs/ext4/xattr.c @@ -190,14 +190,28 @@ ext4_listxattr(struct dentry *dentry, <span class="code-object">char</span> *buffer, size_t size) } <span class="code-keyword">static</span> <span class="code-object">int</span> -ext4_xattr_check_names(struct ext4_xattr_entry *entry, void *end) +ext4_xattr_check_names(struct ext4_xattr_entry *entry, void *end, + void *value_start) { - <span class="code-keyword">while</span> (!IS_LAST_ENTRY(entry)) { - struct ext4_xattr_entry *next = EXT4_XATTR_NEXT(entry); + struct ext4_xattr_entry *e = entry; + + <span class="code-keyword">while</span> (!IS_LAST_ENTRY(e)) { + struct ext4_xattr_entry *next = EXT4_XATTR_NEXT(e); <span class="code-keyword">if</span> ((void *)next &gt;= end) <span class="code-keyword">return</span> -EIO; - entry = next; + e = next; } + + <span class="code-keyword">while</span> (!IS_LAST_ENTRY(entry)) { + <span class="code-keyword">if</span> (entry-&gt;e_value_size != 0 &amp;&amp; + (value_start + le16_to_cpu(entry-&gt;e_value_offs) &lt; + (void *)e + sizeof(__u32) || + value_start + le16_to_cpu(entry-&gt;e_value_offs) + + le32_to_cpu(entry-&gt;e_value_size) &gt; end)) + <span class="code-keyword">return</span> -EIO;{color} ------------&gt; here + entry = EXT4_XATTR_NEXT(entry); + } + <span class="code-keyword">return</span> 0; } </pre> </div></div> <p>While getting xattr, in ext4_xattr_ibody_get():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> end = (void *)raw_inode + EXT4_SB(inode-&gt;i_sb)-&gt;s_inode_size; error = ext4_xattr_check_names(entry, end, entry); </pre> </div></div> <p>For large EA, <tt>value_start + le16_to_cpu(entry-&gt;e_value_offs) + le32_to_cpu(entry-&gt;e_value_size) &gt; end</tt> will fail, which causes ext4_xattr_check_names() return "-EIO".</p> <p>I fixed the above failure by checking <tt>e_value_inum</tt> to see whether the large EA value is saved in an external EA inode or not. Testing showed that conf-sanity test 61 passed on RHEL 7.1 distro.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16012/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16012/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6757" title="conf-sanity test_61: trusted.big different after saving" class="issue-link" data-issue-key="LU-6757"><del>LU-6757</del></a> ldiskfs: acquire i_mutex lock for ext4_update_i_disksize<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7417966b6a2ebf56a49dee327f1561f63bf9224b</p> <p>Landed for 2.8.</p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18449" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18449</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6757" title="conf-sanity test_61: trusted.big different after saving" class="issue-link" data-issue-key="LU-6757"><del>LU-6757</del></a> ldiskfs: large EA support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 813580ebe4a08dddf34eaa057c16832d4be46682</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18449/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18449/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6757" title="conf-sanity test_61: trusted.big different after saving" class="issue-link" data-issue-key="LU-6757"><del>LU-6757</del></a> ldiskfs: large EA support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 721e5ca9b7c1e00dae1d0adc49bcbe403d913a10</p> Related Development Rank 1|hzxgfr: Rank (Obsolete) 9223372036854775807 Severity