https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4483 Lustre <p>Lustre: DEBUG MARKER: == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc == 13:22:32 (1389691352)<br/> proc_file_read: Apparent buffer overflow!<br/> proc_file_read: Apparent buffer overflow!<br/> proc_file_read: Apparent buffer overflow!<br/> proc_file_read: Apparent buffer overflow!<br/> proc_file_read: Apparent buffer overflow!<br/> proc_file_read: Apparent buffer overflow!<br/> proc_file_read: Apparent buffer overflow!<br/> proc_file_read: Apparent buffer overflow!<br/> proc_file_read: Apparent buffer overflow!<br/> general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP <br/> last sysfs file: /sys/devices/system/cpu/online<br/> CPU 0 <br/> Modules linked in: ext2 lustre ofd osp lod ost mdt mdd mgs nodemap osd_ldiskfs ldiskfs exportfs lquota lfsck jbd obdecho mgc lov osc mdc lmv fid fld ptlrpc obdclass ksocklnd lnet sha512_generic sha256_generic crc32c_intel libcfs nfs lockd auth_rpcgss nfs_acl sunrpc cachefiles fscache(T) ib_ipoib ib_cm ipv6 ib_uverbs ib_umad mlx4_ib ib_sa ib_mad ib_core mlx4_en mlx4_core dm_mirror dm_region_hash dm_log dm_mod ppdev parport_pc parport vmware_balloon microcode vmxnet3 sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic ata_piix <span class="error">&#91;last unloaded: scsi_wait_scan&#93;</span></p> <p>Pid: 35, comm: events/0 Tainted: G --------------- T 2.6.32-358.18.1-lustre #0 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffff812a3650&gt;&#93;</span> <span class="error">&#91;&lt;ffffffff812a3650&gt;&#93;</span> list_del+0x10/0xa0<br/> RSP: 0018:ffff88013ddbdc80 EFLAGS: 00010082<br/> RAX: 302f302f302f302f RBX: ffff880011c86040 RCX: ffff880138903c50<br/> RDX: 0020000000000080 RSI: ffff880091faf000 RDI: ffff880011c86040<br/> RBP: ffff88013ddbdc90 R08: ffff880138903c50 R09: 0000000000000001<br/> R10: 0000000000000000 R11: ffff880090218820 R12: ffff880011c86040<br/> R13: ffff8801399817c0 R14: 0000000000000002 R15: 000000000000101a<br/> FS: 0000000000000000(0000) GS:ffff88002be00000(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b<br/> CR2: 00000000022715a8 CR3: 000000008514e000 CR4: 00000000001407f0<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Process events/0 (pid: 35, threadinfo ffff88013ddbc000, task ffff88013ddba1c0)<br/> Stack:<br/> ffff8800487fb040 ffff8801385726c0 ffff88013ddbdcf0 ffffffff8117aab6<br/> &lt;d&gt; ffff880138903ca8 0000000300000000 ffff880011c86850 ffff880138903c50<br/> &lt;d&gt; ffff88013ddbdcf0 ffff880139981760 ffff8801385726c0 0000000000000003<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff8117aab6&gt;&#93;</span> free_block+0xc6/0x230<br/> <span class="error">&#91;&lt;ffffffff8117ae51&gt;&#93;</span> drain_array+0xc1/0xf0<br/> <span class="error">&#91;&lt;ffffffff8117eb3f&gt;&#93;</span> ? cache_reap+0x2f/0x2e0<br/> <span class="error">&#91;&lt;ffffffff8117ebbe&gt;&#93;</span> cache_reap+0xae/0x2e0<br/> <span class="error">&#91;&lt;ffffffff810931bb&gt;&#93;</span> ? worker_thread+0x1cb/0x3d0<br/> <span class="error">&#91;&lt;ffffffff8117eb10&gt;&#93;</span> ? cache_reap+0x0/0x2e0<br/> <span class="error">&#91;&lt;ffffffff8109320c&gt;&#93;</span> worker_thread+0x21c/0x3d0<br/> <span class="error">&#91;&lt;ffffffff810931bb&gt;&#93;</span> ? worker_thread+0x1cb/0x3d0<br/> <span class="error">&#91;&lt;ffffffff81099a00&gt;&#93;</span> ? autoremove_wake_function+0x0/0x40</p> OSTCOUNT=40 sh sanity.sh LU-4483

sanity test 133f: Panic

Bug Critical Resolved Fixed WC Triage Alexey Lyashkov Tue, 14 Jan 2014 09:26:52 +0000 Wed, 23 Dec 2015 19:02:15 +0000 Tue, 23 Dec 2014 09:42:15 +0000 Lustre 2.6.0 Lustre 2.6.0 0 5 <p>Somebody is attempting to read data using too little memory using a non seq_file read routine. If this started to just happen it might be NRS tbf or nodemap. Since it crashed on a list_del I only see NRS tbf working with list. Can you post a link to the maloo test that failed so I can see which proc entry is the source of the problem.</p> <p>James,</p> <p>it's may local sanity.sh run. console output:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cat: /proc/fs/lustre/osc/lustre-OST0005-osc-ffff8800aff251b8/ping: Input/output error cat: /proc/fs/lustre/osc/lustre-OST0004-osc-ffff8800aff251b8/ping: Input/output error cat: /proc/fs/lustre/osc/lustre-OST0003-osc-ffff8800aff251b8/ping: Input/output error cat: /proc/fs/lustre/osc/lustre-OST0002-osc-ffff8800aff251b8/ping: Input/output error cat: /proc/fs/lustre/osc/lustre-OST0001-osc-ffff8800aff251b8/ping: Input/output error cat: /proc/fs/lustre/osc/lustre-OST0000-osc-ffff8800aff251b8/ping: Input/output error cat: /proc/fs/lustre/mdc/lustre-MDT0000-mdc-ffff8800aff251b8/ping: Input/output error cat: /proc/fs/lustre/fld/cli-lustre-clilmv-ffff8800aff251b8/cache_flush: Input/output error Timeout, server 192.168.69.5 not responding. berloga-mac:~ shadow$ </pre> </div></div> <p>from my point view - it's memory corruption issue - when someone overwrite an slab's internal list.</p> <p>Shadow, it would be great if you could include "git describe" output with your bug reports, so we can see what version of the code you are running.</p> <p>James, will the /proc file in question be moved over to seq_file with your patches? Do you know which patch covers this /proc file, and has it landed yet?</p> <p>The output Alexey is seeing for the the sanity test is normal. Both ping and cache_flush are write only so reading will give a error. In the past you could perform a read on a write proc entry. Now the LBUG is not normal. Can you send me your dmesg and lustre debug logs. I like to see which proc entry is exactly crashing. I don't know if the above proc entries are the real problem. </p> <p>Today I looked to see why a test for readable proc file was selecting write only enrty. It appears to be find -readable option still selects write only files. When I used -perm 444 I had no problems. I can't reproduce the Oops you saw so far. This is testing with a unpatched copy of the lastest master from this morning.</p> <p>James,</p> <p>it's not a LBUG (aka lustre bug) it's just found a list corruption. did you run own tests with kernel with debug options enabled? I have all debug options enabled.</p> <p>replicated again</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@rhel6-64 tests]# OSTCOUNT=40 ONLY=133 REFORMAT=yes sh sanity.sh Logging to shared log directory: /tmp/test_logs/1389900285 == sanity test 133e: Verifying OST {read,write}_bytes nid stats =================== 23:26:08 (1389900368) 42+0 records in 42+0 records out 1376256 bytes (1.4 MB) copied, 0.0373407 s, 36.9 MB/s sanity test_133e: @@@@@@ FAIL: Bad write_bytes sum, expected 1376256, got 0 Trace dump: = /Users/shadow/work/lustre/work/WC-review/CLSTR-2003/lustre/tests/test-framework.sh:4430:error_noexit() = /Users/shadow/work/lustre/work/WC-review/CLSTR-2003/lustre/tests/test-framework.sh:4461:error() = sanity.sh:8714:test_133e() = /Users/shadow/work/lustre/work/WC-review/CLSTR-2003/lustre/tests/test-framework.sh:4701:run_one() = /Users/shadow/work/lustre/work/WC-review/CLSTR-2003/lustre/tests/test-framework.sh:4736:run_one_logged() = /Users/shadow/work/lustre/work/WC-review/CLSTR-2003/lustre/tests/test-framework.sh:4562:run_test() = sanity.sh:8720:main() Dumping lctl log to /tmp/test_logs/1389900285/sanity.test_133e.*.1389900369.log Dumping logs only on local client. FAIL 133e (25s) == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc == 23:26:33 (1389900393) Timeout, server 192.168.69.5 not responding. </pre> </div></div> <p>console output</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: Skipped 3513 previous similar messages LustreError: 685:0:(ldlm_resource.c:1154:ldlm_resource_get()) lustre-OST001e: lvbo_init failed for resource 0x43:0x0: rc = -2 LustreError: 685:0:(ldlm_resource.c:1154:ldlm_resource_get()) Skipped 6505 previous similar messages LustreError: 11-0: lustre-OST0018-osc-ffff88008357c488: Communicating with 0@lo, operation ldlm_enqueue failed with -12. LustreError: Skipped 6484 previous similar messages Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =================== 23:26:08 (1389900368) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: sanity test_133e: @@@@@@ FAIL: Bad write_bytes sum, expected 1376256, got 0 Lustre: DEBUG MARKER: == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc == 23:26:33 (1389900393) proc_file_read: Apparent buffer overflow! proc_file_read: Apparent buffer overflow! proc_file_read: Apparent buffer overflow! general protection fault: 0000 [#1] SMP last sysfs file: /sys/devices/system/cpu/online CPU 3 Modules linked in: lustre ofd osp lod ost mdt mdd mgs nodemap osd_ldiskfs ldiskfs lquota lfsck obdecho mgc lov osc mdc lmv fid fld ptlrpc obdclass ksocklnd lnet libcfs exportfs jb d sha512_generic sha256_generic crc32c_intel nfs lockd auth_rpcgss nfs_acl sunrpc cachefiles fscache(T) ib_ipoib ib_cm ipv6 ib_uverbs ib_umad mlx4_ib ib_sa ib_mad ib_core mlx4_en mlx4_core dm_mirror dm_region_hash dm_log dm_mod ppdev parport_pc parport microcode vmware_balloon vmxnet3 sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic ata_piix [last unloaded: libcfs] Pid: 1727, comm: vmmemctl Tainted: G --------------- T 2.6.32-358.18.1-lustre #0 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform RIP: 0010:[&lt;ffffffff811d1e0c&gt;] [&lt;ffffffff811d1e0c&gt;] nr_blockdev_pages+0x3c/0x80 </pre> </div></div> <p>Now I see the problem with test 133e with the nid stats for the obdfilter layer. I was only looking at 133f test. I have some but not all the debug flags enabled. Which one do you have so I can reproduce this?</p> <p>I have a few kernel modification to print sentry name have a overflow.<br/> Lustre: DEBUG MARKER: == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc == 00:17:07 (1389903427)<br/> proc_file_read: 'hash'/ffffffffa055fc60 Apparent buffer overflow!<br/> proc_file_read: 'hash'/ffffffffa055fc60 Apparent buffer overflow!<br/> proc_file_read: 'hash'/ffffffffa055fc60 Apparent buffer overflow!<br/> -----------<del>[ cut here ]</del>-----------<br/> crash&gt; l *(0xffffffffa055fc60)<br/> 0xffffffffa055fc60 is in lprocfs_exp_rd_hash (/Users/shadow/work/lustre/work/WC-review/CLSTR-2003/lustre/obdclass/lprocfs_status.c:2568).<br/> 2563 return 0;<br/> 2564 }<br/> 2565 <br/> 2566 int lprocfs_exp_rd_hash(char *page, char **start, off_t off, int count,<br/> 2567 int *eof, void *data)</p> <p>i hope it help you.</p> <p>as i see lprocfs_exp_rd_hash put a data to a fixed buffer size (page size) without a size control, it's produce a overwriting a random area in memory so random memory corruption.</p> <p>Absolute correct. The problem is that __proc_file_read allocates only a single page to use. We have to test if we overflow the page in cfs_hash_debug_str. Once we overflow we have to return which also requires use remembering were we left off. The fix is quite ugly but we can assume the source of the overflow is going to be the looping through the hash bucket. We have to pass in a offset to this function and then if we have a offset jump to the hash bucket loop and start going through the loop at the right place. The other option is to wait for the seq_file patches for server side to land. This would resolve this since seq_file buffers are handled dynamically. That is they are resized on the fly when a overflow happens.</p> <p>This should be fixed by the landing of <a href="http://review.whamcloud.com/#/c/8049" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8049</a> from <a href="https://jira.whamcloud.com/browse/LU-3319" title="Adapt to 3.10 upstream kernel proc_dir_entry change" class="issue-link" data-issue-key="LU-3319"><del>LU-3319</del></a>. Peter can you link this tick to <a href="https://jira.whamcloud.com/browse/LU-3319" title="Adapt to 3.10 upstream kernel proc_dir_entry change" class="issue-link" data-issue-key="LU-3319"><del>LU-3319</del></a>. We will need a fix for b2_5/b2_4 as well.</p> <p>Fixed as part of <a href="https://jira.whamcloud.com/browse/LU-3319" title="Adapt to 3.10 upstream kernel proc_dir_entry change" class="issue-link" data-issue-key="LU-3319"><del>LU-3319</del></a>.</p> Related LU-3319 LU-5764 Development Rank 1|hzwcs7: Rank (Obsolete) 12274 Severity