[LU-10822] sanity test 27b fails with “* buffer overflow detected *: /usr/bin/lfs terminated”

Related — Fri, 30 Mar 2018 23:47:43 +0000

sanity test_27b fails with error

/usr/lib64/lustre/tests/sanity.sh: line 1434: [: -eq: unary operator expected
 sanity test_27b: @@@@@@ FAIL: two-stripe file doesn't have two stripes

Looking at the client test_log, we see what the real issue is

== sanity test 27b: create and write to two stripe file ============================================== 09:49:29 (1521020969)

*** buffer overflow detected ***: /usr/bin/lfs terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f656102c7e5]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f65610ce15c]
/lib/x86_64-linux-gnu/libc.so.6(+0x117160)[0x7f65610cc160]
/lib/x86_64-linux-gnu/libc.so.6(+0x1168dd)[0x7f65610cb8dd]
/lib/x86_64-linux-gnu/libc.so.6(__snprintf_chk+0x78)[0x7f65610cb7f8]
/usr/lib/liblustreapi.so.1(+0x10065)[0x7f65617ca065]
/usr/lib/liblustreapi.so.1(+0x78ec)[0x7f65617c18ec]
/usr/lib/liblustreapi.so.1(+0x9f64)[0x7f65617c3f64]
/usr/bin/lfs[0x413e87]
/usr/lib/liblustreapi.so.1(Parser_execarg+0x51)[0x7f65617d5381]
/usr/bin/lfs[0x4042cc]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f6560fd5830]
/usr/bin/lfs[0x404349]
======= Memory map: ========
00400000-00421000 r-xp 00000000 fd:01 1451341                            /usr/bin/lfs
00621000-00622000 r--p 00021000 fd:01 1451341                            /usr/bin/lfs
00622000-00623000 rw-p 00022000 fd:01 1451341                            /usr/bin/lfs
009b6000-009d7000 rw-p 00000000 00:00 0                                  [heap]
7f6560408000-7f656041e000 r-xp 00000000 fd:01 5767689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f656041e000-7f656061d000 ---p 00016000 fd:01 5767689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f656061d000-7f656061e000 rw-p 00015000 fd:01 5767689                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f656061e000-7f6560643000 r-xp 00000000 fd:01 5767774                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f6560643000-7f6560842000 ---p 00025000 fd:01 5767774                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f6560842000-7f6560846000 r--p 00024000 fd:01 5767774                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f6560846000-7f6560847000 rw-p 00028000 fd:01 5767774                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f6560847000-7f656094f000 r-xp 00000000 fd:01 5767444                    /lib/x86_64-linux-gnu/libm-2.23.so
7f656094f000-7f6560b4e000 ---p 00108000 fd:01 5767444                    /lib/x86_64-linux-gnu/libm-2.23.so
7f6560b4e000-7f6560b4f000 r--p 00107000 fd:01 5767444                    /lib/x86_64-linux-gnu/libm-2.23.so
7f6560b4f000-7f6560b50000 rw-p 00108000 fd:01 5767444                    /lib/x86_64-linux-gnu/libm-2.23.so
7f6560b50000-7f6560b6d000 r-xp 00000000 fd:01 1453147                    /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.4
7f6560b6d000-7f6560d6d000 ---p 0001d000 fd:01 1453147                    /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.4
7f6560d6d000-7f6560d6e000 r--p 0001d000 fd:01 1453147                    /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.4
7f6560d6e000-7f6560d6f000 rw-p 0001e000 fd:01 1453147                    /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.4
7f6560d6f000-7f6560dac000 r-xp 00000000 fd:01 5767753                    /lib/x86_64-linux-gnu/libreadline.so.6.3
7f6560dac000-7f6560fac000 ---p 0003d000 fd:01 5767753                    /lib/x86_64-linux-gnu/libreadline.so.6.3
7f6560fac000-7f6560fae000 r--p 0003d000 fd:01 5767753                    /lib/x86_64-linux-gnu/libreadline.so.6.3
7f6560fae000-7f6560fb4000 rw-p 0003f000 fd:01 5767753                    /lib/x86_64-linux-gnu/libreadline.so.6.3
7f6560fb4000-7f6560fb5000 rw-p 00000000 00:00 0
7f6560fb5000-7f6561175000 r-xp 00000000 fd:01 5767448                    /lib/x86_64-linux-gnu/libc-2.23.so
7f6561175000-7f6561375000 ---p 001c0000 fd:01 5767448                    /lib/x86_64-linux-gnu/libc-2.23.so
7f6561375000-7f6561379000 r--p 001c0000 fd:01 5767448                    /lib/x86_64-linux-gnu/libc-2.23.so
7f6561379000-7f656137b000 rw-p 001c4000 fd:01 5767448                    /lib/x86_64-linux-gnu/libc-2.23.so
7f656137b000-7f656137f000 rw-p 00000000 00:00 0
7f656137f000-7f6561397000 r-xp 00000000 fd:01 1451349                    /usr/lib/liblnetconfig.so.4.0.0
7f6561397000-7f6561596000 ---p 00018000 fd:01 1451349                    /usr/lib/liblnetconfig.so.4.0.0
7f6561596000-7f6561597000 r--p 00017000 fd:01 1451349                    /usr/lib/liblnetconfig.so.4.0.0
7f6561597000-7f6561598000 rw-p 00018000 fd:01 1451349                    /usr/lib/liblnetconfig.so.4.0.0
7f6561598000-7f65615a0000 rw-p 00000000 00:00 0
7f65615a0000-7f65615b9000 r-xp 00000000 fd:01 5767786                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f65615b9000-7f65617b8000 ---p 00019000 fd:01 5767786                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f65617b8000-7f65617b9000 r--p 00018000 fd:01 5767786                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f65617b9000-7f65617ba000 rw-p 00019000 fd:01 5767786                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7f65617ba000-7f65617de000 r-xp 00000000 fd:01 1451350                    /usr/lib/liblustreapi.so.1.0.0
7f65617de000-7f65619dd000 ---p 00024000 fd:01 1451350                    /usr/lib/liblustreapi.so.1.0.0
7f65619dd000-7f65619de000 r--p 00023000 fd:01 1451350                    /usr/lib/liblustreapi.so.1.0.0
7f65619de000-7f65619df000 rw-p 00024000 fd:01 1451350                    /usr/lib/liblustreapi.so.1.0.0
7f65619df000-7f65619e8000 rw-p 00000000 00:00 0
7f65619e8000-7f6561a0e000 r-xp 00000000 fd:01 5767446                    /lib/x86_64-linux-gnu/ld-2.23.so
7f6561bfd000-7f6561c03000 rw-p 00000000 00:00 0
7f6561c0c000-7f6561c0d000 rw-p 00000000 00:00 0
7f6561c0d000-7f6561c0e000 r--p 00025000 fd:01 5767446                    /lib/x86_64-linux-gnu/ld-2.23.so
7f6561c0e000-7f6561c0f000 rw-p 00026000 fd:01 5767446                    /lib/x86_64-linux-gnu/ld-2.23.so
7f6561c0f000-7f6561c10000 rw-p 00000000 00:00 0
7ffedfb87000-7ffedfba9000 rw-p 00000000 00:00 0                          [stack]
7ffedfbef000-7ffedfbf2000 r--p 00000000 00:00 0                          [vvar]
7ffedfbf2000-7ffedfbf4000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

/usr/lib64/lustre/tests/sanity.sh: line 1429:  5280 Aborted                 (core dumped) $LFS getstripe -c $DIR/$tdir/$tfile

It looks like this issue started on 2018-02-27 16:23:17 UTC. So far, we’ve only seen this while testing Ubuntu 16.04 clients.

Here are logs for a few failed test suites

https://testing.hpdd.intel.com/test_sets/5ae8fc72-287c-11e8-9e0e-52540065bddc

https://testing.hpdd.intel.com/test_sets/9f2131ee-2894-11e8-b3c6-52540065bddc

https://testing.hpdd.intel.com/test_sets/f530fcd0-1cb5-11e8-a7cd-52540065bddc

https://testing.hpdd.intel.com/test_sets/4d231688-2195-11e8-b046-52540065bddc

In these test sessions, test 27f, 27i, 27z, 27C, 27F, 56a, 56w, 56wb, 56wc, 56x, 56xa, 56xb, 65i, 102b, 102c, 102d, 102f, 102j, 130a/b/c/d/e, 229, 270a/c/d, 311 fail with a buffer overflow detected in lfs.

In these test sessions, we also see the follow tests fail with this error:

sanity-lfsck 18d, 18e, 18g, 20a, 20b

sanityn test 51c

sanity-hsm test 11a

sanity-flr test 0a, 0b, 0c, 0d, 0e, 0f, 0g, 0h, 1, 2, 4, 5, 32, 32, 36, 37, 38, 39, 40, 41, 42, 43, 45, 200

sanity-dom test 42e, 251, 51c, 51d

Whamcloud Community JIRA

[LU-10822] sanity test 27b fails with “*** buffer overflow detected ***: /usr/bin/lfs terminated”

[LU-10822] sanity test 27b fails with “* buffer overflow detected *: /usr/bin/lfs terminated”