Details
-
Bug
-
Resolution: Fixed
-
Major
-
Lustre 2.11.0
-
3
-
9223372036854775807
Description
sanity test_27b fails with error
/usr/lib64/lustre/tests/sanity.sh: line 1434: [: -eq: unary operator expected sanity test_27b: @@@@@@ FAIL: two-stripe file doesn't have two stripes
Looking at the client test_log, we see what the real issue is
== sanity test 27b: create and write to two stripe file ============================================== 09:49:29 (1521020969) *** buffer overflow detected ***: /usr/bin/lfs terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f656102c7e5] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f65610ce15c] /lib/x86_64-linux-gnu/libc.so.6(+0x117160)[0x7f65610cc160] /lib/x86_64-linux-gnu/libc.so.6(+0x1168dd)[0x7f65610cb8dd] /lib/x86_64-linux-gnu/libc.so.6(__snprintf_chk+0x78)[0x7f65610cb7f8] /usr/lib/liblustreapi.so.1(+0x10065)[0x7f65617ca065] /usr/lib/liblustreapi.so.1(+0x78ec)[0x7f65617c18ec] /usr/lib/liblustreapi.so.1(+0x9f64)[0x7f65617c3f64] /usr/bin/lfs[0x413e87] /usr/lib/liblustreapi.so.1(Parser_execarg+0x51)[0x7f65617d5381] /usr/bin/lfs[0x4042cc] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f6560fd5830] /usr/bin/lfs[0x404349] ======= Memory map: ======== 00400000-00421000 r-xp 00000000 fd:01 1451341 /usr/bin/lfs 00621000-00622000 r--p 00021000 fd:01 1451341 /usr/bin/lfs 00622000-00623000 rw-p 00022000 fd:01 1451341 /usr/bin/lfs 009b6000-009d7000 rw-p 00000000 00:00 0 [heap] 7f6560408000-7f656041e000 r-xp 00000000 fd:01 5767689 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f656041e000-7f656061d000 ---p 00016000 fd:01 5767689 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f656061d000-7f656061e000 rw-p 00015000 fd:01 5767689 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f656061e000-7f6560643000 r-xp 00000000 fd:01 5767774 /lib/x86_64-linux-gnu/libtinfo.so.5.9 7f6560643000-7f6560842000 ---p 00025000 fd:01 5767774 /lib/x86_64-linux-gnu/libtinfo.so.5.9 7f6560842000-7f6560846000 r--p 00024000 fd:01 5767774 /lib/x86_64-linux-gnu/libtinfo.so.5.9 7f6560846000-7f6560847000 rw-p 00028000 fd:01 5767774 /lib/x86_64-linux-gnu/libtinfo.so.5.9 7f6560847000-7f656094f000 r-xp 00000000 fd:01 5767444 /lib/x86_64-linux-gnu/libm-2.23.so 7f656094f000-7f6560b4e000 ---p 00108000 fd:01 5767444 /lib/x86_64-linux-gnu/libm-2.23.so 7f6560b4e000-7f6560b4f000 r--p 00107000 fd:01 5767444 /lib/x86_64-linux-gnu/libm-2.23.so 7f6560b4f000-7f6560b50000 rw-p 00108000 fd:01 5767444 /lib/x86_64-linux-gnu/libm-2.23.so 7f6560b50000-7f6560b6d000 r-xp 00000000 fd:01 1453147 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.4 7f6560b6d000-7f6560d6d000 ---p 0001d000 fd:01 1453147 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.4 7f6560d6d000-7f6560d6e000 r--p 0001d000 fd:01 1453147 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.4 7f6560d6e000-7f6560d6f000 rw-p 0001e000 fd:01 1453147 /usr/lib/x86_64-linux-gnu/libyaml-0.so.2.0.4 7f6560d6f000-7f6560dac000 r-xp 00000000 fd:01 5767753 /lib/x86_64-linux-gnu/libreadline.so.6.3 7f6560dac000-7f6560fac000 ---p 0003d000 fd:01 5767753 /lib/x86_64-linux-gnu/libreadline.so.6.3 7f6560fac000-7f6560fae000 r--p 0003d000 fd:01 5767753 /lib/x86_64-linux-gnu/libreadline.so.6.3 7f6560fae000-7f6560fb4000 rw-p 0003f000 fd:01 5767753 /lib/x86_64-linux-gnu/libreadline.so.6.3 7f6560fb4000-7f6560fb5000 rw-p 00000000 00:00 0 7f6560fb5000-7f6561175000 r-xp 00000000 fd:01 5767448 /lib/x86_64-linux-gnu/libc-2.23.so 7f6561175000-7f6561375000 ---p 001c0000 fd:01 5767448 /lib/x86_64-linux-gnu/libc-2.23.so 7f6561375000-7f6561379000 r--p 001c0000 fd:01 5767448 /lib/x86_64-linux-gnu/libc-2.23.so 7f6561379000-7f656137b000 rw-p 001c4000 fd:01 5767448 /lib/x86_64-linux-gnu/libc-2.23.so 7f656137b000-7f656137f000 rw-p 00000000 00:00 0 7f656137f000-7f6561397000 r-xp 00000000 fd:01 1451349 /usr/lib/liblnetconfig.so.4.0.0 7f6561397000-7f6561596000 ---p 00018000 fd:01 1451349 /usr/lib/liblnetconfig.so.4.0.0 7f6561596000-7f6561597000 r--p 00017000 fd:01 1451349 /usr/lib/liblnetconfig.so.4.0.0 7f6561597000-7f6561598000 rw-p 00018000 fd:01 1451349 /usr/lib/liblnetconfig.so.4.0.0 7f6561598000-7f65615a0000 rw-p 00000000 00:00 0 7f65615a0000-7f65615b9000 r-xp 00000000 fd:01 5767786 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f65615b9000-7f65617b8000 ---p 00019000 fd:01 5767786 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f65617b8000-7f65617b9000 r--p 00018000 fd:01 5767786 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f65617b9000-7f65617ba000 rw-p 00019000 fd:01 5767786 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f65617ba000-7f65617de000 r-xp 00000000 fd:01 1451350 /usr/lib/liblustreapi.so.1.0.0 7f65617de000-7f65619dd000 ---p 00024000 fd:01 1451350 /usr/lib/liblustreapi.so.1.0.0 7f65619dd000-7f65619de000 r--p 00023000 fd:01 1451350 /usr/lib/liblustreapi.so.1.0.0 7f65619de000-7f65619df000 rw-p 00024000 fd:01 1451350 /usr/lib/liblustreapi.so.1.0.0 7f65619df000-7f65619e8000 rw-p 00000000 00:00 0 7f65619e8000-7f6561a0e000 r-xp 00000000 fd:01 5767446 /lib/x86_64-linux-gnu/ld-2.23.so 7f6561bfd000-7f6561c03000 rw-p 00000000 00:00 0 7f6561c0c000-7f6561c0d000 rw-p 00000000 00:00 0 7f6561c0d000-7f6561c0e000 r--p 00025000 fd:01 5767446 /lib/x86_64-linux-gnu/ld-2.23.so 7f6561c0e000-7f6561c0f000 rw-p 00026000 fd:01 5767446 /lib/x86_64-linux-gnu/ld-2.23.so 7f6561c0f000-7f6561c10000 rw-p 00000000 00:00 0 7ffedfb87000-7ffedfba9000 rw-p 00000000 00:00 0 [stack] 7ffedfbef000-7ffedfbf2000 r--p 00000000 00:00 0 [vvar] 7ffedfbf2000-7ffedfbf4000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] /usr/lib64/lustre/tests/sanity.sh: line 1429: 5280 Aborted (core dumped) $LFS getstripe -c $DIR/$tdir/$tfile
It looks like this issue started on 2018-02-27 16:23:17 UTC. So far, we’ve only seen this while testing Ubuntu 16.04 clients.
Here are logs for a few failed test suites
https://testing.hpdd.intel.com/test_sets/5ae8fc72-287c-11e8-9e0e-52540065bddc
https://testing.hpdd.intel.com/test_sets/9f2131ee-2894-11e8-b3c6-52540065bddc
https://testing.hpdd.intel.com/test_sets/f530fcd0-1cb5-11e8-a7cd-52540065bddc
https://testing.hpdd.intel.com/test_sets/4d231688-2195-11e8-b046-52540065bddc
In these test sessions, test 27f, 27i, 27z, 27C, 27F, 56a, 56w, 56wb, 56wc, 56x, 56xa, 56xb, 65i, 102b, 102c, 102d, 102f, 102j, 130a/b/c/d/e, 229, 270a/c/d, 311 fail with a buffer overflow detected in lfs.
In these test sessions, we also see the follow tests fail with this error:
sanity-lfsck 18d, 18e, 18g, 20a, 20b
sanityn test 51c
sanity-hsm test 11a
sanity-flr test 0a, 0b, 0c, 0d, 0e, 0f, 0g, 0h, 1, 2, 4, 5, 32, 32, 36, 37, 38, 39, 40, 41, 42, 43, 45, 200
sanity-dom test 42e, 251, 51c, 51d