[LU-12253] lfs: double free or corruption (!prev)

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.13.0
• Affects Version/s: Lustre 2.13.0
• Severity: 3
• Assignee: WC Triage
• Reporter: Oleg Drokin

Description

current master has this problem in sanity-pfl (sanity-pfl test 6: Migrate composite file)

I suspect one of the recent lfs landings is the culprit?

run_suite sanity-pfl /home/green/git/lustre-release/lustre/tests/sanity-pfl.sh
*** Error in `/home/green/git/lustre-release/lustre/utils/lfs': double free or corruption (!prev): 0x0000000001b21990 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81489)[0x7f19b4458489]
/home/green/git/lustre-release/lustre/utils/lfs[0x42003a]
/home/green/git/lustre-release/lustre/utils/lfs[0x419af7]
/home/green/git/lustre-release/lustre/utils/lfs[0x41a2d6]
/home/green/git/lustre-release/lustre/utils/lfs[0x40f033]
/home/green/git/lustre-release/lustre/utils/lfs[0x429e91]
/home/green/git/lustre-release/lustre/utils/lfs[0x4031fe]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f19b43f93d5]
/home/green/git/lustre-release/lustre/utils/lfs[0x4032fa]
======= Memory map: ========
00400000-00442000 r-xp 00000000 fd:00 1786                               /home/green/git/lustre-release/lustre/utils/lfs
00641000-00642000 r--p 00041000 fd:00 1786                               /home/green/git/lustre-release/lustre/utils/lfs
00642000-00643000 rw-p 00042000 fd:00 1786                               /home/green/git/lustre-release/lustre/utils/lfs
00643000-0064c000 rw-p 00000000 00:00 0 
01b07000-01b28000 rw-p 00000000 00:00 0                                  [heap]
7f19afdea000-7f19afdff000 r-xp 00000000 00:25 1049992                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f19afdff000-7f19afffe000 ---p 00015000 00:25 1049992                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f19afffe000-7f19affff000 r--p 00014000 00:25 1049992                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f19affff000-7f19b0000000 rw-p 00015000 00:25 1049992                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f19b0000000-7f19b0021000 rw-p 00000000 00:00 0 
7f19b0021000-7f19b4000000 ---p 00000000 00:00 0 
7f19b41d3000-7f19b41d5000 r-xp 00000000 00:25 1051404                    /usr/lib64/libdl-2.17.so
7f19b41d5000-7f19b43d5000 ---p 00002000 00:25 1051404                    /usr/lib64/libdl-2.17.so
7f19b43d5000-7f19b43d6000 r--p 00002000 00:25 1051404                    /usr/lib64/libdl-2.17.so
7f19b43d6000-7f19b43d7000 rw-p 00003000 00:25 1051404                    /usr/lib64/libdl-2.17.so
7f19b43d7000-7f19b4599000 r-xp 00000000 00:25 1051741                    /usr/lib64/libc-2.17.so
7f19b4599000-7f19b4799000 ---p 001c2000 00:25 1051741                    /usr/lib64/libc-2.17.so
7f19b4799000-7f19b479d000 r--p 001c2000 00:25 1051741                    /usr/lib64/libc-2.17.so
7f19b479d000-7f19b479f000 rw-p 001c6000 00:25 1051741                    /usr/lib64/libc-2.17.so
7f19b479f000-7f19b47a4000 rw-p 00000000 00:00 0 
7f19b47a4000-7f19b47a7000 r-xp 00000000 00:25 1050461                    /usr/lib64/libkeyutils.so.1.5
7f19b47a7000-7f19b49a6000 ---p 00003000 00:25 1050461                    /usr/lib64/libkeyutils.so.1.5
7f19b49a6000-7f19b49a7000 r--p 00002000 00:25 1050461                    /usr/lib64/libkeyutils.so.1.5
7f19b49a7000-7f19b49a8000 rw-p 00003000 00:25 1050461                    /usr/lib64/libkeyutils.so.1.5
7f19b49a8000-7f19b4aa9000 r-xp 00000000 00:25 1054169                    /usr/lib64/libm-2.17.so
7f19b4aa9000-7f19b4ca8000 ---p 00101000 00:25 1054169                    /usr/lib64/libm-2.17.so
7f19b4ca8000-7f19b4ca9000 r--p 00100000 00:25 1054169                    /usr/lib64/libm-2.17.so
7f19b4ca9000-7f19b4caa000 rw-p 00101000 00:25 1054169                    /usr/lib64/libm-2.17.so
7f19b4caa000-7f19b4cc9000 r-xp 00000000 00:25 1050150                    /usr/lib64/libyaml-0.so.2.0.4
7f19b4cc9000-7f19b4ec8000 ---p 0001f000 00:25 1050150                    /usr/lib64/libyaml-0.so.2.0.4
7f19b4ec8000-7f19b4ec9000 r--p 0001e000 00:25 1050150                    /usr/lib64/libyaml-0.so.2.0.4
7f19b4ec9000-7f19b4eca000 rw-p 0001f000 00:25 1050150                    /usr/lib64/libyaml-0.so.2.0.4
7f19b4eca000-7f19b4edf000 r-xp 00000000 00:25 1050148                    /usr/lib64/libz.so.1.2.7
7f19b4edf000-7f19b50de000 ---p 00015000 00:25 1050148                    /usr/lib64/libz.so.1.2.7
7f19b50de000-7f19b50df000 r--p 00014000 00:25 1050148                    /usr/lib64/libz.so.1.2.7
7f19b50df000-7f19b50e0000 rw-p 00015000 00:25 1050148                    /usr/lib64/libz.so.1.2.7
7f19b50e0000-7f19b5102000 r-xp 00000000 00:25 1053941                    /usr/lib64/ld-2.17.so
7f19b52f1000-7f19b52f6000 rw-p 00000000 00:00 0 
7f19b52fe000-7f19b5301000 rw-p 00000000 00:00 0 
7f19b5301000-7f19b5302000 r--p 00021000 00:25 1053941                    /usr/lib64/ld-2.17.so
7f19b5302000-7f19b5303000 rw-p 00022000 00:25 1053941                    /usr/lib64/ld-2.17.so
7f19b5303000-7f19b5304000 rw-p 00000000 00:00 0 
7ffcf6288000-7ffcf62aa000 rw-p 00000000 00:00 0                          [stack]
7ffcf6346000-7ffcf6348000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Issue Links

• is related to: LU-11367 integrate LSOM with lfs find

Comments

[Oleg Drokin]

Reverted LU-11367 from master to make this go away for now.

[Oleg Drokin]

yes, it does not cause the test to fail but use after free like that in the tools is still a problem.

I seem to get it go away by reverting LU-11367

[James Nunez (Inactive)]

I see this issue in sanity-pfl test 6, but it doesn't cause the test to fail. Looking at the suite_log at https://testing.whamcloud.com/test_sets/aff2a216-6b3f-11e9-bd0e-52540065bddc ,

== sanity-pfl test 6: Migrate composite file ========================================================= 09:14:18 (1556615658)
striped dir -i2 -c2 /mnt/lustre/d6.sanity-pfl
*** Error in `/usr/bin/lfs': double free or corruption (!prev): 0x0000000001548990 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81489)[0x7faf4e47a489]
/lib64/liblustreapi.so.1(+0xff2a)[0x7faf4f57af2a]
/lib64/liblustreapi.so.1(+0x99e7)[0x7faf4f5749e7]
/lib64/liblustreapi.so.1(+0xa1c6)[0x7faf4f5751c6]
/usr/bin/lfs[0x40ffb3]
/lib64/liblustreapi.so.1(Parser_execarg+0x51)[0x7faf4f585961]
/usr/bin/lfs[0x4041fe]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7faf4e41b3d5]
/usr/bin/lfs[0x40426c]
======= Memory map: ========
00400000-00425000 r-xp 00000000 fd:01 145944                             /usr/bin/lfs
00624000-00625000 r--p 00024000 fd:01 145944                             /usr/bin/lfs
00625000-00626000 rw-p 00025000 fd:01 145944                             /usr/bin/lfs
0152e000-0154f000 rw-p 00000000 00:00 0                                  [heap]
7faf48000000-7faf48021000 rw-p 00000000 00:00 0 
7faf48021000-7faf4c000000 ---p 00000000 00:00 0 
7faf4ddb5000-7faf4ddca000 r-xp 00000000 fd:01 131092                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7faf4ddca000-7faf4dfc9000 ---p 00015000 fd:01 131092                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7faf4dfc9000-7faf4dfca000 r--p 00014000 fd:01 131092                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7faf4dfca000-7faf4dfcb000 rw-p 00015000 fd:01 131092                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7faf4dfcb000-7faf4dfcd000 r-xp 00000000 fd:01 136292                     /usr/lib64/libdl-2.17.so
7faf4dfcd000-7faf4e1cd000 ---p 00002000 fd:01 136292                     /usr/lib64/libdl-2.17.so
7faf4e1cd000-7faf4e1ce000 r--p 00002000 fd:01 136292                     /usr/lib64/libdl-2.17.so
7faf4e1ce000-7faf4e1cf000 rw-p 00003000 fd:01 136292                     /usr/lib64/libdl-2.17.so
7faf4e1cf000-7faf4e1f4000 r-xp 00000000 fd:01 136651                     /usr/lib64/libtinfo.so.5.9
7faf4e1f4000-7faf4e3f4000 ---p 00025000 fd:01 136651                     /usr/lib64/libtinfo.so.5.9
7faf4e3f4000-7faf4e3f8000 r--p 00025000 fd:01 136651                     /usr/lib64/libtinfo.so.5.9
7faf4e3f8000-7faf4e3f9000 rw-p 00029000 fd:01 136651                     /usr/lib64/libtinfo.so.5.9
7faf4e3f9000-7faf4e5bb000 r-xp 00000000 fd:01 136286                     /usr/lib64/libc-2.17.so
7faf4e5bb000-7faf4e7bb000 ---p 001c2000 fd:01 136286                     /usr/lib64/libc-2.17.so
7faf4e7bb000-7faf4e7bf000 r--p 001c2000 fd:01 136286                     /usr/lib64/libc-2.17.so
7faf4e7bf000-7faf4e7c1000 rw-p 001c6000 fd:01 136286                     /usr/lib64/libc-2.17.so
7faf4e7c1000-7faf4e7c6000 rw-p 00000000 00:00 0 
7faf4e7c6000-7faf4e7c9000 r-xp 00000000 fd:01 136945                     /usr/lib64/libkeyutils.so.1.5
7faf4e7c9000-7faf4e9c8000 ---p 00003000 fd:01 136945                     /usr/lib64/libkeyutils.so.1.5
7faf4e9c8000-7faf4e9c9000 r--p 00002000 fd:01 136945                     /usr/lib64/libkeyutils.so.1.5
7faf4e9c9000-7faf4e9ca000 rw-p 00003000 fd:01 136945                     /usr/lib64/libkeyutils.so.1.5
7faf4e9ca000-7faf4ea06000 r-xp 00000000 fd:01 137326                     /usr/lib64/libreadline.so.6.2
7faf4ea06000-7faf4ec06000 ---p 0003c000 fd:01 137326                     /usr/lib64/libreadline.so.6.2
7faf4ec06000-7faf4ec08000 r--p 0003c000 fd:01 137326                     /usr/lib64/libreadline.so.6.2
7faf4ec08000-7faf4ec0e000 rw-p 0003e000 fd:01 137326                     /usr/lib64/libreadline.so.6.2
7faf4ec0e000-7faf4ec10000 rw-p 00000000 00:00 0 
7faf4ec10000-7faf4ed11000 r-xp 00000000 fd:01 136294                     /usr/lib64/libm-2.17.so
7faf4ed11000-7faf4ef10000 ---p 00101000 fd:01 136294                     /usr/lib64/libm-2.17.so
7faf4ef10000-7faf4ef11000 r--p 00100000 fd:01 136294                     /usr/lib64/libm-2.17.so
7faf4ef11000-7faf4ef12000 rw-p 00101000 fd:01 136294                     /usr/lib64/libm-2.17.so
7faf4ef12000-7faf4ef31000 r-xp 00000000 fd:01 141229                     /usr/lib64/libyaml-0.so.2.0.4
7faf4ef31000-7faf4f130000 ---p 0001f000 fd:01 141229                     /usr/lib64/libyaml-0.so.2.0.4
7faf4f130000-7faf4f131000 r--p 0001e000 fd:01 141229                     /usr/lib64/libyaml-0.so.2.0.4
7faf4f131000-7faf4f132000 rw-p 0001f000 fd:01 141229                     /usr/lib64/libyaml-0.so.2.0.4
7faf4f132000-7faf4f14b000 r-xp 00000000 fd:01 145979                     /usr/lib64/liblnetconfig.so.4.0.0
7faf4f14b000-7faf4f34b000 ---p 00019000 fd:01 145979                     /usr/lib64/liblnetconfig.so.4.0.0
7faf4f34b000-7faf4f34c000 r--p 00019000 fd:01 145979                     /usr/lib64/liblnetconfig.so.4.0.0
7faf4f34c000-7faf4f34d000 rw-p 0001a000 fd:01 145979                     /usr/lib64/liblnetconfig.so.4.0.0
7faf4f34d000-7faf4f355000 rw-p 00000000 00:00 0 
7faf4f355000-7faf4f36a000 r-xp 00000000 fd:01 136604                     /usr/lib64/libz.so.1.2.7
7faf4f36a000-7faf4f569000 ---p 00015000 fd:01 136604                     /usr/lib64/libz.so.1.2.7
7faf4f569000-7faf4f56a000 r--p 00014000 fd:01 136604                     /usr/lib64/libz.so.1.2.7
7faf4f56a000-7faf4f56b000 rw-p 00015000 fd:01 136604                     /usr/lib64/libz.so.1.2.7
7faf4f56b000-7faf4f58e000 r-xp 00000000 fd:01 145983                     /usr/lib64/liblustreapi.so.1.0.0
7faf4f58e000-7faf4f78e000 ---p 00023000 fd:01 145983                     /usr/lib64/liblustreapi.so.1.0.0
7faf4f78e000-7faf4f78f000 r--p 00023000 fd:01 145983                     /usr/lib64/liblustreapi.so.1.0.0
7faf4f78f000-7faf4f790000 rw-p 00024000 fd:01 145983                     /usr/lib64/liblustreapi.so.1.0.0
7faf4f790000-7faf4f799000 rw-p 00000000 00:00 0 
7faf4f799000-7faf4f7bb000 r-xp 00000000 fd:01 136279                     /usr/lib64/ld-2.17.so
7faf4f9a8000-7faf4f9af000 rw-p 00000000 00:00 0 
7faf4f9b7000-7faf4f9ba000 rw-p 00000000 00:00 0 
7faf4f9ba000-7faf4f9bb000 r--p 00021000 fd:01 136279                     /usr/lib64/ld-2.17.so
7faf4f9bb000-7faf4f9bc000 rw-p 00022000 fd:01 136279                     /usr/lib64/ld-2.17.so
7faf4f9bc000-7faf4f9bd000 rw-p 00000000 00:00 0 
7fffce35b000-7fffce37e000 rw-p 00000000 00:00 0                          [stack]
7fffce3e3000-7fffce3e5000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
5+0 records in