[LU-482] Test failure on test suite replay-dual, subtest test_0a Created: 05/Jul/11 Updated: 07/Jun/16 Resolved: 02/Feb/15 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.3.0, Lustre 2.1.3, Lustre 1.8.8 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor |
| Reporter: | Maloo | Assignee: | WC Triage |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Attachments: |
Create4thPartition
|
||||||||||||||||
| Issue Links: |
|
||||||||||||||||
| Severity: | 3 | ||||||||||||||||
| Rank (Obsolete): | 4591 | ||||||||||||||||
| Description |
|
This issue was created by maloo for yujian <yujian@whamcloud.com> This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/38eaf1c4-a6f2-11e0-bd2a-52540025f9af. The sub-test test_0a failed with the following error:
Info required for matching: replay-dual 0a |
| Comments |
| Comment by Jian Yu [ 05/Jul/11 ] |
|
Console log on the MDS node showed that: 21:07:08:LDISKFS-fs (dm-0): ldiskfs_check_descriptors: Checksum for group 128 failed (37201!=47318) 21:07:08:LDISKFS-fs (dm-0): group descriptors corrupted! 21:07:08:LustreError: 24509:0:(obd_mount.c:1392:server_kernel_mount()) ll_kern_mount failed: rc = -22 21:07:08:LustreError: 24509:0:(obd_mount.c:1672:server_fill_super()) Unable to mount device /dev/mapper/lvm-P0: -22 21:07:08:LustreError: 24509:0:(obd_mount.c:2151:lustre_fill_super()) Unable to mount (-22) 21:07:08:Lustre: DEBUG MARKER: replay-dual test_0a: @@@@@@ FAIL: Restart of mds1 failed! |
| Comment by Peter Jones [ 05/Jul/11 ] |
|
Niu will look into this one |
| Comment by Niu Yawei (Inactive) [ 06/Jul/11 ] |
|
I recalled that there are several similar metadata corruption reports on lustre-discuss, seems they were all running MDT on dm device. I suspect it's kind of issue of running ext4 on dm device, but I didn't find any bug report on ext4 maillist yet. |
| Comment by Niu Yawei (Inactive) [ 11/Jul/11 ] |
|
just found several other different symptoms: 04:38:26:LustreError: 14663:0:(md_local_object.c:433:llo_local_objects_setup()) creating obj [REM_OBJ_DIR] fid = [0x200000001:0xc:0x0] rc = -116 04:38:26:LustreError: 14663:0:(mdt_handler.c:4603:mdt_init0()) Can't init device stack, rc -116 04:38:26:LustreError: 14663:0:(obd_config.c:519:class_setup()) setup lustre-MDT0000 failed (-116) 04:38:26:LustreError: 14663:0:(obd_config.c:1358:class_config_llog_handler()) Err -116 on cfg command: 04:38:26:Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f 04:38:26:LustreError: 15c-8: MGC10.10.4.72@tcp: The configuration from log 'lustre-MDT0000' failed (-116). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. 04:38:27:LustreError: 14624:0:(obd_mount.c:1183:server_start_targets()) failed to start server lustre-MDT0000: -116 04:38:27:LustreError: 14624:0:(obd_mount.c:1710:server_fill_super()) Unable to start targets: -116 04:38:27:LustreError: 14624:0:(obd_config.c:564:class_cleanup()) Device 3 not setup 04:38:32:LustreError: 14624:0:(obd_mount.c:2151:lustre_fill_super()) Unable to mount (-116) 03:41:33:LustreError: 11951:0:(obd_mount.c:291:ldd_parse()) cannot open CONFIGS/mountdata: rc = -2 03:41:34:LustreError: 11951:0:(obd_mount.c:1364:server_kernel_mount()) premount parse options failed: rc = -2 03:41:34:LustreError: 11951:0:(obd_mount.c:1672:server_fill_super()) Unable to mount device /dev/mapper/lvm-P0: -2 03:41:34:LustreError: 11951:0:(obd_mount.c:2151:lustre_fill_super()) Unable to mount (-2) |
| Comment by Niu Yawei (Inactive) [ 12/Jul/11 ] |
|
The failure can also be found in early test which running over non-lvm device: 06:30:34:LustreError: 24808:0:(osd_handler.c:935:osd_ro()) *** setting device osd-ldiskfs read-only *** 06:30:34:Turning device sdb (0x800010) read-only 06:30:34:Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 06:30:35:Lustre: Failing over lustre-MDT0000 06:30:35:Lustre: Skipped 9 previous similar messages 06:30:35:Lustre: mdd_obd-lustre-MDT0000-0: shutting down for failover; client state will be preserved. 06:30:35:LustreError: 24914:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 06:30:35:LustreError: 24914:0:(ldlm_request.c:1169:ldlm_cli_cancel_req()) Skipped 3 previous similar messages 06:30:35:LustreError: 24914:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 06:30:35:LustreError: 24914:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) Skipped 3 previous similar messages 06:30:41:Removing read-only on unknown block (0x800010) 06:30:41:Lustre: server umount lustre-MDT0000 complete 06:30:41:Lustre: Skipped 3 previous similar messages 06:30:52:LDISKFS-fs (sdb): ldiskfs_check_descriptors: Checksum for group 128 failed (9760!=628) 06:30:52:LDISKFS-fs (sdb): group descriptors corrupted! 06:30:52:LustreError: 25027:0:(obd_mount.c:1394:server_kernel_mount()) ll_kern_mount failed: rc = -22 06:30:52:LustreError: 25027:0:(obd_mount.c:1675:server_fill_super()) Unable to mount device /dev/sdb: -22 06:30:52:LustreError: 25027:0:(obd_mount.c:2149:lustre_fill_super()) Unable to mount (-22) 06:30:52:Lustre: DEBUG MARKER: replay-dual test_0a: @@@@@@ FAIL: Restart of mds1 failed! |
| Comment by Niu Yawei (Inactive) [ 12/Jul/11 ] |
|
Hi, Chris Looks this failure often happen on vm, could you reserve a vm node for me to do the manual test? (seems now all vm node are reserved for auto-test). Thanks. |
| Comment by Chris Gearing (Inactive) [ 13/Jul/11 ] |
|
I'm creating a VM, a little bit delay because mjmac is working on git issues. As soon as it's up I will let you know. It will be client-14vm[1-4]. Everything like loadjenkinsbuild should work on it. As I said, I'll update when it's ready. |
| Comment by Chris Gearing (Inactive) [ 19/Jul/11 ] |
|
Niu, can I provide any more assistance to you one this. It’s not clear to me how I can help but if there is a way please ask. |
| Comment by Niu Yawei (Inactive) [ 19/Jul/11 ] |
|
Hi, Chris |
| Comment by Chris Gearing (Inactive) [ 20/Jul/11 ] |
|
Hi, How long will you require it for? The problem is that client-14 is the Chris |
| Comment by Chris Gearing (Inactive) [ 20/Jul/11 ] |
|
OK, I've finished with the nodes, they are now reserved in your name. |
| Comment by Niu Yawei (Inactive) [ 21/Jul/11 ] |
|
Thank you, Chris. I think two days is enough for me, so you can get it back next week. Could you show me how to create the lvm device for testing? Looks the auto-test vm nodes usually uses /dev/vda4 as the physical volume, but I didn't find /dev/vda4 on the client-14 vm nodes. |
| Comment by Chris Gearing (Inactive) [ 21/Jul/11 ] |
|
vm1 and vm2 are clients, vm3 is the MDS, vm4 is the OSS So I partition sda4 on each of the server nodes to be the rest of the disk. fdisk /dev/vda < Create4thPartition [I've attached Create4thPartition] Then I reboot the node. Then on the OSS. vgcreate lvm /dev/vda4 On the mds vgcreate lvm /dev/vda4 Then you have the disks. |
| Comment by Niu Yawei (Inactive) [ 24/Jul/11 ] |
|
Thanks a lot, Chris. I tried ran replay-dual 0a and replay-ost-single + replay-dual 0a many times on the lvm device, but failed to reproduce the bug. I don't see why replay-single 0a doesn't suffer from this failure, there is an extra 'sleep 10' in the replay-single 0a, I'm thinking that we could also add 'sleep 10' to the replay-dual 0a, then enable it in the auto-test to see if can be reproduced? Peter, Chris, is it ok? Thanks. |
| Comment by Peter Jones [ 25/Jul/11 ] |
|
Oleg would be ok with this as a test but would not want to make a habit of this practice |
| Comment by Niu Yawei (Inactive) [ 25/Jul/11 ] |
|
Thank you, Peter. My gerrit account has some problem recently, so I can't push patch for review now, I've asked Robert for help, and will push this diagnostic patch as soon as the account get ready. |
| Comment by Chris Gearing (Inactive) [ 25/Jul/11 ] |
|
This sounds OK you should be careful to fully comment the push so that Chris |
| Comment by Peter Jones [ 25/Jul/11 ] |
|
Dropping as blocker for now. May re-add if it starts reoccurring regularly. |
| Comment by Niu Yawei (Inactive) [ 28/Jul/11 ] |
|
Ok, I pushed the patch at http://review.whamcloud.com/#change,1157, let's see if the bug is reproduceable after the patch is pushed. Thanks. |
| Comment by nasf (Inactive) [ 04/Aug/11 ] |
|
Another failure instance: https://maloo.whamcloud.com/test_sets/048f8c56-bdfd-11e0-8bdf-52540025f9af |
| Comment by Andreas Dilger [ 16/Sep/11 ] |
|
Another failure https://maloo.whamcloud.com/test_sets/b965245c-e089-11e0-9909-52540025f9af |
| Comment by Minh Diep [ 23/Sep/11 ] |
|
another failure: https://maloo.whamcloud.com/test_sets/fac8e9cc-e5bd-11e0-9909-52540025f9af |
| Comment by Zhenyu Xu [ 27/Sep/11 ] |
|
another failure: https://maloo.whamcloud.com/test_sets/669543a6-e62e-11e0-9909-52540025f9af |
| Comment by nasf (Inactive) [ 09/Oct/11 ] |
|
another failure instance: |
| Comment by nasf (Inactive) [ 13/Oct/11 ] |
|
another failure: https://maloo.whamcloud.com/test_sets/3c4a840a-f5ef-11e0-9b90-52540025f9af |
| Comment by nasf (Inactive) [ 16/Oct/11 ] |
|
another failure: |
| Comment by Niu Yawei (Inactive) [ 27/Oct/11 ] |
|
I'm wondering if it's already been fixed in |
| Comment by Peter Jones [ 01/Nov/11 ] |
|
Lai will look into this one |
| Comment by Lai Siyao [ 03/Nov/11 ] |
|
A failure after In MDS dmesg, it shows barrier is not supported and disabled: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode JBD: barrier-based sync failed on dm-0-8 - disabling barriers There is an article about barrier support on LVM and on VM hypervisor: Write caching and write re-ordering by the hard drive is important to good performance, but due to lack of complete write barrier support in kernel versions < 2.6.33 (2.6.31 has some support, while 2.6.33 works for all types of device target), you may well find that a large amount of FS metadata (including journals) is lost by a hard crash that leaves data in the hard drives' write buffers. ... Running LVM on top of a VM hypervisor such as VMware, Xen or KVM can create similar problems to a kernel without write barriers, due to write caching and write re-ordering. So if kernel < 2.6.33, barrier is not fully supported by LVM, and if system is deployed upon VM hypervisor, even kernel is okay, the similar data lost will be seen. I'm afraid we can't really fix this issue on current test environment, if Niu's patch (sleep 10 seconds before test start) can circumvent this failure, it's an acceptable solution. |
| Comment by nasf (Inactive) [ 05/Nov/11 ] |
|
Another failure instance: https://maloo.whamcloud.com/test_sets/04174d90-071d-11e1-b0d9-52540025f9af |
| Comment by Niu Yawei (Inactive) [ 05/Nov/11 ] |
This failure looks different, not sure if they are same. |
| Comment by nasf (Inactive) [ 05/Nov/11 ] |
|
It maybe not the same as the original issue, but it is quite similar as the failure case you mentioned at 11/Jul/11 10:16 AM: 09:34:54:LustreError: 3764:0:(md_local_object.c:433:llo_local_objects_setup()) creating obj [fld] fid = [0x200000001:0x3:0x0] rc = -116 |
| Comment by nasf (Inactive) [ 05/Nov/11 ] |
|
Another failure instance: https://maloo.whamcloud.com/test_sets/edb71d5e-0803-11e1-b0d9-52540025f9af |
| Comment by Oleg Drokin [ 06/Nov/11 ] |
|
I reviewed all of this again and I am not suspecting the no barrier has little to do with our problems. The barrier is important for hard crashes of the nodes which does not happen in our tests most of the time. Instead we do readonly trick where all subsequent writes to devices would fail. Of course first we do sync to flush currently dirty data. My suspicion is a single sync would only move data from dirty cache of lvm to dirty cache of next in line block device and depending on how the readonly was implemented it still might be discarded in the end. So I have an alternative patch here: http://review.whamcloud.com/1656 that would hopefully work better. I was tempted to do "sync ; sleep 1 ; sync", but that would add an extra 1 second delay to all failover testing which is probably way too much overhead. Also note that this bug might hit tests other than 1st one after all where some of important data we expect to be committed would not be committed just as journal or initial formatting data would be lost in the first test. |
| Comment by Niu Yawei (Inactive) [ 06/Nov/11 ] |
|
In fact, this failure is also seen on non-lvm device in real machine, see my previous comment on 12/Jul/11 12:53 AM, or check the link:https://maloo.whamcloud.com/test_sets/e69561f8-82c4-11e0-b4df-52540025f9af So I still don't quite understand why it could happen. From the failure log, we can see when mds restart after replay_barrier(), the fisrt kernel mount always success, then it failed for:
so my guess is that the filesystem is corrupt after first mount, which might be caused by journal recovery during first mount? |
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 08/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Niu Yawei (Inactive) [ 09/Nov/11 ] |
|
the failure is still showing up: https://maloo.whamcloud.com/test_sets/87c26e80-0abb-11e1-b0d9-52540025f9af |
| Comment by Niu Yawei (Inactive) [ 09/Nov/11 ] |
|
In the replay_barrier(), we did a create after sync but before set rdonly, I think that could cause trouble if the set rdonly happened in the middle of journal commit, though it can't explain why we never seen the failure on the following tests or replay-single, but anyway, I think we'd better move this create before sync. |
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 30/Nov/11 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Andreas Dilger [ 01/Dec/11 ] |
|
Please see comment in http://review.whamcloud.com/1157. I'd like to make this timeout conditional on the kernel version instead of permanent. |
| Comment by Andreas Dilger [ 06/Dec/11 ] |
|
This is being addressed in http://review.whamcloud.com/#change,1761 |
| Comment by Zhenyu Xu [ 20/Dec/11 ] |
|
also hit in replay-ost-single test (https://maloo.whamcloud.com/test_sets/08bf83f4-2a8f-11e1-a412-5254004bbbd3) from OSS console log of test 0 22:20:49:LDISKFS-fs (dm-5): recovery complete |
| Comment by Andreas Dilger [ 24/Feb/12 ] |
|
Hit this three times in a row for a patch that cannot possibly be causing problems. https://maloo.whamcloud.com/test_sessions/55893158-58f2-11e1-a226-5254004bbbd3 (RHEL5.7 x86_64) I wonder if there is some other way to avoid this problem? I notice that while replay-dual.sh has a "sleep 10" after formatting the filesystem, it does not have "sync" before that, so it may well be that the data is not even starting to be flushed to the underlying disk yet. Once test_0a() sets replay_barrier it may discard all of the dirty data on the device. Adding a sync before the sleep is one simple solution that may hide this problem. Another possibility is to stop using LVM inside the VM guest, and instead map partitions or LVs in the host to be disk devices in the VM so that at least the flush operations in the guest will push the data from the guest out to the host where it will be persistent even if it is not on disk in the host. |
| Comment by Andreas Dilger [ 28/Feb/12 ] |
|
I've submitted a new test fix in http://review.whamcloud.com/2223, which adds sync; wait; sync before the first test is run, to avoid losing the blocks from just-formatted filesystems. This also includes the change I requested in http://review.whamcloud.com/1761 to limit this workaround to the affected kernels (currently all vendor kernels), so that we don't keep carrying this hack indefinitely in the future. |
| Comment by Andreas Dilger [ 28/Feb/12 ] |
|
Since this bug only hits in about 10% of test failures, it won't immediately be clear whether this change is resolving the problem, but is itself not harmful to the code and can be landed to master (after a successful autotest) so that it is included in all test runs. |
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 01/Mar/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Sarah Liu [ 19/Mar/12 ] |
|
Hit this issue when doing 2.2-RC1 test, RHEL6 server and client: |
| Comment by nasf (Inactive) [ 21/Mar/12 ] |
|
Similar failure: https://maloo.whamcloud.com/test_sets/33c75cc0-7233-11e1-8ebe-5254004bbbd3 |
| Comment by Lai Siyao [ 15/Apr/12 ] |
|
The failure shows mountdata file is missing on MGS, and previous fix added sync on lustre client, IMO it won't help for MGS. I'll submit a patch to do sync on MGS to see how it goes. |
| Comment by Peter Jones [ 16/Apr/12 ] |
| Comment by Peter Jones [ 16/Apr/12 ] |
|
Landed for 2.3 |
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 16/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 17/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 17/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 17/Apr/12 ] |
|
Integrated in Result = SUCCESS
|
| Comment by Build Master (Inactive) [ 02/May/12 ] |
|
Integrated in Result = SUCCESS
Oleg Drokin : c3d36069ca342a2f1ee97f983e7c49ae540688e5
|
| Comment by Build Master (Inactive) [ 02/May/12 ] |
|
Integrated in Result = SUCCESS
Oleg Drokin : c3d36069ca342a2f1ee97f983e7c49ae540688e5
|
| Comment by Build Master (Inactive) [ 02/May/12 ] |
|
Integrated in Result = SUCCESS
Oleg Drokin : c3d36069ca342a2f1ee97f983e7c49ae540688e5
|
| Comment by Build Master (Inactive) [ 02/May/12 ] |
|
Integrated in Result = SUCCESS
Oleg Drokin : c3d36069ca342a2f1ee97f983e7c49ae540688e5
|
| Comment by Build Master (Inactive) [ 02/May/12 ] |
|
Integrated in Result = SUCCESS
Oleg Drokin : c3d36069ca342a2f1ee97f983e7c49ae540688e5
|
| Comment by Build Master (Inactive) [ 02/May/12 ] |
|
Integrated in Result = SUCCESS
Oleg Drokin : c3d36069ca342a2f1ee97f983e7c49ae540688e5
|
| Comment by Build Master (Inactive) [ 02/May/12 ] |
|
Integrated in Result = SUCCESS
Oleg Drokin : c3d36069ca342a2f1ee97f983e7c49ae540688e5
|
| Comment by Andreas Dilger [ 03/May/12 ] |
|
This is still being hit in testing on master. |
| Comment by Andreas Dilger [ 03/May/12 ] |
|
This is still failing about 10% of the time: https://maloo.whamcloud.com/test_sets/697e6e7c-94c9-11e1-a028-525400d2bfa6 |
| Comment by Andreas Dilger [ 09/May/12 ] |
|
This is still causing 15% of all test runs to fail. I'm thinking that before we take the drastic step of disabling this test from being run entirely, it makes sense to catch the test_0a failure as it happens, and exit from the script in that case, but continue running the rest of replay-dual if test_0a passes. Something like: mount_facets () {
local facets=${1:-$(get_facets)}
local facet
for facet in ${facets//,/ }; do
if [ "$TESTSUITE.$TESTNAME" = "replay-dual.test_0a" ]; then
mount_facet $facet || skip "Restart of $facet failed! See LU-482."
else
mount_facet $facet || error "Restart of $facet failed!"
fi
done
}
and then in test_0a() it should check whether the MDS was actually restarted after facet_failover returns, or abort the test entirely (exit 0). Also, for posterity, here is the actual test failure output, which doesn't exist anywhere in this bug report. == replay-dual test 0a: expired recovery with lost client ============================================ 21:49:18 (1336538958)
Filesystem 1K-blocks Used Available Use% Mounted on
client-23vm3@tcp:/lustre
49867944 2266012 45068324 5% /mnt/lustre
total: 50 creates in 0.18 seconds: 285.24 creates/second
fail_loc=0x80000514
Failing mds1 on node client-23vm3
Stopping /mnt/mds1 (opts:) on client-23vm3
affected facets: mds1
Failover mds1 to client-23vm3
21:49:37 (1336538977) waiting for client-23vm3 network 900 secs ...
21:49:37 (1336538977) network interface is UP
Starting mds1: -o user_xattr,acl /dev/lvm-MDS/P1 /mnt/mds1
client-23vm3: mount.lustre: mount /dev/mapper/lvm--MDS-P1 at /mnt/mds1 failed: No such file or directory
client-23vm3: Is the MGS specification correct?
client-23vm3: Is the filesystem name correct?
client-23vm3: If upgrading, is the copied client log valid? (see upgrade docs)
mount -t lustre /dev/lvm-MDS/P1 /mnt/mds1
Start of /dev/lvm-MDS/P1 on mds1 failed 2
replay-dual test_0a: @@@@@@ FAIL: Restart of mds1 failed!
|
| Comment by nasf (Inactive) [ 14/May/12 ] |
|
Another failure instance: https://maloo.whamcloud.com/test_sets/628fd846-9d7d-11e1-a1d8-52540035b04c |
| Comment by Andreas Dilger [ 14/May/12 ] |
|
I've uploaded http://review.whamcloud.com/2731, which will abort replay-dual if test_0a fails to mount the MDS. This should allow 90% of replay-dual tests to continue to run properly, while avoiding spurious test failures due to this bug. |
| Comment by Jian Yu [ 16/May/12 ] |
|
Lustre Tag: v1_8_8_WC1_RC1 replay-ost-single test 0b failed with the similar issue: Console log on the OSS node (client-30vm4) showed that: 06:47:59:Lustre: DEBUG MARKER: mkdir -p /mnt/ost1; mount -t lustre /dev/lvm-OSS/P1 /mnt/ost1 06:47:59:LDISKFS-fs (dm-0): ldiskfs_check_descriptors: Checksum for group 0 failed (60782!=34767) 06:48:00:LDISKFS-fs (dm-0): group descriptors corrupted! 06:48:00:LustreError: 12142:0:(obd_mount.c:1307:server_kernel_mount()) premount /dev/lvm-OSS/P1:0x0 ldiskfs failed: -22, ldiskfs2 failed: -19. Is the ldiskfs module available? 06:48:00:LustreError: 12142:0:(obd_mount.c:1633:server_fill_super()) Unable to mount device /dev/lvm-OSS/P1: -22 06:48:00:LustreError: 12142:0:(obd_mount.c:2065:lustre_fill_super()) Unable to mount (-22) 06:48:01:Lustre: DEBUG MARKER: /usr/sbin/lctl mark replay-ost-single test_0b: @@@@@@ FAIL: Restart of ost1 failed! recovery-double-scale also failed with the similar issue: |
| Comment by Andreas Dilger [ 21/May/12 ] |
|
It looks like the first failure with my patch applied is in: The test_0a() is reported to pass, but for some reason the test is marked is timed out. I'm not sure why the test script is reported as a failure, since all of the steps to exit the script normally should be run if the LU482 file is detected. |
| Comment by Jian Yu [ 22/May/12 ] |
|
Lustre Branch: master The lustre-initialization-1 stage failed with the same issue: 00:59:22:Lustre: DEBUG MARKER: mkdir -p /mnt/ost13; mount -t lustre /dev/lvm-OSS/P6 /mnt/ost13 00:59:22:LDISKFS-fs (dm-5): recovery complete 00:59:22:LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: 00:59:22:LDISKFS-fs (dm-5): ldiskfs_check_descriptors: Checksum for group 0 failed (16572!=40708) 00:59:22:LDISKFS-fs (dm-5): group descriptors corrupted! 00:59:22:LustreError: 6307:0:(obd_mount.c:1499:server_kernel_mount()) vfs_kern_mount failed: rc = -22 00:59:22:LustreError: 6307:0:(obd_mount.c:1794:server_fill_super()) Unable to mount device /dev/mapper/lvm--OSS-P6: -22 00:59:22:LustreError: 6307:0:(obd_mount.c:2359:lustre_fill_super()) Unable to mount (-22) https://maloo.whamcloud.com/test_sessions/0777c22a-a129-11e1-b20c-52540035b04c |
| Comment by Andreas Dilger [ 22/May/12 ] |
|
It seems that the client (I think?) is not being cleaned up properly (needs a "umount -f $DIR"?), and possibly the script needs to do "exit 0" in the error handling case. |
| Comment by Peter Jones [ 22/May/12 ] |
|
Keith Could you please look into why this tests sometimes fails in vm environments? Thanks Peter |
| Comment by Keith Mannthey (Inactive) [ 23/May/12 ] |
|
I am working on recreating this issue on my laptop VMs. This error 00:59:22:LDISKFS-fs (dm-5): group descriptors corrupted! Seems to be a common point that isn't good. Was there ever a time we didn't see this issue? We only see it in VM + LVM/DM + LDISKFS-fs? I tried for a few hours today in a loop to trigger it without LVM/DM on a very basic setup without luck. Tomorrow I will try and emulate the 8 osts + LVM/DM setup. Have we had different failure rates with and without the various patches in Master? |
| Comment by Jinshan Xiong (Inactive) [ 24/May/12 ] |
|
Hi Keith, can you please make a patch to disable this test and add it back after this issue has been fixed? |
| Comment by Chris Gearing (Inactive) [ 28/May/12 ] |
|
I carried out a bunch of VM + Physical testing and found that the error appear possible on either. The results are in these Maloo sessions; Physical: VM: So I don't think it's in anyway safe to say this is a VM issue, because the first case is physical. |
| Comment by Lai Siyao [ 29/May/12 ] |
|
Function replay_barrier() runs mcreate after syncing target, this looks suspicious, I'll commit a patch to move sync after mcreate. |
| Comment by Lai Siyao [ 29/May/12 ] |
|
Patch is at: http://review.whamcloud.com/#change,2931, Chris, could you help conduct repeated test against this patch? |
| Comment by Keith Mannthey (Inactive) [ 29/May/12 ] |
|
1500 runs later on my laptop VM setup: No sign of this issue but a hard panic on the MDS. The panic hung the box and I didn't have serial running so only a 90% screen shot of the panic. It was 3 days of testing and I didn't see "this" issue. Since we are seeing alot of failures I am lead to believe our visualization test environment is part of the issue. My laptop run ran the MDS and OSTs on SSD, I am going to move them fully to a single spindle put them into a constrained IO environment. If it is agreed that we want to skip this test I can provided a patch. It seems we have been living with this problem for a while and I don't know the full history. |
| Comment by Oleg Drokin [ 31/May/12 ] |
|
Here are my thoughts of the issue and how to hopefully better reproduce it. It was noted when we moved to IU this was hit much more often. So the theory is when you start with a fresh (zero filled essentially) disk image for would be lustre FS, when the initial lustre format fails to make it to disk, the mdt mount fails. So for reproduction it seems it's a good idea to zero-fill (important for ext4) parts of the underlying disk before every test restart on the host, outside of VM. If tis is confirmed, then the efforts could be 2-way. One is to perhaps try to impose a lustre-looking fs image at the fs (as a temp workaround), and then in parallel seeing how to make the mdt actually write stuff to disk. |
| Comment by Keith Mannthey (Inactive) [ 12/Jun/12 ] |
|
Well I have been running a few different test paths and I thought I would make notes of them here. 1st: Running the test directly in any form I still have not been able to locally trigger the issue. 2: I wrote a quick script do to an IO to the LVM device, mkfs.lustre, mount, umount loop. It not quite clear if this make/mount/unmount loop test is related this this bug but is seems worth investigating as the real test produces similar actions. |
| Comment by Peter Jones [ 25/Jun/12 ] |
|
Latest workaround patch has made this less invasive so dropping priority |
| Comment by Jodi Levi (Inactive) [ 26/Jul/12 ] |
|
Reduced to Critical because it is not happening as often. |
| Comment by Jian Yu [ 13/Aug/12 ] |
|
More instances on Lustre 2.1.3 RC1: |
| Comment by Jian Yu [ 21/Aug/12 ] |
|
More instances on Lustre 2.1.3 RC2: |
| Comment by Jian Yu [ 10/Oct/12 ] |
|
Lustre Server Build: http://build.whamcloud.com/job/lustre-b2_3/32 |
| Comment by Jian Yu [ 18/Dec/12 ] |
|
Lustre Client: v2_1_4_RC1 |
| Comment by Keith Mannthey (Inactive) [ 02/May/13 ] |
|
I took a good look at maloo today a few things stand out. Master review has not see this issue since last summer but review seem to no run replay_dual since about the same time. The error: test_0a failed with 1 Looks like it might be the same same error as it reports stat: cannot read file system information for `/mnt/lustre': Resource temporarily unavailable replay-dual test_0a: @@@@@@ FAIL: test_0a failed with 1 And we know what part of the problem was there was some I can't seem to find a spot where this has failed on master in a while. It for sure plagues 2.1,2.2 and 2.3 but I don't see many signs of issues on present day Master. The more recent: gethostbyname("hostname") failed
Errors were fixed by: There is no sign of the lu-482 skip that the test makes possible. The test reports 7 out of 100. Those 7 are "full" tests running interopt with 2.1 |
| Comment by Kelsey Prantis (Inactive) [ 11/Oct/13 ] |
|
Was the source of the corruption of the group descriptors ever nailed down? I am seeing something similar in some failover testing on 2.3. |
| Comment by Brian Murrell (Inactive) [ 12/Oct/13 ] |
|
To add to kelsey's question, the environment we are seeing this in is also VM clusters however the block devices in the VMs that are used for Lustre target are not on LVs, but are directly on the virtual disks provided by KVM. |
| Comment by Andreas Dilger [ 02/Feb/15 ] |
|
This appears to be the reason that replay-dual was first disabled. The recent test runs don't show any failures in this test. |
| Comment by Andreas Dilger [ 02/Feb/15 ] |
|
This does not appear to be failing anymore. |
| Comment by Gerrit Updater [ 08/Apr/15 ] |
|
Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: http://review.whamcloud.com/14414 |
| Comment by Andreas Dilger [ 07/Jun/16 ] |
|
Pushed a patch under |
