[LU-5242] Test hang sanity test_132, test_133: umount ost

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.6.0, Lustre 2.7.0, Lustre 2.5.3
• Labels: prz
• Severity: 3
• Assignee: Isaac Huang (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for Nathaniel Clark <[e-mail]>

This issue relates to the following test suite run:

http://maloo.whamcloud.com/test_sets/e5783778-f887-11e3-b13a-52540035b04c.

The sub-test test_132 failed with the following error:

test failed to respond and timed out

Info required for matching: sanity 132

Issue Links

• duplicates: LU-5163 (lu_object.h:852:lu_object_attr()) ASSERTION( ((o)->lo_header->loh_attr & LOHA_EXISTS) != 0 ) failed
• duplicates: LU-6105 Update ZFS/SPL version to 0.6.3-1.2
• is duplicated by: LU-4716 replay-ost-single test_5: stuck in dbuf_read->zio_wait
• is related to: LU-6089 qsd_handler.c:1139:qsd_op_adjust()) ASSERTION( qqi ) failed
• is related to: LU-6155 osd_count_not_mapped() calls dbuf_hold_impl() without the lock
• is related to: LU-5277 sanity test_132: mdt_build_target_list(), unable to handle kernel NULL pointer dereference
• is related to: LU-5737 sanity test_132: client NULL req->rq_import in ptlrpc_request_committed()
• is related to: LU-6008 sanity test_102b: setstripe failed
• is related to: LU-6195 osd-zfs: osd_declare_object_destroy() calls dmu_tx_hold_zap() with wrong keys
• is related to: LU-4950 sanity-benchmark test fsx hung: txg_sync was stuck on OSS
• is related to: LU-7020 OST_DESTROY message times out on MDS repeatedly, indefinitely
• is related to: LU-2160 Implement ZFS dmu_tx_hold_append() declarations for llog
• is related to: LU-4968 Test failure sanity test_132: umount /mnt/ost2
• is related to: LU-3665 obdfilter-survey test_3a: unmount stuck in obd_exports_barrier()

Comments

[Peter Jones]

Landed for 2.8

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13630/
Subject: LU-5242 osd-zfs: umount hang in sanity 133g
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9b704e4088d867851cdb011f0a2560b1e622555c

[Gerrit Updater]

Nathaniel Clark ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13805
Subject: LU-5242 osd-zfs: umount hang in sanity 133g
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 817cf8a2e781d546508929a9f58b44561ae3361c

[Bob Glossman (Inactive)]

another seen on b2_5 with zfs:
https://testing.hpdd.intel.com/test_sessions/435c3152-b816-11e4-9ecb-5254006e85c2

[Andreas Dilger]

I would agree with Alex on this. By deferring unlink of small files it will probably double or triple the total IO that the MDT is doing because in addition to the actual dnode deletion it also needs to insert the dnode into the deathrow ZAP in one TXG and then delete it from the same ZAP in a different txg. If there are a large number of objects being deleted at once (easily possible on the MDT), then the deathrow ZAP may get quite large (and never shrink) and updates would become less efficient than if it is kept small.

[Isaac Huang (Inactive)]

Thanks all. I'll work on a patch first without the small object optimization to get this bug fixed; then will benchmark to figure out whether to optimize the small object path or not.

[Alex Zhuravlev]

I have no objection to do this as simple as possible. my point was that MDT is known to be CPU-bound and ZAP (even micro ZAP) isn't free.

[Brian Behlendorf]

I could see a case for zero length files as an possible optimization. But I suspect that even for the MDT it would be more efficient to handle the freeing asynchronously outside of any request processing. Even if the file is zero length you're still going to be freeing a spill block for the xattrs and updating a dbuf for the dnode object. Personally I'd keep it as simple and concise as possible until it's clear something more is required. But that's just my preference.

Keep in mind that none of this free space will be available for a couple TXGs anyway.

[Alex Zhuravlev]

I'd think that the optimization for small objects (notice most of MDT's objects are literally empty) make sense as we don't need to modify yet another ZAP twice. I guess there is no strong requirement to implement this right away, but still.

[Brian Behlendorf]

I like Andreas's idea of keeping the unlink behavior compatible with the ZPL. It would be ideal if you could reuse the existing ZPL functions but those functions are tied quite closely to ZPL specific data structures so that's probably not workable. But the ZFS_UNLINKED_SET object itself is just a ZAP containing a list of object ids. And since objects on disk are already constructed to be compatible with the ZPL we should be able to safely use it. Isaac's design is nice, but let me suggest a few minor tweaks:

• Use the existing ZFS_UNLINKED_SET object linked the MASTER_NODE_OBJ as the deathrow object.
• In declare_object_destroy() and object_destroy() just handle moving the object to the ZFS_UNLINKED_SET in a single TX.
• In a dedicated thread, taskq, or generic linux worker thread regularly walk the ZFS_UNLINKED_SET and rely on dmu_free_long_range() to split the free over as many TXGs as required.
• I don't think there's any advantage in handling small object destruction synchronously in object_destroy(). It's simpler and probably more efficient to always do this asynchronously.
• Start draining the ZFS_UNLINKED_SET right away when remounting the OSD (this happening during mount for the ZPL).