[LU-3716] mds-survey ASSERTION( lu_device_is_mdt(o->lo_dev) )

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.7.0
• Affects Version/s: Lustre 2.4.0, Lustre 2.5.1
• Labels: HB llnl server zfs
• Severity: 3
• Assignee: Jian Yu
• Reporter: Brian Behlendorf

Description

Observed fairly repeatably when running mds-survey.sh after the first few million creates. It appears unlikely to occur when creating less than 100,00 objects (the default).

2013-08-06 15:05:51 LustreError: 4379:0:(mdt_handler.c:2352:mdt_obj()) ASSERTION( lu_device_is_mdt(o->lo_dev) ) failed:
2013-08-06 15:05:51 LustreError: 4379:0:(mdt_handler.c:2352:mdt_obj()) LBUG

The failures where observed with code post 2.4.0 which includes the LU-2758 fix.

Comments

[Gerrit Updater]

Jian Yu ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13530
Subject: LU-3716 obdecho: create a separate root object for echo access
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 9a208e3e205a1a31b2773893f8333dbc275e0e6e

[Peter Jones]

Landed for 2.7

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/10130/
Subject: LU-3716 obdecho: create a separate root object for echo access
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 897580eb1562e0509bbe8ea72a273ed71f878eaa

[Jian Yu]

Can this patch be rebased?

Sure, will do.

[Doug Oucharek (Inactive)]

Can this patch be rebased?

[Jian Yu]

Patch for master branch: http://review.whamcloud.com/10130

[Di Wang]

IMHO, I do not think you need(should) change MDD. I think you create create a separate root object for echo access, so in echo_resolve_path, it will always start from that root object, and form its own tree, instead of using file system ROOT. And normal client will never see this tree under this special root.

You can see how MDD create the root object in mdd_prepare.

[Jian Yu]

Currently, in mdd_create(), both echo object and normal object are created via:

mdd_object_create_internal()->...->osd_object_ea_create()

and inserted into normal namespace via:

__mdd_index_insert()->...->osd_index_ea_insert()->...->__osd_ea_add_rec()->osd_ldiskfs_add_entry()

The objects are looked up via:

mdd_lookup()->...->osd_index_ea_lookup()->osd_ea_lookup_rec()->osd_ldiskfs_find_entry()

To separate echo objects from normal namespace, it seems we can check "FID_SEQ_ECHO" sequence number after mdd_object_create() and do not run __mdd_index_insert() for echo objects. However, this will make the echo objects unable to be looked up.

A possible way is to make sure the "parent" object for echo object is always different from the one for normal object, as Di mentioned in previous comment that "Probably echo test should be done in a special directory".

Hi Di, is this a proper way to resolve the issue? If yes, could you please suggest what special directory I should create in jt_obd_md_common()? Thanks.

[Di Wang]

Yes, that is what I mean, i.e. we should separate echo objects from the normal namespace, so normal clients will not access these echo objects.

[Jian Yu]

For echo client, the echo md objects are created by echo_md_create_internal()->mdd_create():

00008000:00100000:1.0:1397818339.647451:0:3611:0:(echo_client.c:1602:echo_md_create_internal()) Start creating object [0x200000400:0x1:0x0] 3373 ffff88031cd5a5a0
00000004:00000001:1.0:1397818339.647453:0:3611:0:(mdd_dir.c:2164:mdd_create()) Process entered
......
00000004:00000001:1.0:1397818339.647535:0:3611:0:(mdd_object.c:375:mdd_object_create_internal()) Process entered
......
00000004:00000001:1.0:1397818339.647564:0:3611:0:(mdd_object.c:383:mdd_object_create_internal()) Process leaving (rc=0 : 0 : 0)
......
00000004:00000001:1.0:1397818339.647606:0:3611:0:(mdd_dir.c:2272:mdd_create()) Process leaving
00008000:00100000:1.0:1397818339.647614:0:3611:0:(echo_client.c:1614:echo_md_create_internal()) End creating object [0x200000400:0x1:0x0] 3373 ffff88031cd5a5a0 rc  = 0

And looked up by echo_lookup_object()->mdd_lookup():

00008000:00100000:1.0:1397818505.932739:0:3626:0:(echo_client.c:1897:echo_lookup_object()) Start lookup object [0x200000400:0x1:0x0] 3373 ffff88031cd5a5a0
00000004:00000001:1.0:1397818505.932741:0:3626:0:(mdd_dir.c:115:mdd_lookup()) Process entered
00000004:00000001:1.0:1397818505.932741:0:3626:0:(mdd_dir.c:72:__mdd_lookup()) Process entered
......
00000004:00000001:1.0:1397818505.932760:0:3626:0:(mdd_dir.c:106:__mdd_lookup()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:1.0:1397818505.932761:0:3626:0:(mdd_dir.c:122:mdd_lookup()) Process leaving (rc=0 : 0 : 0)
00008000:00100000:1.0:1397818505.932762:0:3626:0:(echo_client.c:1905:echo_lookup_object()) End lookup object [0x200000400:0x1:0x0] 3373 ffff88031cd5a5a0

And destroyed by echo_md_destroy_internal()->mdd_unlink():

00008000:00100000:9.0:1397818817.663055:0:3650:0:(echo_client.c:1944:echo_md_destroy_internal()) Start destroy object [0x200000400:0x1:0x0] 3373 ffff88031cd5a5a0
00000004:00000001:9.0:1397818817.663057:0:3650:0:(mdd_dir.c:1499:mdd_unlink()) Process entered
......
00000004:00000001:9.0:1397818817.663166:0:3650:0:(mdd_dir.c:1603:mdd_unlink()) Process leaving
00008000:00100000:9.0:1397818817.663182:0:3650:0:(echo_client.c:1953:echo_md_destroy_internal()) End destroy object [0x200000400:0x1:0x0] 3373 ffff88031cd5a5a0

Hi Di,

If we allow echo client and normal client attached and operate at the same time, we should distinguish echo md objects from normal md objects, do I understand correctly?