[LU-2200] Test failure on test suite conf-sanity, subtest test_32a

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.4.1, Lustre 2.5.0
• Affects Version/s: Lustre 2.4.0, Lustre 2.4.1
• Severity: 3
• Assignee: Nathaniel Clark
• Reporter: Maloo

Description

This issue was created by maloo for Oleg Drokin <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/1e253cd6-17c2-11e2-a41f-52540035b04c.

The sub-test test_32a failed with the following error:

CMD: client-20-ib mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt
client-20-ib: mount.lustre: mount /dev/loop0 at /tmp/t32/mnt/mdt failed: No such file or directory
client-20-ib: Is the MGS specification correct?
client-20-ib: Is the filesystem name correct?
client-20-ib: If upgrading, is the copied client log valid? (see upgrade docs)
conf-sanity test_32a: @@@@@@ FAIL: Mounting the MDT
test_32a failed with 1

Info required for matching: conf-sanity 32a

Issue Links

• is blocked by: LU-3357 lctl replace_nids loses Target UUID from llog header
• is related to: LU-1997 'exclude' mount option does not work in lustre b2_1
• is related to: LU-1430 Changing network address without --writeconf option

Comments

[Jian Yu]

Patch http://review.whamcloud.com/6197 was cherry-picked to Lustre b2_4 branch.

[Jinshan Xiong (Inactive)]

reproduced again: https://maloo.whamcloud.com/test_sets/2b3a0a00-fcc8-11e2-9fdb-52540035b04c

from the debug log of client 2: https://maloo.whamcloud.com/test_logs/b0e58a3a-fcc8-11e2-9fdb-52540035b04c/show_text

10000000:01000000:1.0:1375567124.192540:0:27905:0:(mgc_request.c:1820:mgc_process_log()) MGC192.168.4.20@o2ib: configuration from log 'lustre-sptlrpc' failed (-2).

is it the same problem?

[Nathaniel Clark]

Patch merged to master (post 2.4.51)

[Nathaniel Clark]

Because of LU-3357, using replace_nids causes the first lctl conf_param ($fsname-OST0000.osc.max_dirty_mb=15) to fail.

[Nathaniel Clark]

Testing lctl replace_nids now. Skipping non-writeconf tests will does allow 32b to be run successfully on IB nodes (See patch set 10)

[Andreas Dilger]

What about using "lctl replace_nids" to fix up the configuration for IB testing?

[Nathaniel Clark]

Because the tcp based nids are in the config logs, a writeconf is needed before the existing filesystem tar balls can be mounted on IB nodes. I hope that any conf-sanity/32 test that does writeconf will be able to work on IB, but the non-writeconf test (32a), I believe, has no chance of passing without major revisions to the lustre stack.

[Keith Mannthey (Inactive)]

I opend a possible related LU. LU-3347 (local_storage.c:872:local_oid_storage_init()) ASSERTION( (*los)->los_last_oid >= first_oid ) failed: 0 < 1

It is a timeout error for test_32a.

[Nathaniel Clark]

This seems to be the order that llog_process_thread sends LCFG commands through class_config_llog_handler().

1) LCFG_MARKER (10)	- find if excluded
2) LCFG_ADD_UUID (5)	
3) LCFG_ATTACH (1)		
4) LCFG_SETUP (3)	- ERROR
5) LCFG_LOV_ADD_OBD (d)	- Only interation with EXCLUDED flag: Change to LCFG_LOV_ADD_INA (13)

Step 5 would be where the OST is excluded by adding it as inactive, but during step 4 OBD tries to setup a connection and that fails

The call chain for LCFG_SETUP is:

class_setup
 obd_setup
  osp_device_alloc (as osp::ldto_device_alloc)
   osp_init0
    client_obd_setup
     client_import_add_conn
      import_set_conn
       ptlrpc_uuid_to_connection
        ptrlrpc_uuid_to_peer -- Fails to find peer

[Nathaniel Clark]

Adding an OST to the exlude list seems to only affect the LCFG_LOV_ADD_OBD command (turning it into LCFG_LOV_ADD_INA), the problem (from my rough reading) seems to stem from the fact that the command in question is LCFG_ADD_UUID, and the NID in question is 192.168.203.129@tcp which fails on IB.