[LU-3534] async update cross-MDTs

• Type: New Feature
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.8.0
• Severity: 3
• Assignee: Di Wang
• Reporter: Di Wang

Description

This bug is for tracking async updates between MDTs.

Issue Links

• is duplicated by: LU-5302 Test failure sanity-lfsck test_13: mdt panic
• is related to: LU-6297 Move rename is_subdir check from MDD to MDT
• is related to: LU-6380 OI scrub should be able to scan the update log file before replay
• is related to: LU-5571 Test failure sanity-lfsck test_13: (2) unexpected status
• is related to: LU-7102 replay-dual test_26: FAIL: set default dirstripe failed
• is related to: LU-6287 sanity test 17n ASSERTION( atomic_read(&lu->ld_ref) == 0 ) failed
• is related to: LU-6288 conf-sanity test_2: test failed to respond and timed out
• is related to: LU-6289 sanity-scrub test_1c: unable to handle kernel NULL pointer dereference at 0000000000000010 19:17:56:IP: [<ffffffffa05d5079>] dt_los_put+0x9/0x30 [obdclass]
• is related to: LU-6290 sanity-lfsck test_14: unable to handle kernel NULL pointer dereference at 0000000000000010 13:56:17:IP: [<ffffffffa108678e>] osp_declare_object_create+0x12e/0x640 [osp]
• is related to: LU-6291 conf-sanity test_41a: failed to respond and timed out
• is related to: LU-6292 replay-single test_101: osd_trans_exec_op()) ASSERTION( oh->ot_handle != ((void *)0) ) failed:
• is related to: LU-6293 runtests test_1: panic on dbuf_dirty (async update for zfs DNE)
• is related to: LU-6294 sanity-scrub test_1a: test failed to respond and timed out
• is related to: LU-6295 sanity-lfsck test_4: oom on MDT0
• is related to: LU-6296 insanity test_1: check_for_recovery_ready()) ASSERTION( clnts <= obd->obd_max_recoverable_clients )
• is related to: LU-6328 sanity-lfsck test_14:unexpected size
• is related to: LU-6329 replay-single test_101: kernel panic on out_tx_attr_set_undo+0x64
• is related to: LU-6330 sanity test_17n:migrate failed -1
• is related to: LU-4583 Test failure on test suite replay-single, subtest test_80b (remote creation failed)
• is related to: LU-6362 DNE2: add dt object and thandle check in OSP and OSD
• is related to: LU-3541 add sanity tests for async updates between MDT
• is related to: LU-4837 DNE 2 async update cross-MDTs Test Plan
• is related to: LU-5222 Some operation should never happened on the sub-stripe.
• is related to: LU-6663 DNE2 directories has very very bad performance
• is related to: LU-6262 replay-single test_101: Oops at out_tx_attr_set_undo()

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/15163/
Subject: LU-3534 tests: a few tests cases for async update.
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c2b82ef8a12e6c55f143fbd8986c094425ed667e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13786/
Subject: LU-3534 osp: transfer updates with bulk RPC
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 044fbff00b25f127560338d35caa4d89faa4c207

[Gerrit Updater]

wangdi ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15482
Subject: LU-3534 osp: transfer updates with bulk RPC
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e1e3d405464b7807f0a016aff8c7df9fb174fc47

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/15421/
Subject: LU-3534 ptlrpc: mbits is sent within ptlrpc_body
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d099fdd6cd15d0d00d9b573da5d3bfd3e4bbcb9d

[Gerrit Updater]

Liang Zhen ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15421
Subject: LU-3534 ptlrpc: mbits is sent within ptlrpc_body
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b363b70a0c82bccb0d50026ccee756f72f7d3d3d

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12450/
Subject: LU-3534 update: change sync updates to async updates
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0136a91b6d629556ef091f5ca210c13772207df9

[Gerrit Updater]

wangdi ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15275
Subject: LU-3534 tests: Add dne-2.5 upgrade test
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f17fe3836034a2359f53839cd116d08d640c1c31

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12825/
Subject: LU-3534 osp: send updates by separate thread
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2fe22edfe3c365b5c270050fdeed0a86fa74a919

[Di Wang]

Yes, this is expected. LWP(light weight proxy) is not a replayable client, so it will never get into recovery mode. But OSP is a replayable client, which will always enter into recovery mode when required.

[James A Simmons]

In my testing I noticed end after a cleanup shut down of the file system when the file system is restarted all the MDS servers go into recovery with each other. The lwp layers reconnect fine without going into recovery but the OSP layer always enters recovery mode. Is this the excepted behavior?

[Gerrit Updater]

wangdi ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15163
Subject: LU-3534 tests: a few tests cases for async update.
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0394d9938e655b774d86a4eef15408e7c7e01e3f