[#LU-909] changes to OSD API

[LU-909] changes to OSD API Created: 12/Dec/11 Updated: 27/Mar/12 Resolved: 23/Jan/12
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.2.0
Fix Version/s:	Lustre 2.2.0

Type:

New Feature

Priority:

Minor

Reporter:

Alex Zhuravlev

Assignee:

Hongchao Zhang

Resolution:

Fixed

Votes:

Labels:

None

Attachments:

EuroLUG-2011-OSD-Alex-technical-version.pdf

mds_survey-OSD_API-perf-20111212

Rank (Obsolete):

4776

Description

the main purpose of the patch is to get declare methods in the API
and to teach osd-based devices to use that

Comments

Comment by Alex Zhuravlev [ 12/Dec/11 ]

http://review.whamcloud.com/#change,1669

Comment by Andreas Dilger [ 14/Dec/11 ]

EuroLUG 2011 OSD API presentation by Alex

Comment by Andreas Dilger [ 14/Dec/11 ]

DMU API

Explicit declarations in transaction mechanism
Create transaction
Declare changes in terms of objects and operations
Start transaction
Do changes
Stop transaction
No OS specifics except
- Bottom OSD
- libcfs

DMU OSD

Implements OSD API
Simple and obvious
Supposed to work with multiple backing stores
Sits atop of DMU, not ZPL

OSD API: before

Tables with credits in blocks
- Not very portable
No 0-copy IO support
No support for truncate
No explicit object destroy
- Hard to know when object should be destroyed

OSD API: after

Declare methods
- Very simple to use/implement
Support for 0-copy IO
Support for truncate
Object destroy method
- Called by MDD/OFD through the stack

llog: before

No way to declare changes
- Hard to use with DMU
llog_vfs.c
- Relying on linux vfs
- Sometimes looks like a stack
Not efficient in few cases
- 8K+ writes

llog: after

Set of new methods
- To declare
llog_osd.c
- Relying on OSD API
- More like a library now
To be fixed
- By skipping writes?

Obdfilter: before

Uses VFS
- Does low-level IO
- Using Linux specific methods

OFD: after

Uses OSD API
- Passes pages to OSD
- Where low-level IO is implemented

Configuration: before

Hardcoded in MDT
obd_mount does mounting
obd_mount parses mountdata
Svname from mountdata
Non-portable
Supports conf. Backups

Configuration: after

Driven by mountconf
OSD mounts device internally
mount.lustre parses mountdata
Svname come from mount.lustre
Portable
No local conf backups

LOV/OSC: before

Use OBD API
Too much interaction
Internals exposed
Llogs through the stack
Last-used-objids

LOD/OSP: after

Use OSD API
Much less interaction
Internals hidden
- Llogs in OSP
- Last-used-ids in OSP
- LOV EA in LOD
- Should be easier
MDS initiates destroy
- Up on commit

LOV EA handling

Hard to follow
Clients initiates destroy
Not very robust
Races

Quota: before

Relies on Linux
Enforcement in VFS
Accounting in diskfs
Part of MDD/obdfilter

Quota: after

Mostly OS-neutral
Enforcement in lquota
- Accounting in diskfs
Few calls from OSD

MGS: before

Uses LVFS
Driven by mountdata

MGS: after

Uses OSD API
Driven by mount opti

Comment by Andreas Dilger [ 14/Dec/11 ]

The mdd-survey test result for OSD-API patch Di did in the week-end.

Tests were run two times.

The external journal being used is 8G ssd external journal.

1. the unlink performance is indeed better with osd-patch.

2. For setxattr, I indeed saw the performance drop with 1024 threads (with
OSD-patch), and it happened every time, and should be repeatable. It is better
to figure out why. It is possible that this is caused by extra transaction
credits being requested for setxattr, which is being investigated in LU-910.

Comment by Hongchao Zhang [ 16/Dec/11 ]

the updated patch has been pushed to Gerrit, the changes are mainly related to "mdd_close".

Comment by Hongchao Zhang [ 19/Dec/11 ]

there are two failed tests in Maloo, one is test_101 in recovery-small, the other is test_85b in replay-single

1, test_101 in recovery-small,
the detailed failed message is as follows,

https://maloo.whamcloud.com/test_sets/58a80f0e-2926-11e1-b25e-5254004bbbd3
...
Error: 'version must match'
Failure Rate: 24.00% of last 100 executions [all branches]

client-23vm4: subsystem_debug=0xffb7e3ff
client-23vm4: debug_mb=32
Started lustre-OST0005
Starting ost7: /dev/lvm-OSS/P6 /mnt/ost7
client-23vm4: debug=-1
client-23vm4: subsystem_debug=0xffb7e3ff
client-23vm4: debug_mb=32
Started lustre-OST0006
recovery-small test_101: @@@@@@ FAIL: version must match
...

by searching the results in maloo, this issue was ever shown up in
https://maloo.whamcloud.com/test_sets/c8b23804-26df-11e1-87bf-5254004bbbd3

it is related to imperative recovery, and the ticket ~~LU-939~~ is created to track it

2, test_85b in replay-single
the purpose of this test is verifying whether the unused locks are canceled after recovery,
in this failed test, there was even no unused lock before recovery, then it failed
...
== replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) == 15:41:26 (1324165286)
before recovery: unused locks count = 0
Failing ost1 on node client-23vm4
+ pm -h powerman --off client-23vm4
Command completed successfully
affected facets: ost1,ost2,ost3,ost4,ost5,ost6,ost7
+ pm -h powerman --on client-23vm4
Command completed successfully
Failover ost1 to client-23vm4
Failover ost2 to client-23vm4
Failover ost3 to client-23vm4
Failover ost4 to client-23vm4
Failover ost5 to client-23vm4
Failover ost6 to client-23vm4
Failover ost7 to client-23vm4
15:41:50 (1324165310) waiting for client-23vm4 network 900 secs ...
waiting ping -c 1 -w 3 client-23vm4, 895 secs left ...
waiting ping -c 1 -w 3 client-23vm4, 890 secs left ...
waiting ping -c 1 -w 3 client-23vm4, 885 secs left ...
waiting ping -c 1 -w 3 client-23vm4, 880 secs left ...
15:42:20 (1324165340) network interface is UP
Starting ost1: /dev/lvm-OSS/P0 /mnt/ost1
...
after recovery: unused locks count = 0
replay-single test_85b: @@@@@@ FAIL: unused locks are not canceled
...

there is a bug in the test script, which can not make sure there are unused locks before recovery.

the updated patch which fixed the bug in test_85b of replay-single is pushed to Gerrit