[LU-1812] 3.6/FC18 Server Patches

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0, Lustre 2.5.0
• Fix Version/s: Lustre 2.4.0, Lustre 2.5.0
• Type: New Feature
• Priority: Major
• Reporter: Jodi Levi (Inactive)
• Assignee: Lai Siyao
• Resolution: Fixed

Issue Links

• is duplicated by: LU-2641 ldiskfs: Updated support for SLES 11 ...
• is related to: LU-2642 lustre: Updated support for SLES11 SP2
• is related to: LU-2686 Kernel update for 3.7.2-201.fc18
• is related to: LU-3043 server build for sles11sp2 is broken
• is related to: LU-3337 enable server builds for sles11sp2
• is related to: LUDOC-82 3.6/FC18 Server Doc Changes

Comments

[Lai Siyao]

This work can be divided into several parts:

• finish patchless server (LU-20) or port remaining kernel patches (8 patches) to FC17.
• port ldiskfs patches (40 patches) to support FC17 ext4.
• port server code to FC17, and this implies client has support FC17 already.

[Andreas Dilger]

There are patches for SLES11 SP2 (3.0.38) at http://gauss.credativ.com/~mne/lustre/patchsets/, which may be a good starting point for the 3.5.2 ldiskfs support.

[Peter Jones]

Well, they will need to sign off on their usage for us to be able to land them so we should not look at that code unless they push it into gerrit. I have been in touch with them about this but have not heard back.

[James A Simmons]

I'm willing to pick this up. Now I don't have a test bed with SP2 images but I can use a SP2 build box with extra disks. Can you validate ldisk without a lustre file system?

[Andreas Dilger]

It should be possible to mount and use ldiskfs as a local filesystem, though we never test it that way so there may be some unforeseen bugs.

It would also be possible to build the lustre code against the newer kernels before ldiskfs is available by disabling the ldiskfs code and instead building against ZFS.

[James A Simmons]

First draft patch at http://review.whamcloud.com/#change,4649

[chas williams - CONTRACTOR]

i have a set of patches for ldiskfs for SLES11SP2 that i produced independent of the above (since i didn't know about it). any interest? i haven't tested yet since i am still working toward getting a server built.

[James A Simmons]

My patch is for basic kernel support not ldiskfs. Yes I'm interested and I will test it. I like to base it against Chris work for LU-2473.

[James A Simmons]

Besides the current out standing patches we have some work left for the osd-ldiskfs layer. The first problem seems to be a bit easier to tackle. The api for fsync has changed between kernel versions with it excepting 2, 3 and 4 args. The 2 and 3 argument version seems to be easy to handle but the 4 arugment requires the start and end loff_t for the file. So we have

static int osd_object_sync(const struct lu_env *env, struct dt_object *dt)
{
...

mutex_lock(&inode->i_mutex);
#ifdef HAVE_FILE_FSYNC_4ARGS
rc = file->f_op->fsync(file, start, end, 0);
#elif defined(HAVE_FILE_FSYNC_2ARGS)
rc = file->f_op->fsync(file, 0);
#else
rc = file->f_op->fsync(file, dentry, 0);
#endif
mutex_unlock(&inode->i_mutex);
RETURN(rc);
}

The question is where do we get the start and end using the osd api?

[James A Simmons]

Now the second issue is the quota api is very very different in newer kernels. For example all the vfs_dq_* functions have been removed. Reintroducing them does not seem to be the right approach.

[chas williams - CONTRACTOR]

with regard to ->fsync() i think the right thing is to simply call vfs_fsync(). inside it does
vfs_fsync_range(file, 0, LLONG_MAX, datasync).

with regard to ldiskfs, i got an ldiskfs from sles11sp2 to pass small acceptance. the hardest part is that dentry has grown a new member d_sb which is used instead inode->i_sb in some of the calls. so it isnt enough to just set dentry->d_inode to inode before entering ldiskfs from the osd-ldiskfs layer.

[Andreas Dilger]

James, the Lustre network protocol already contains the ability to pass the start/end offset as part of the OST_SYNC RPC. This is passed down to ofd_sync(), but is lost in the call to dt_object_sync(). I think it makes sense to change the OSD API to include the start and end parameters down the stack, and just drop them for the older kernels that only support sync on the whole inode.

It seems the client-sync ll_fsync() code didn't wire this up correctly either, and are dropping the start and end parameters and passing 0 and OBD_OBJECT_EOF to cl_file_sync_range(). That should be fixed at the same time. It makes sense to create a separate sub-task for issues like this and the quota changes, so that they can get an appropriate audience (e.g. Niu and Johann for quota).

[Andreas Dilger]

James, seems that there may be a new patch for the quota renames as well: http://review.whamcloud.com/5119

[chas williams - CONTRACTOR]

this is one more issue with fsync(). kernel commit 02c24a82187d5a628c68edfe71ae60dc135cd178 moved the i_mutex into the fsync() handlers.

[James A Simmons]

Ugh!!! It appears duplicate work was done in LU-2642

[Jeff Mahoney]

I've uploaded my ldiskfs changes that bring it up to (at least 3.0) as http://review.whamcloud.com/5133

[Bob Glossman (Inactive)]

I see two competing sets of ldiskfs patches for sles11sp2. One from Jeff
http://review.whamcloud.com/5133
and one from Chas
http://review.whamcloud.com/#change,4972

With a bit of hand waving and careful selection of what other patches I use, I can get either one to compile. Do we have a decision yet about which one is better? Seems to need an ext4/ldiskfs expert to decide. Jeff's set is a bit bigger if that means anything.

[Chris Gearing (Inactive)]

To work on fc18 these changes need to have updates in at least

autodetect_target in funcs.h
check_options in build

When I run on fc18 I get this

++ sed -ne '/^BUILDID =/s/.*= *//p' lustre/META
+ BUILDID=g932548b
+ load_target
+ EXTRA_VERSION_save=
+ for patchesdir in '"$EXTERNAL_PATCHES"' '"$TOPDIR/lustre/lustre/kernel_patches"'
+ TARGET_FILE=/targets/.target
+ '[' -r /targets/.target ']'
+ for patchesdir in '"$EXTERNAL_PATCHES"' '"$TOPDIR/lustre/lustre/kernel_patches"'
+ TARGET_FILE=/var/lib/jenkins/workspace/lustre-reviews-fc18/arch/x86_64/build_type/client/distro/fc18/ib_stack/inkernel/BUILD/lustre/lustre/kernel_patches/targets/.target
+ '[' -r /var/lib/jenkins/workspace/lustre-reviews-fc18/arch/x86_64/build_type/client/distro/fc18/ib_stack/inkernel/BUILD/lustre/lustre/kernel_patches/targets/.target ']'
+ '[' -r /var/lib/jenkins/workspace/lustre-reviews-fc18/arch/x86_64/build_type/client/distro/fc18/ib_stack/inkernel/BUILD/lustre/lustre/kernel_patches/targets/.target ']'
+ fatal 1 'Target  was not found.'
+ cleanup
+ true
+ error 'Target  was not found.'
+ local 'msg=Target  was not found.'
+ '[' -n 'Target  was not found.' ']'
+ echo -e '\nlbuild: Target  was not found.'

lbuild: Target  was not found.
+ exit 1
+ run_exit_traps
+ local i num_items=1
++ seq 0 -1 0
+ for i in '$(seq $((num_items-1)) -1 0)'
+ '[' -z 'kill -INT -22239 || true' ']'
+ eval kill -INT -22239 '||' true
++ kill -INT -22239
../build/exit_traps.sh: line 59: kill: (-22239) - No such process
++ true
+ rc=1
+ '[' 1 '!=' 0 ']'
+ echo 'Build failed'
Build failed
+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE

I will try and get a builder running so you can see the issues, this may or may be be easy because for logistical reasons of not stopping other builds.

[Jeff Mahoney]

I also have ldiskfs patches that work up until the 3.7 kernel (openSUSE 12.3 – had some free time this weekend).

At some point, dropping most of the inode_map_pages infrastructure in favor of the in-ext4 implementation should be discussed. The open coded stuff is falling behind (WRT bigalloc in particular, which isn't explicitly disallowed on ldiskfs) and the only reason not to use it is that the map of created blocks isn't passed back from the in-ext4 version. It turns out that nothing actually uses the map of created blocks or the optional_mutex argument to fsfilt_operations->fs_map_inode_pages anyway, so the in-ext4 version should be sufficient. Unless there's another user of that API out there that's out of the tree.

3.8 drops the kernel_thread export entirely. kthread_run replaces it. That API has been around forever, so it should be a fairly straightforward drop-in, with the exception that the naming of the thread in cfs_daemonize moves to cfs_create_thread. That's easy enough but took a little bit of wiggling to make it work on Darwin (untested of course, but looks sane). I'd like someone who knows the Lustre code ins-and-outs a bit more than I do to review the usage of cfs_daemonize_ctxt vs cfs_block_allsigs. In some cases the signal blocking happens only on linux and, in others, universally. The big difference between kernel_thread and kthread_run is that kthread_run already starts from a clean context, so daemonize isn't needed – unshare_fs_struct may be in some cases. Signals are ignored instead of blocked, so the signal handling logic needs to be looked at as well.

Chris, can you pull in your jenkins changes into my ldiskfs work? I've only done local builds against SLE11 SP2 so I don't have any of that stuff set up.

[James A Simmons]

Bob I hope to have Jeff and Chris merge their work. It looks like Jeff has more fixes so it will be more favored Great work Jeff on support up to 3.7. Once this is sorted away I was hoping to shrink the number of patches. If you look at ldiskfs support for SLES11 SP1 it reuses all the rhel6 patches plug one more patch specifically for SuSE. The reason for this approach was to take advantage of any fixes that go into ldiskfs for rhel6 with no extra work. I hope we can do the same for the new ldiskfs work. Also keep a eye on patch http://review.whamcloud.com/#change,5004 since it will impact this ldiskfs.

[Bob Glossman (Inactive)]

One possibly significant difference I see between the two is Chas' set fixes autoconf to #define HAVE_LDISKFS_PDO but Jeffs' set doesn't (unless I missed something). Both needed some refresh on the pdirops patch before they would build right with current sp2 update kernel version 3.0.51-0.7.9

[James A Simmons]

While working on the kernel patches for SLES11 SP2 I noticed two stray patches, bio_add_page.patch and quota-support-64-bit-quota-format.patch. They don't seem to be used by anyone. Any ideas on those?

[chas williams - CONTRACTOR]

yes, just saw the pdir op problem today after updating my system. i already have a fix for that.

about about ext4's fsync() now taking inode->i_mutex? what is the right behavior? not taking the i_mutex in the lustre code or reverting the i_mutex lock from fsync() when you build the ldiskfs module?

[Bob Glossman (Inactive)]

Chas, one of Jeff's patches outside the ldiskfs set seems to deal with the i_mutex/fsync. see http://review.whamcloud.com/#change,5115

Don't know if more changes are needed or not.

[Bob Glossman (Inactive)]

Just FYI my recipe to get things building on sles11sp2 pulled in a lot of Jeff's other patches, including http://review.whamcloud.com/#change,5115

[Jeff Mahoney]

The expectation is that fs implementation of ->fsync takes i_mutex. When things are nested like they are in Lustre, it's less clear, but the code shows that it's always just a passthrough to fsync anyway. In any case, it's dependent on HAVE_FILE_FSYNC_4ARGS. The commit that added the ranges to ->fsync also pushed the locking down into the file system implementation.

There are only two explicit calls to ->fsync left: lustre_fsync, which is a pure passthrough and never had the lock anyway. osd_object_sync call only braces the call to ->fsync, so the proper locking is in ldiskfs already. So the fsync handling I have in my patches is correct.

[Jeff Mahoney]

Bob, in addition to the ldiskfs patches, the following were required for SLES11SP2:
LU-2642 ldiskfs: ->dirty_inode takes two args
LU-2642 osd-ldiskfs: handle 4-arg fsync properly
LU-2642 ldiskfs: handle ext4_ext_cache without ec_type member
LU-2642 ldiskfs: ext4_free_blocks takes 6 arguments
LU-2642 osd-ldiskfs: use struct fs_disk_quota for quotas
LU-2642 llite: update quota APIs
LU-2642 osd-ldiskfs: set dentry->d_sb for xattr calls
LU-2642 osd-ldiskfs: Use __REQ_WRITE instead of BIO_RW.
LU-2642 mgs: fix cleanup of nonexistant MGSSELF_NAME proc file
LU-2642 mgs: fix missing error code return in mgs_fs_setup
LU-2642 osd-ldiskfs: Unlock new inodes before iput

[Christopher Morrone]

One possibly significant difference I see between the two is Chas' set fixes autoconf to #define HAVE_LDISKFS_PDO but Jeffs' set doesn't

FYI, HAVE_LDISKFS_PDO will disappear when the landing backlog is flushed.
http://review.whamcloud.com/5004

[Bob Glossman (Inactive)]

The recipe I used was:
http://review.whamcloud.com/#change,4859
http://review.whamcloud.com/#change,4860
http://review.whamcloud.com/#change,4861
http://review.whamcloud.com/#change,4863
http://review.whamcloud.com/#change,4966
http://review.whamcloud.com/#change,4970
http://review.whamcloud.com/#change,5001
http://review.whamcloud.com/#change,4991
http://review.whamcloud.com/#change,5115
http://review.whamcloud.com/#change,5118
http://review.whamcloud.com/#change,5119
http://review.whamcloud.com/#change,5121

Some of those are Jeff's, some aren't.

The following didn't seem to be absolutely necessary, but I put them in anyway:

http://review.whamcloud.com/#change,5120
http://review.whamcloud.com/#change,5122
http://review.whamcloud.com/#change,5124

On top of those I add the these for Jeff's ldiskfs:

http://review.whamcloud.com/#change,5113
http://review.whamcloud.com/#change,5133

or this one for Chas's ldiskfs:

http://review.whamcloud.com/#change,4972

[Bob Glossman (Inactive)]

Chris, thanks for the pointer to http://review.whamcloud.com/5004. I missed that one. I probably would have grabbed it too. Doesn't play nicely with http://review.whamcloud.com/#change,4972 in it's current form, but I could have worked around that.

[James A Simmons]

That is about the same combo of patches, we me currently testing with Chas ldiskfs patch. One other patch is missing is the lustre kernel patches in http://review.whamcloud.com/#change,4649
I just updated it after I got my build system to properly patch a SuSE kernel and tested it out. Please give it a try as well. Thanks

[Bob Glossman (Inactive)]

Thanks, James. I didn't forget about http://review.whamcloud.com/#change,4649. I left it out of my recipe for two reasons:

1) It only involved the kernel patch series and didn't seem critical in getting the ldiskfs/lvfs portion of the world to build.

2) Until your refresh just now it didn't play well with the latest sp2 kernel. I had to massage nearly every patch a bit for context in the available commit to get it to work. I will retry with the latest.

Having and using the kernel patch series and doing a kernel build is an important part of the building a sles11sp2 server process, I just didn't think that patch was quite ready yet.

[Bob Glossman (Inactive)]

The refresh of http://review.whamcloud.com/#change,4649 seems good for sles11sp2. Applied smoothly, kernel build in progress.

I'm not in a position to try out the fc18 stuff right now.

[Bob Glossman (Inactive)]

Kernel build finished up fine.
My complete recipe for sles11sp2 now looks like:

http://review.whamcloud.com/#change,4649
http://review.whamcloud.com/#change,4859
http://review.whamcloud.com/#change,4860
http://review.whamcloud.com/#change,4861
http://review.whamcloud.com/#change,4863
http://review.whamcloud.com/#change,4966
http://review.whamcloud.com/#change,4970
http://review.whamcloud.com/#change,5001
http://review.whamcloud.com/#change,4991
http://review.whamcloud.com/#change,5115
http://review.whamcloud.com/#change,5118
http://review.whamcloud.com/#change,5119
http://review.whamcloud.com/#change,5121
http://review.whamcloud.com/#change,5004
http://review.whamcloud.com/#change,5120
http://review.whamcloud.com/#change,5122
http://review.whamcloud.com/#change,5124
http://review.whamcloud.com/#change,5113
http://review.whamcloud.com/#change,5133

The ldiskfs patches in http://review.whamcloud.com/#change,5133 still need a bit of hand waving. Probably due for a refresh.

[Bob Glossman (Inactive)]

Seems to be something wrong with rpm building in sles11sp2.

'make' goes to completion and does build ldiskfs.
'make rpms' appears to finish, but no ldiskfs-* rpms get built. Don't see any ldiskfs modules in lustre-modules rpm.

[Bob Glossman (Inactive)]

suspect this is due to includes being in include/generated. There is a patch that fixed this in lustre.spec.in, but nothing was ever done in ldiskfs/lustre-ldiskfs.spec.in

[Bob Glossman (Inactive)]

My theory is looking good. If I take the '%{!?kversion: %global kversion...' line from lustre.spec.in and put it in ldiskfs/lustre-ldiskfs.spec.in, then I see lustre-ldiskfs-* rpms being built OK.

[James A Simmons]

Since fixes up are needed for ldiskfs rpms to build and the ldiskfs patches both need refreshing it would be a excellent time to merge all the work together.

[chas williams - CONTRACTOR]

yes bob, that is what i did to get the rpm's to build for ldiskfs. this should be a separate patch/fix from ldiskfs support i would think. this .spec should have been fixed at the same time as the lustre.spec

[chas williams - CONTRACTOR]

http://review.whamcloud.com/#change,5188

[Bob Glossman (Inactive)]

I see a recent change went into ldiskfs patches in master: http://review.whamcloud.com/#change,5179

Might a change like this also be needed in http://review.whamcloud.com/#change,5133 or http://review.whamcloud.com/#change,4972 ?

[chas williams - CONTRACTOR]

it seems likely. i updated http://review.whamcloud.com/#change,4972

[chas williams - CONTRACTOR]

patchset 9 broke some stuff that i copied into patchset 10. patchset11 of http://review.whamcloud.com/#change,4972 does build now.

[Bob Glossman (Inactive)]

Yes, I got caught trying to use patchset 10 for a few minutes. patchset 11 works much better.

[Jeff Mahoney]

What code base are you successfully building 4972 against? I fetched patchset 11 and the ldiskfs patch set doesn't apply to my SP2 tree's copy of ext4. Based on some of the context for changes to fs/ext4/super.c, which still has the "Fourth Extended Filesystem with extents" module description that was changed to remove the "with extents" in 2.6.29, it looks like this is against SLE11 GA's 2.6.27 kernel. Am I missing something?

[Bob Glossman (Inactive)]

I'm building OK against the 3.0.51 kernel I see in most recent update of sles11sp2. Not 2.6.<anything>.

[Jeff Mahoney]

Ah, I know what it is. My quilt configuration explicitly disallows fuzz and this patch set has fuzz when applied to the SP2 tree. It should be refreshed.

[chas williams - CONTRACTOR]

i think you need to allow fuzz. if another kernel comes out, there might be slight differences. i am not sure it makes sense to support older kernels than 'the latest release' from sle11sp2 (unless whamcloud is going to release a specific lustre kernel for suse).

[James A Simmons]

Unfortunately cray tends to use one kernel for the life of their products so we need to make sure the patches work over a range of kernels of version 3.0. Later kernels like 3.1 are a different story.

[chas williams - CONTRACTOR]

that might be difficult since ext4_isize() in the p_dirop patch changes signature and in one set of 3.0 kernels you need to patch two instances and in another you only need to patch one.

[James A Simmons]

Which version did this occur at? Currently the kernel version in use is 3.0.34-0.7

[Jeff Mahoney]

I'm using a snapshot of our git repo from December, 3.0.51-15. The thing is the context hasn't changed for the bits I'm talking about – it changed in the 2.6.29 days. I get Chas's point about fuzz – it was just unexpected since my quilt setup is for our kernel repo where everything must apply with 0 fuzz.

BTW, what's the standard procedure for updating other people's patch sets here? I noticed James updated a few of my patches responding to review, and I added some review notes to Chas's patchset 11 in 4972. Should I update it and submit? I ask because in the cases of the patches I suggested be switched over to using backports of the mainline commits, I already have them exported and backported.

[James A Simmons]

Okay so ext4_isize is fine at my version. Makes sense since I haven't seen any issues with building the current patch set. As for updating patches well that depends on the people you are working with. For me I have no problem with someone updating any of the patches I have worked on Can't speak for Chas tho I did update his ldiskfs patch but only to rebase on top of LU-2473. The goal here is to merge all the ldiskfs work everyone has done. Chas should we label the patch with LU-2641 instead of LU-1812 since it is ldiskfs specific for SLES11 SP2?

[chas williams - CONTRACTOR]

I don't really care one way or the other about owner/authorship of these kinds of patches. There isn't any massive intellectual property associated with them. If you have updates to a patch, resubmit it. I will address Jeff's comments this weekend if he doesn't get to it before then.

Somewhere between 3.0.34 and 3.0.51 ext4 got an extra ext4_isize(). My sources tell me there was an update or two between those. Never upgrading the kernel is a strange idea. What happens when there is an exploit?

[Peter Jones]

Our usual practice is to supported the latest released kernel update for the distros we support. Cray create their own releases so can make any adaptations that they need to make for older versions.

[Jeff Mahoney]

3.0.46 added the second ext4_isize as a backport of commit b71fc079 (ext4: fix fdatasync() for files with only i_size changes).

I've updated the SP2 ldiskfs patchset, though it has a new dependency on not using LDISKFS_FS(sb)->s_qf_inums since that part of ext4-quota-first-class.patch wasn't incorporated into mainline. Since the fs fails the mount if the inodes aren't there, we can just grab them out of the superblock like ext4 does itself.

After I get some testing in, I'll push it to gerrit.

[Jeff Mahoney]

I've updated and pushed the patchset. My current recipe looks like:
http://review.whamcloud.com/5264 – LU-1812 ldiskfs: don't use sb->s_qf_inums
http://review.whamcloud.com/5121 – LU-1812 osd-ldiskfs: Use __REQ_WRITE instead of BIO_RW.
http://review.whamcloud.com/5122 – LU-2729 mgs: fix cleanup of nonexistant MGSSELF_NAME proc file
http://review.whamcloud.com/5115 – LU-1812 osd-ldiskfs: handle 4-arg fsync properly
http://review.whamcloud.com/5188 – LU-1812 ldiskfs: detect generated utsrelease.h
http://review.whamcloud.com/4966 – LU-812 ldiskfs: super_operations->dirty_inode now takes a flag
http://review.whamcloud.com/4970 – LU-1812 ext4: ec_type has been dropped from ext4_ext_cache
http://review.whamcloud.com/5001 – LU-1812 fsfilt: ext_pblock renamed to ext4_ext_pblock
http://review.whamcloud.com/5118 – LU-1812 osd-ldiskfs: use struct fs_disk_quota for quotas
http://review.whamcloud.com/5119 – LU-1812 llite: update quota APIs
http://review.whamcloud.com/5120 – LU-1812 osd-ldiskfs: oti_obj_dentry needs d_sb set
http://review.whamcloud.com/4803 – LU-2473 ldiskfs: Reorganize ldiskfs kernel patches
http://review.whamcloud.com/4972 – LU-1812 ldiskfs: Add kernel patch series for SLES11SP2

[Bob Glossman (Inactive)]

Jeff,
Thanks for the latest. Trying out the new recipe now.

Just to help keep things sane could you please mark http://review.whamcloud.com/#change,5133 Abandoned since all the new work is now happening in http://review.whamcloud.com/4972 ?

Would hate for anybody to waste time looking at or picking up obsolete stuff.

[Jeff Mahoney]

Bob, since our patch series differ, I took a minute to reconcile the differences between them. You can drop http://review.whamcloud.com/5124. That was a result of a bug introduced in my ldiskfs patches. It resulted in a working SLES but applying that patch on a RHEL system would make it unhappy. http://review.whamcloud.com/5113 and http://review.whamcloud.com/#change,5133 can be dropped as well. I've compared and integrated the differences between those and 4972 and pushed the result to 4972.

These have been pulled into master in the past few weeks:
http://review.whamcloud.com/5004
http://review.whamcloud.com/4649
http://review.whamcloud.com/4859
http://review.whamcloud.com/4860
http://review.whamcloud.com/4861

http://review.whamcloud.com/4991 looks like http://review.whamcloud.com/5001 is superseding it, which is why I took it in my series.

So that just leaves http://review.whamcloud.com/4649, which didn't conflict with any of my patches at all.

[Jeff Mahoney]

Ok, all of the patches I submitted initially that aren't in that list have been marked as abandoned now.

[Bob Glossman (Inactive)]

The new recipe isn't working for me. Builds go to completion and look fine, but mounts of ldiskfs aren't working. In particular the mount done in mkfs.lustre to write out config inodes fails.

client cmd:

# mkfs.lustre --mgs --mdt --index=0 --fsname=lustre /dev/sdb

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters:

checking for existing Lustre data: not found
device size = 1024MB
formatting backing filesystem ldiskfs on /dev/sdb
	target name  lustre:MDT0000
	4k blocks     262144
	options        -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000  -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/sdb 262144
mkfs.lustre: Unable to mount /dev/sdb: No such process

mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with 3 (No such process)

strace of the cmd shows that it's the mount() syscall of ldiskfs that fails:

  . . .
mkdir("/tmp/mntMBwvlZ", 0700)           = 0
mount("/dev/sdb", "/tmp/mntMBwvlZ", "ldiskfs", 0, "user_xattr,errors=remount-ro") = -1 ESRCH (No such process)
write(2, "mkfs.lustre: Unable to mount /de"..., 55mkfs.lustre: Unable to mount /dev/sdb: No such process
) = 55
  . . .

/var/log/messages says:

Feb  4 15:29:26 suse2 kernel: [19073.692340] LDISKFS-fs warning (device sdb): ldiskfs_enable_quotas:4939: Failed to enable quota (type=0) tracking. Please run e2fsck to fix.
Feb  4 15:29:26 suse2 kernel: [19073.692350] LDISKFS-fs (sdb): mount failed
Feb  4 15:34:44 suse2 kernel: [19390.782444] LDISKFS-fs warning (device sdb): ldiskfs_enable_quotas:4939: Failed to enable quota (type=0) tracking. Please run e2fsck to fix.
Feb  4 15:34:44 suse2 kernel: [19390.782456] LDISKFS-fs (sdb): mount failed

[Jeff Mahoney]

I ran into that too but I hadn't used a working 2.4 system yet. modprobe quota_v2 fixes it. Since I now know it's unexpected, I'll look into fixing it.

[Bob Glossman (Inactive)]

modprobe quota_v2 works for me, too.
Seems like there's some kind of dependency failure there somewhere.
Thanks, Jeff.

[Jeff Mahoney]

It looks like it's because QFMT_VFS_V1 doesn't have an entry in the module list that the dquot uses to request a module be loaded. The SLES kernel has CONFIG_QFMT_V1=m. The RHEL kernel has CONFIG_QFMT_V1=y, so it's not hitting the problem. I've written up a patch and sent it to linux-fsdevel. It'll land in a SLES kernel update as soon as it's accepted there. In the interim, we'll need to manually load quota_v2.ko for testing.

[chas williams - CONTRACTOR]

yeah, i dropped the ball on that. i mentioned the problem on http://jira.whamcloud.com/browse/LU-1842
it would be nice if this got fixed.

[James A Simmons]

Jeff can you point me to the fix for the quota. I can include that in the 4649 patch.

[Jeff Mahoney]

This is the fix: http://www.spinics.net/lists/linux-fsdevel/msg62124.html but it's not something that can be included except in lustre/kernel_patches. I expect the patch itself to land in -stable updates soon and distro release kernels from other vendors soon thereafter. Of course, it could be worked around by doing an explicit request_module("quota_v2") but that's just ugly.

[Bob Glossman (Inactive)]

I note that many of the recommended configs in lustre/kernel_patches/kernel_configs have CONFIG_QFMT_V2=y. If in the future we have published configs for sles11 and pre-built kernel rpms with this setting the issue may become less important.

[Jeff Mahoney]

It looks like Ted beat me to the punch by a few days. http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=c3ad83d9efdfe6a86efd44945a781f00c879b7b4 Funny how a bug that's been there since 2.6.33 drew attention from both of us within days of each other.

I've checked it into the SLE11 SP2 tree, the openSUSE 12.3 tree, and our master branch so it should roll out in the next SLE11 SP2 update. BTW, we publish our kernel git repos publicly, so you can get it more quickly here: http://kernel.suse.com/

[James A Simmons]

Since you are pushing patches to your distro once I'm done testing your latest rounds of patches I will update my kernel side patch and then we could the patch upstream in time for the next release of SP2.

[Jeff Mahoney]

This is in reference to change 4649.

Neil Brown, the MD RAID maintainer, is in my team at SUSE. I've passed on a copy of that patch to have him take a look at the raid5 fix for mop.

A version of my dev_set_rdonly patch can be found here: http://kernel.suse.com/cgit/kernel-source/tree/patches.suse/block-add-dev_check_rdonly-and-friends-for-lustre-testing?h=SLE11-SP3 There are some small cosmetic differences between that one and the one I'm planning to use. I'm going to pr_warn instead of printk, add a message about clearing the read-only state, and remove an extra prototype in there that we don't need. By the time you read this, that might actually be fixed. It is already in my repository but our internal git server (which mirrors out to that one) isn't being cooperative right now.

[James A Simmons]

MUch nicer look Jeff. I will integrate that into my next patch set. Will this be integrated into SP2?

[James A Simmons]

Okay I had to do some rebasing with the latest code so I pushed the needed changes. Patch 5001 is needed before the patch for LU-2473 since it alters one of the core rhel5 ldiskfs patches. Give it a try

[James A Simmons]

It looks like the major shack up of the fsfilt layer is mostly done. The working combo; must apply in order; I'm testing is

LU-2158
http://review.whamcloud.com/#change,5246
http://review.whamcloud.com/#change,5250
http://review.whamcloud.com/#change,5331

LU-1812
http://review.whamcloud.com/#change,4970
http://review.whamcloud.com/#change,4972
http://review.whamcloud.com/#change,4991
http://review.whamcloud.com/#change,5001
http://review.whamcloud.com/#change,5115
http://review.whamcloud.com/#change,5120
http://review.whamcloud.com/#change,5188

LU-2059
http://review.whamcloud.com/#change,5049

LU-2473
http://review.whamcloud.com/#change,4803

LU-2642
http://review.whamcloud.com/#change,5121

LU-2729
http://review.whamcloud.com/#change,5122

Also we have patches from LU-2760, LU-2761, LU-2762 but they conflict with my SLES11 SP1 platform. Will need to work on that later.

[Bob Glossman (Inactive)]

James,
Think you are missing this one. I use it in my recipe:

http://review.whamcloud.com/#change,4966

[James A Simmons]

Oops. I'm using that as well. With the fsfilt cleanup hopefully this will not be changing to much going forward. Bob have you been getting any results yet?

[Bob Glossman (Inactive)]

Yes, I have been getting good results with my current recipe on sles11sp2. I haven't started using changes 5246, 5250, 5336 yet. I will try adding them & see what happens.

Have been trying to work with 5279,5280,5281,5282,5283. Haven't included them for real in my current recipe yet because 5282, the one relocating dynlocks, breaks things in its current form. Have been waiting for that to refresh and settle out. Didn't realize some of them had problems in sles11sp1.

[Jeff Mahoney]

I have an updated version of the dynlocks patch I'll upload in a minute.

[Bob Glossman (Inactive)]

I see the refresh of the dynlocks patch. Seems smoother than before. However it does generate an alarming looking build warning:

WARNING: /home/bogl/lustre-release/lustre/obdclass/obdclass.o(.text+0x30d7): Section mismatch in reference from the function class_dynlock_init() to the function .init.text:dynlock_cache_init()
The function class_dynlock_init() references
the function __init dynlock_cache_init().
This is often because class_dynlock_init lacks a __init 
annotation or the annotation of dynlock_cache_init is wrong.

[Bob Glossman (Inactive)]

Probably should have put that last comment on LU-2757

[Jeff Mahoney]

Ah, it's because class_dynlock_init wasn't marked __init and dynlock_cache_init was. Non-init functions aren't allowed to call __init functions. In this case it wouldn't have had any runtime impact other than a bit of memory lost since class_dynlock_init is also only called from an __init function. I've fixed it so class_dynlock_init is also __init in Patchset 5.

[Andreas Dilger]

Just a comment here - I've been completely ignoring all of these patches until there is some indication that they are not being refreshed continuously, and are in a shape that they are ready for landing. In particular that the patches are working on both SLES11 SP2 and RHEL6.

If I'm incorrect in waiting to start my inspections, or if there are patches early in the series that are stable and can be landed while later patches are being refreshed frequently, please let me know.

[James A Simmons]

I would say the patches ready for inspection and landing are:

http://review.whamcloud.com/#change,4966 - you asked about passing in the flag so if it needs to be refresh that is okay.
http://review.whamcloud.com/#change,4970
http://review.whamcloud.com/#change,5115
http://review.whamcloud.com/#change,5120
http://review.whamcloud.com/#change,5121
http://review.whamcloud.com/#change,5188

The reset depend on other patches landing (LU-2473, LU-2059, and LU-2158) or need more work.

[Bob Glossman (Inactive)]

http://review.whamcloud.com/#change,5122 should also be mentioned. Looks like it's already inspected and just needs landing.

[Andreas Dilger]

Bob, except that patch is at the end of a series of patches, and cannot typically be landed until all of the previous ones are. If patches are independent of each other, they should be submitted from separate branches, otherwise they are subject to much longer delays for landing.

[James A Simmons]

Patch 5122 is only dependent on one patch (LU-2642) which is unrelated. The question is which is faster, refreshing the patch with no dependency and going through new review or waiting until the previous patch is merged?

[Bob Glossman (Inactive)]

Patchset 5 fixes the build warning for me.

Have added the following changes into my recipe with good success on sles11sp2:

5330, 5279, 5280, 5281, 5282.

Haven't been able to make http://review.whamcloud.com/#change,5283 apply nicely lately. Think it may need refresh due to overlapping changes in one of the rhel6 ldiskfs patches from recent http://review.whamcloud.com/#change,5001.

[Bob Glossman (Inactive)]

That last report may have been a tad optimistic. Builds complete OK, but functionality is not so good. Seeing failed mounts and crashes. May have more details later.

[James A Simmons]

Bob without patches 5330, 5279, 5280, 5281, 5282 it works fine correct?

[Bob Glossman (Inactive)]

yes, worked fine without.
Now after a few reboots it's working better with those added patches too. I think I may have had a mix of old and new bits impacting me before. Will try to be more careful with updates and installs.

[James A Simmons]

Andreas can you review

http://review.whamcloud.com/#change,4970
http://review.whamcloud.com/#change,5120

Both are ready for merger.

[Bob Glossman (Inactive)]

James, Jeff,
Is it time to turn on Andreas for review of the sles11sp2 ldiskfs patches? The only official dependency left is http://review.whamcloud.com/#change,4803, which has been reviewed & tested and looks ready to land.

http://review.whamcloud.com/#change,4972 has been stable for a week now. I've had good results using it on sles11sp2 so far.

Is much more change to come expected?

[James A Simmons]

I would say so, also a few other patches are ready as well.

Andreas can you review.

http://review.whamcloud.com/#change,4966
http://review.whamcloud.com/#change,4972
http://review.whamcloud.com/#change,4991
http://review.whamcloud.com/#change,5115

[James A Simmons]

Sorry had to finish untangling patch http://review.whamcloud.com/#change,4991 from all the dependencies it had. Now it can be inspected and merged on its own.

[James A Simmons]

The patches still need to get SLES11 SP2 working are:

http://review.whamcloud.com/#change,4966 # 2 arg ->dirty_inode; fixes build failure
http://review.whamcloud.com/#change,4991 # 6 arg ext4_free_blocks; fixes build failure
http://review.whamcloud.com/#change,5001 # ext4_ext_pblock rename; fixes build failure
http://review.whamcloud.com/#change,5115 # locking around ->fsync; fixes deadlock
http://review.whamcloud.com/#change,5120 # set ->d_sb; fixes crash in xattr operations
http://review.whamcloud.com/#change,5503 # fix typo in REQ_WRITE check

Of those 4966 needs more work. Patches 5115 and 5120 are in the queue to be merged.
Patches 4991,5001,5503 need to be inspected.

Andreas can you take a look at 4991, 5001, and 5503. Note I slimmed down patch 5001 since fsfilt will be cleaned up in patches for LU-2158 from me. Thanks.

[Bob Glossman (Inactive)]

http://review.whamcloud.com/#change,4649 still needs to land too. Has yet to pass tests.

[James A Simmons]

Can almost build. It fails to build because of mdt_fid2path in mdt_handler.c is not setting rc to some default value. Other than that it builds. Will need to create a patch for that. Is rc = o the right choice or is the bug larger than that?

As for 4649 yes it also needs to be updated and worked on. I was thinking about Jeff's patch for SP3 and dev_readonly. Instead of dev_readonly why not use the FAIL_MAKE_REQUEST functionality. Jeff also are you fine with my approach for tunable handling? Can we push that upstream for SP2?

[James A Simmons]

For a update where we are at. The following set of patches is needed to build and run lustre on SLES11 SP2

http://review.whamcloud.com/#change,4966 - LU-812
http://review.whamcloud.com/#change,5675 - LU-1199
http://review.whamcloud.com/#change,4991 - LU-1812
http://review.whamcloud.com/#change,5001
http://review.whamcloud.com/#change,5120
http://review.whamcloud.com/#change,5656 - LU-2929
http://review.whamcloud.com/#change,5503 - LU-2642
http://review.whamcloud.com/#change,5708 - LU-2748

http://review.whamcloud.com/#change,5709 - is a bug fix for ldiskfs on SP2
http://review.whamcloud.com/#change,4649 - optional and needs to be updated.

[Bob Glossman (Inactive)]

James,
Thanks for the update on the recipe. This more or less matches what I've been using the last few days. http://review.whamcloud.com/#change,5708 was new to me. http://review.whamcloud.com/#change,5503 is ready to land now, I think.

[James A Simmons]

The patches currently in gatekeeper waiting to land are

http://review.whamcloud.com/#change,4966
http://review.whamcloud.com/#change,5120
http://review.whamcloud.com/#change,5656
http://review.whamcloud.com/#change,5503

[Bob Glossman (Inactive)]

While they aren't absolutely required for sles11sp2 to work, I added the following to James' latest recipe without any problems:

http://review.whamcloud.com/#change,5279
http://review.whamcloud.com/#change,5280
http://review.whamcloud.com/#change,5281
http://review.whamcloud.com/#change,5282

[Jeff Mahoney]

http://review.whamcloud.com/5786 will be needed after the next SP2 update since some ext4 bug fixes were pulled in via my submission to -stable.

[Bob Glossman (Inactive)]

Jeff, Those changes in #5786 look fine to me but we will have to be careful not to land them until the next SP2 update is really available and shows up when anybody does 'zipper up'. Can you please let us know when that happens?

[James A Simmons]

Jeff have you looked at http://review.whamcloud.com/#change,4649 for how I handle the optimizations as module parameters as you suggested. It would be nice to have those upstream for SLES11.

[James A Simmons]

Time for a update for where we are at. Patches needed now are:

http://review.whamcloud.com/#change,5675 LU-1199
http://review.whamcloud.com/#change,4991 LU-1812
http://review.whamcloud.com/#change,5001
http://review.whamcloud.com/#change,5120

http://review.whamcloud.com/#change,4649 - finally passed
http://review.whamcloud.com/#change,5709 - LU-2926 bug fix for ldiskfs

[Bob Glossman (Inactive)]

Using James' most recent recipe, I'm seeing a build failure in mdt_handler.c. I can fix it on the fly with the following patch:

diff --git a/lustre/mdt/mdt_handler.c b/lustre/mdt/mdt_handler.c
index ba3f24d..97c56a5 100644
--- a/lustre/mdt/mdt_handler.c
+++ b/lustre/mdt/mdt_handler.c
@@ -5547,7 +5547,7 @@ static int mdt_path_current(struct mdt_thread_info *info,
 	char			*ptr;
 	int			reclen;
 	struct linkea_data	ldata = { 0 };
-	int			rc;
+	int			rc = 0;
 	ENTRY;
 
 	/* temp buffer for path element, the buffer will be finally freed

Think this was introduced by a recent commit that builds fine in Centos 6.x
Haven't yet filed a bug on it.

[Bob Glossman (Inactive)]

I should mention that I add http://review.whamcloud.com/#change,4804 to the recipe. While it's optional in sles11sp2 it includes changes to the ldiskfs patch series selection rules for all distros. Dosen't hurt to make sure those changes work right in sles11sp2 too.

[James A Simmons]

Ug the rc not set issue again. We really need to set the compiler flags on rhel so these issues can be avoid. Please post the JIRA ticket once you open it.

[Bob Glossman (Inactive)]

New bug is LU-3043. Have already linked it and pushed a 1-line fix.

[James A Simmons]

Looks like all the patches are good to go now. Andreas can you please inspect patches:

http://review.whamcloud.com/#change,5675
http://review.whamcloud.com/#change,4991
http://review.whamcloud.com/#change,5001
http://review.whamcloud.com/#change,4649
http://review.whamcloud.com/#change,5709

Thanks.

[James A Simmons]

Also http://review.whamcloud.com/#change,5120 has to reviewed again since it had to rebased to master due to changes in the code from other commits.

[Bob Glossman (Inactive)]

As Andreas is on vacation we may want to round up somebody else to do these reviews.

http://review.whamcloud.com/#change,5854 is also good to go & just needs another +review.

[Peter Jones]

Don't worry - I will be assigning alternate reviewers to cover for Andreas's vacation

[James A Simmons]

Time to do some reviews of the patches. Due to false negatives with Maloo and patch conflicts a few of the patches need to be updated and are ready for reviewed again.

http://review.whamcloud.com/#change,5675
http://review.whamcloud.com/#change,5001
http://review.whamcloud.com/#change,5120
http://review.whamcloud.com/#change,4649

[Bob Glossman (Inactive)]

The following are now ready for landing:

http://review.whamcloud.com/#change,5675
http://review.whamcloud.com/#change,4991
http://review.whamcloud.com/#change,5001

http://review.whamcloud.com/#change,4649 is still awaiting some review.

With any luck http://review.whamcloud.com/#change,5120 should be ready soon too if maloo stays healthy.

[Bob Glossman (Inactive)]

http://review.whamcloud.com/#change,4804 also looks good to go now.

[James A Simmons]

Down to one major patch - http://review.whamcloud.com/#change,5120. Looking at the Maloo results I see it failed due to LU-3057 and memory exhaustion for sanity65ic. Can some one with the power to restart the test please due for this patch.

We also have http://review.whamcloud.com/#change,4649 for server kernel optimization and support to simulate disk failure.

[Peter Jones]

The progress is certainly encouraging! I have retriggered the testing for the 5120 change set.

[James A Simmons]

Yahoo!!! Last patch needed for SLES11 SP2 to work server side landed. Only patch

http://review.whamcloud.com/#change,4649

is left for this ticket which is kernel tuning and the dev_read_only patch to simulate a failed disk.

[James A Simmons]

Only one patch left, http://review.whamcloud.com/#change,4649, which is for FC18 only. Peter can you link this ticket to LU-2686 and label this as a Lustre 2.5 item.

[Peter Jones]

ok

[James A Simmons]

This patch has landed to master. The work for kernel side support is now done so we can close this ticket.

[Peter Jones]

ok. Thanks James