https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1687 Lustre <p>The sanity-quota test_32 failed in my testing.<br/> After modules were unloaded and reloaded, the mds was left with an empty /proc/fs/lustre/. No file or subdirs under that directories. I had to reboot the mds and the clients to get them back. There were no error messages on service337 (mds) about problems of unloading and loading modules.</p> <p>The problem is reproducible. Fortunately, we do not need to perform this sequence of operation on mds often.</p> <a href="https://github.com/jlan/lustre-nas/tree/nas-2.1.2">https://github.com/jlan/lustre-nas/tree/nas-2.1.2</a><br/> sanity-quota test_32<br/> mds: service337<br/> oss1: service261<br/> oss2: service262<br/> client: service331, service332 LU-1687

Unloading lustre modules and reloading again leaves MDS with an empty /proc/fs/lustre

Bug Minor Resolved Not a Bug Lai Siyao Jay Lan Fri, 27 Jul 2012 20:17:54 +0000 Fri, 19 Apr 2019 00:14:36 +0000 Mon, 6 Aug 2012 13:47:03 +0000 Lustre 2.1.2 0 1 <p>Lai</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I can't reproduce in my setup, I'll look more into debuglog and code to find the cause.</p> <p>Fortunately I am still able to reproduce. Let me know what I can do to help debug this problem.</p> <p>BTW, the source of our lustre source can be git cloned from<br/> <a href="https://github.com/jlan/lustre-nas/commits/nas-2.1.2" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jlan/lustre-nas/commits/nas-2.1.2</a></p> <p>I can't build lustre against your git code because LUSTRE_KERNEL_VERSION is undefined. Git log shows it's removed in commit c2751b31e55518d1791cd5b87adc842f4fbbee83, could you help verify it? And if the code can be built on your system, could you output /proc/fs/lustre/version?</p> <p>I checked again, the commit c2751b3 is in nas-2.1.2 branch. It was not removed or reversed.</p> <p>service337 ~ # cat /proc/fs/lustre/version<br/> lustre: 2.1.2<br/> kernel: 2.6.32-220.4.1.el6.20120607.x86_64.lustre212<br/> build: 2nasS_ofed154<br/> service337 ~ #</p> <p>I built the 2.6.32-220.4.1.el6 kernel with kernel_patches for el6 from the 2.1.2 branch. Your kernel can be named different. Also, I used 1.5.4.1 ofa_kernel modules. </p> <p>I hereby attach two scripts file I used for my build: nas-config.sh.rhel62.212 and nas-make.sh.rhel62.212.<br/> They are for your reference. You should adjust for your target system.</p> <p>Yes, I can compile with your script, but this test still passes on my system. After this test fails, could you successfully mount lustre on MDS?</p> <p>I am able to reproduce "good" and "bad" cases without running sanity-quota test_32.</p> <p>After a lustre system has been set up, the "good" case operation sequence will be:</p> <ol> <li>#Start to tear down mds<br/> #1 lustre_rmmod<br/> #2 umount /mnt/mds1</li> <li>#Now to recover mds<br/> #1 modprobe lustre<br/> #2 mount -t lustre -o errors=panic,acl,noextents /dev/sdb1 /mnt/mds1</li> </ol> <p>The filesystem will be in good shape and usable.</p> <p>If I 'umount /mnt/mds1' before run 'lustre_rmmod', the the empty /proc/fs/lustre problem will happen after I do the same mds recovery opearations. The mount command will return and 'mount' will show mds1 mounted. However, the filesystem is not usable.</p> <p>Can you try the operation sequence and let me know if you can reproduce the problem?</p> <p>No, I can't reproduce here. BTW could you explain why you need do mds recovery? IMO it's shutdown and started up normally in your case.</p> <p>Ah, I meant to say restart. </p> <p>Somehow the restart after shutdown does not behave the same way as the initial start. I will do more debugging tomorrow.</p> <p>I put in printk to lprocfs_seq_create():</p> <p>int lprocfs_seq_create(cfs_proc_dir_entry_t *parent, char *name, mode_t mode,<br/> struct file_operations *seq_fops, void *data)<br/> {<br/> struct proc_dir_entry *entry;<br/> ENTRY;</p> <p> LPROCFS_WRITE_ENTRY();<br/> entry = create_proc_entry(name, mode, parent);<br/> if (entry) </p> { entry-&gt;proc_fops = seq_fops; entry-&gt;data = data; } <p> LPROCFS_WRITE_EXIT();</p> <p> if (entry == NULL) </p> { printk("lprocfs_seq_create: failed to create %s\n", name); RETURN(-ENOMEM); } <p> else<br/> printk("lprocfs_seq_create: successfully created %s\n", name);</p> <p> RETURN(0);<br/> }</p> <p>And the syslog showed:<br/> Aug 2 15:42:32 kern:info:service337 Lustre: Lustre: Build Version: 2.1nasS_ofed154<br/> Aug 2 15:42:32 kern:warning:service337 class_procfs_init: registering /proc/fs/lustre<br/> Aug 2 15:42:32 kern:warning:service337 lprocfs_seq_create: successfully created devices<br/> Aug 2 15:42:32 kern:warning:service337 class_procfs_init: successfully created lustre/devices</p> <p>So the lprocfs_seq_create() thought the /proc/fs/devices was created successfully. Yet, `ls /proc/fs/lustre` returned empty. This is weird.</p> <p>I will continue look into this.</p> <p>Oh, in addition to the changes to lproc_seq_create(), I also made changes to class_procfs_init(). Without this, the syslog output I quoted above did not make sense.</p> <p>diff -<del>git a/lustre/obdclass/linux/linux-module.c b/lustre/obdclass/linux/linux</del><br/> index 06cda1f..05b0390 100644<br/> &#8212; a/lustre/obdclass/linux/linux-module.c<br/> +++ b/lustre/obdclass/linux/linux-module.c<br/> @@ -421,13 +421,17 @@ int class_procfs_init(void)<br/> int rc;<br/> ENTRY;</p> <p>+ printk("class_procfs_init: registering /proc/fs/lustre\n");<br/> obd_sysctl_init();<br/> proc_lustre_root = lprocfs_register("fs/lustre", NULL,<br/> lprocfs_base, NULL);<br/> rc = lprocfs_seq_create(proc_lustre_root, "devices", 0444,<br/> &amp;obd_device_list_fops, NULL);</p> <ul class="alternate" type="square"> <li>if (rc)<br/> + if (rc) { + printk("class_procfs_init: Failed to add lustre/devices, rc=%d\ CERROR("error adding /proc/fs/lustre/devices file\n"); + } <p> else<br/> + printk("class_procfs_init: successfully created lustre/devices\<br/> #else<br/> ENTRY;<br/> #endif</p></li> </ul> <p>In the test case, when /proc/fs/lustre appeared to have been removed, it actually not.</p> <p>service337 /proc/fs # ls<br/> fscache jbd2 nfs nfsd nfsfs<br/> service337 /proc/fs # ls -lid lustre<br/> 4026532497 dr-xr-xr-x 17 root root 0 Aug 3 16:30 lustre</p> <p>You need to use 'ls -lid' to see it. So when next time we restart mds, another /proc/fs/lustre was created (with a different inode number.) All other inodes were created succssfully under the new /proc/fs/lustre. Unfortunately the system can not see it. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>What caused the original /proc/fs/lustre to hang around? From lustre's perspective, it was completed:<br/> Aug 3 16:40:27 kern:warning:service337 lprocfs_remove: removing lustre, parent=fs<br/> Aug 3 16:40:27 kern:warning:service337 about to remove lustre<br/> Aug 3 16:40:27 kern:warning:service337 removing devices from lustre<br/> Aug 3 16:40:27 kern:warning:service337 removing health_check from lustre<br/> Aug 3 16:40:27 kern:warning:service337 removing pinger from lustre<br/> Aug 3 16:40:27 kern:warning:service337 removing version from lustre<br/> Aug 3 16:40:27 kern:warning:service337 removing lustre from fs</p> <p>Systemtap into kernel showed:<br/> remove_proc_entry: lustre/devices<br/> free_proc_entry: devices, inode=4026532501<br/> remove_proc_entry: lustre/health_check<br/> free_proc_entry: health_check, inode=4026532500<br/> remove_proc_entry: lustre/pinger<br/> free_proc_entry: pinger, inode=4026532499<br/> remove_proc_entry: lustre/version<br/> remove_proc_entry: fs/lustre</p> <p>So, /proc/fs/lustre/version was not removed for some reason. Function remove_proc_entry() did not call free_proc_entry() in the case of lustre/version. Consequently, /proc/lustre can not be removed I suppose.</p> <p>service337 /proc/fs # ls -lid lustre/version<br/> 4026532498 <del>r</del><del>r</del><del>r</del>- 1 root root 0 Aug 3 16:30 lustre/version</p> <p>Why? I will investigate more...</p> <p>service337 /proc/fs # fuser lustre/version<br/> lustre/version: 4937<br/> service337 /proc/fs # ps -ef |grep 4937<br/> root 4937 4933 0 Aug03 ? 00:00:01 perl /var/lib/pcp/pmdas/lustre/pmdalustre.pl<br/> root 28872 28473 0 10:35 pts/0 00:00:00 grep 4937<br/> service337 /proc/fs #</p> <p>Please close this ticket. The problem was caused by /proc/fs/lustre/version being used by a NASA admin script.</p> <p>ok thanks Jay!</p> Related Development Rank 1|hzvfmv: Rank (Obsolete) 6179 Severity