https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4277 Lustre <p>The OFD statfs() handled can optionally add an OS_STATE_DEGRADED flag to the statfs reply, which the MDS uses to help decide which OSTs to allocate new file objects from. Unless all other OSTs are also degraded, offline, or full, the DEGRADED OSTs will be skipped for newly created files. </p> <p>This avoids the application waiting for slow writes because of the rebuild long after it has completed on other healthy OSTs. It also avoids the new writes from interfering with the OST rebuild process, so it is a double win.</p> <p>This was previously implemented as a /proc tunable suitable for mdadm or a hardware-RAID utility to set from userspace, but since ZFS RAID is in the kernel it should be possible to query this status directly from the kernel when the MDS statfs() arrives. </p> LU-4277

Integrate ZFS zpool resilver status with OFD OS_STATE_DEGRADED flag

Improvement Minor Resolved Fixed Nathaniel Clark Andreas Dilger zfs Wed, 20 Nov 2013 06:28:21 +0000 Tue, 20 Mar 2018 21:41:29 +0000 Tue, 6 Feb 2018 05:15:19 +0000 Lustre 2.11.0 Lustre 2.11.0 0 12 <p><a href="http://review.whamcloud.com/8378" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8378</a> is a basic patch to fix handling in the LOD code for DEGRADED and READONLY flags. It doesn't yet fix the osd-zfs code in udmu_objset_statfs() that should be setting the flags.</p> <p>Don or Brian,<br/> is there some straight forward way for osd-zfs at the DMU level to determine if ZFS is currently degraded and/or doing a drive resilver operation, or would this need some new API to access this info from the VDEV? If we had some mechanism to determine this easily, I think it would be straight forward for someone to add this functionality to Lustre. The alternative would be for ZED to set <tt>lctl set_param ofd.&lt;ost&gt;.degraded=1</tt> from userspace when it detects a degraded device and/or when the device is undergoing resilvering, and clearing it afterward.</p> <p>The degrade state is part of the vdev. Getting this info strictly through the spa interface would yield a ton of data (i.e. the entire config) and require nvlist parsing. A new API, like a spa_get_vdev_state(), to pull out the vdev state of the root vdev would be require to get at this state in a simple manner.</p> <p>We can easily set the state as it changes using a zedlet. We now have a state change event for all healthy&lt;--&gt;degraded vdev states that could be used to initiate a check of the pool state and post that state via lctl as you suggest above.</p> <p>Attached a zedlet, <b>statechange-lustre.sh</b>, that will propagate degraded state changes from zfs to Lustre.</p> <p>It would be good to get the script in the form of a patch against the fs/lustre-release repo so that it can be reviewed properly. Some general comments first, however:</p> <ul class="alternate" type="square"> <li>the license needs to be dual CDDL/GPL or possibly dual GPL/BSD so that there isn't any problem to package it with other GPL Lustre code (though technically GPL only affects distribution of binaries and not sources, it is better to avoid any uncertainty).</li> <li>there are some pathnames that hard-code Don's home directory, which are not suitable for use in a script that is deployed in production. The location of the <tt>zfs</tt>, <tt>zpool</tt>, <tt>lctl</tt> and <tt>grep</tt> commands should be found in <tt>$PATH</tt>.</li> <li>the ZFS pool GUID is hard-coded, or is that some sort of event GUID for the state change?</li> <li>the echos are fine for a demo, but not suitable for production use if they are too noisy. Maybe a non-issue if this script is only run rarely.</li> <li>should it actually be an error if the state change is not <tt>DEGRADED</tt> or <tt>ONLINE</tt>? I don't know what the impact of an error return from zedlet is, so maybe a non-issue?</li> <li>I don't know if it makes sense for set_degraded_state() to check the current state before setting the new state. This could introduce races, and I don't think it reduces them. I don't think there is much more overhead to always (re)set the state rather than to check it first and then set it.</li> </ul> <p>My thought is that the script would be installed as part of the <tt>ost-mount-zfs</tt> RPM in some directory (something like <tt>/etc/zfs/zed/zedlets.d/</tt> akin to <tt>/etc/modprobe.d</tt> or <tt>/etc/logrotate.d</tt>) that is a place to dump zedlets that will be run (at least the next time zed is started) and do not really need any kind of edit from the user to specify the Lustre targets. Then, it would get events from the kernel when a zpool becomes degraded and update <tt>lctl obdfilter.$target.degraded</tt> for targets in that zpool, and does nothing for non-Lustre pools (e.g. root pool for OS).</p> <p>Thanks Andreas for the feedback. I inadvertently attached my local copy used for testing but I can provide the generic one. I'll also address the issues are repost an update. Is there an example license block I can refer to?</p> <p>Don,<br/> is there a directory for zedlet scripts to be installed (e.g. <tt>/etc/zfs/zed.d/</tt>) where they will be run automatically when installed?</p> <p>Then, it should be straight forward to submit a patch (per above process) to add your script as <tt>lustre/scripts/statechange-lustre.sh</tt> and install it into the above directory via <tt>lustre/scripts/Makefile.am</tt> if ZFS is enabled:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if ZFS_ENABLED sbin_SCRIPTS += zfsobj2fid +zeddir = $(sysconfdir)/zfs/zed.d +zed_SCRIPTS = statechange-lustre.sh endif : : +EXTRA_DIST += statechange-lustre.sh </pre> </div></div> <p>and then package it in <tt>lustre.spec.in</tt> and <tt>lustre-dkms.spec.in</tt> as part of the <tt>osd-zfs-mount</tt> RPM (Lustre userspace tools for ZFS-backed targets):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> %files osd-zfs-mount %defattr(-,root,root) %{_libdir}/@PACKAGE@/mount_osd_zfs.so +%{_sysconfdir}/zfs/zed.d/statechange-lustre.sh </pre> </div></div> <p>Now, when ZFS server support is installed, your zedlet will also be installed on all the servers, and should start to handle the degraded/offline events automatically.</p> <p>The /etc/zfs/zed.d links are links to: /usr/libexec/zfs/zed.d/ for example: </p> <ol> <li>ls -lart /etc/zfs/zed.d/<b>notify</b>.sh<br/> lrwxrwxrwx. 1 root root 45 Apr 27 15:59 /etc/zfs/zed.d/scrub_finish-notify.sh -&gt; /usr/libexec/zfs/zed.d/scrub_finish-notify.sh<br/> lrwxrwxrwx. 1 root root 48 Apr 27 15:59 /etc/zfs/zed.d/resilver_finish-notify.sh -&gt; /usr/libexec/zfs/zed.d/resilver_finish-notify.sh<br/> lrwxrwxrwx. 1 root root 37 Apr 27 15:59 /etc/zfs/zed.d/data-notify.sh -&gt; /usr/libexec/zfs/zed.d/data-notify.sh<br/> lrwxrwxrwx. 1 root root 44 Apr 27 15:59 /etc/zfs/zed.d/statechange-notify.sh -&gt; /usr/libexec/zfs/zed.d/statechange-notify.sh</li> </ol> <p>Nathaniel</p> <p>Can you please see what is required to move this forward?</p> <p>Thanks</p> <p>Peter</p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30907" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30907</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4277" title="Integrate ZFS zpool resilver status with OFD OS_STATE_DEGRADED flag" class="issue-link" data-issue-key="LU-4277"><del>LU-4277</del></a> scripts: ofd status integrated with zpool status<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2f58dcb71a6246ce29a3a548f9ccafd006b97d44</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30907/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30907/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4277" title="Integrate ZFS zpool resilver status with OFD OS_STATE_DEGRADED flag" class="issue-link" data-issue-key="LU-4277"><del>LU-4277</del></a> scripts: ofd status integrated with zpool status<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8ef3ddd2f2798d04b495c8223673a38452ac5c99</p> <p>Landed for 2.11</p> <p> </p> Related LU-6767 Development Rank 1|hzw9rz: Rank (Obsolete) 11749