https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6538 Lustre <p>Some clients can't write to some OSTs.</p> <p>user@gordon-ln3 LTEST]$ cd /oasis/projects/nsf/use300/mahidhar/LTEST<br/> <span class="error">&#91;user@gordon-ln3 LTEST&#93;</span>$ lfs setstripe -c 1 -i 54 .<br/> <span class="error">&#91;user@gordon-ln3 LTEST&#93;</span>$ ls<br/> IOR-2.10.3.tar<br/> <span class="error">&#91;user@gordon-ln3 LTEST&#93;</span>$ rm IOR-2.10.3.tar <br/> <span class="error">&#91;user@gordon-ln3 LTEST&#93;</span>$ cp ../IOR-2.10.3.tar .<br/> cp: writing `./IOR-2.10.3.tar': No such process<br/> <span class="error">&#91;user@gordon-ln3 LTEST&#93;</span>$ </p> <p>Client dmesg got:</p> <p>LustreError: 11-0: meerkat-OST0015-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx244@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST000e-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST000d-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0024-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST002a-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0009-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST000e-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST000d-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0024-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST002a-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0009-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0026-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0026-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0025-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0017-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0031-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0025-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0036-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST0031-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3.<br/> LustreError: 11-0: meerkat-OST000c-osc-ffff88102f998c00: Communicating with xx.xx.xx.xx@tcp, operation ost_write failed with -3. </p> <p>Server log see attachment</p> Linux meerkat-mds-10-1.local 2.6.32-431.29.2.el6_lustre.gb8d9077.x86_64 #1 SMP Mon Apr 27 12:55:47 PDT 2015 x86_64 x86_64 x86_64 GNU/Linux<br/> LU-6538

Client s getting "operation ost_write failed with -3."

Bug Blocker Resolved Fixed Niu Yawei Haisong Cai sdsc Wed, 29 Apr 2015 05:01:06 +0000 Wed, 22 Jun 2016 07:40:31 +0000 Wed, 22 Jun 2016 07:40:30 +0000 Lustre 2.5.3 0 6 <p>Minh, can you add an sdsc label to this ticket?</p> <p>From the log, it possibly relates to osd quota operation osd_acct_index_lookup() which calls vfs_get_dqblk()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> vfs_get_dqblk(struct super_block *sb, <span class="code-object">int</span> type, qid_t id, struct if_dqblk *di) { struct dquot *dquot; dquot = dqget(sb, id, type); <span class="code-keyword">if</span> (!dquot) <span class="code-keyword">return</span> -ESRCH; do_get_dqblk(dquot, di); dqput(dquot); <span class="code-keyword">return</span> 0; } </pre> </div></div> <p>and it cannot get reference to dquot.</p> <p>Hi Zhenyu,</p> <p>Is this a bug or setting problem?</p> <p>Haisong</p> <p>Hi, Haisong</p> <p>This looks like a setting problem.</p> <p>Do all the OST/MDT devices have quota feature enabled? You can run following command to verify it: "debugfs -c -R features $dev"<br/> And please check the quota proc file "/proc/fs/lustre/osd-ldiskfs/$MDT_or_$OST/quota_slave/info" to see if the quota accounting is setup properly. Thank you.</p> <p>Hi Niu,</p> <p>proc and debugfs checked out in OSS and MDS. <br/> Besides a few fields differences between OST and MDT, all targets have identical settings (see attached 2 output files)</p> <p>thanks,<br/> Haisong</p> <p>All devices looks have quota feature enabled, then I suspect that quota wasn't enabled on mount for some OSTs. (OST0034 for instance)</p> <p>When did you observe such write failures (-3)? Any error happened on these OSTs before the -3 failures? Do you happen to have earlier log messages so that we can check if quota wasn't enabled successfully on mount time?</p> <p>Could you pick on problematic OST (OST0034 for instance), and perform umount -&gt; mount to see if that will resolve the problem? And please also attach the log messages when umount/mount the OST. Thank you. </p> <p>Niu,</p> <p>1) We are mounting OST simply like this:<br/> mount -t lustre /dev/ost_sdc /meerkat/ost_sdc</p> <p> Are there options we should specify when mounting?</p> <p>2) (-3) error is happening one client side when someone write to problematic OSTs. It doesn't happen for user and either every time. The filesystem was upgraded from 1.8.7 to 2.4.3 a year ago and didn't had problem. Last Monday we upgraded from 2.4.3 to 2.5.3. Almost right after that we started getting write errors. </p> <p>3) I will gather the logs when we mount OSTs last Monday and upload here </p> <p>4) I will also try to umount and mount the problematic and attach log here. </p> <p>Haisong</p> <p>Tried unmount and mount an bad OST:</p> <p><span class="error">&#91;root@meerkat-oss-11-4 ~&#93;</span># mount -t lustre /dev/ost_sdi /meerkat/ost_sdi<br/> mount.lustre: set /sys/block/sdi/queue/max_sectors_kb to 280</p> <p><span class="error">&#91;root@meerkat-oss-11-4 ~&#93;</span># </p> <p>Last few lines from dmesg on 11-4:</p> <p>VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> LustreError: 4787:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000e: fail to locate lqe for id:505418, type:0<br/> LustreError: 4787:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 9787 previous similar messages<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> Lustre: 12307:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0036: reint global for <span class="error">&#91;0x200000006:0x20000:0x0&#93;</span> failed. -3<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> LustreError: 4787:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000e: fail to locate lqe for id:505418, type:0<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.<br/> VFS: find_free_dqentry(): Data block full but it shouldn't.<br/> VFS: Error -5 occurred while creating quota.</p> <p>Looks the quota accounting files are corrupted somehow, could you make sure that latest e2fsprogs are installed on the problematic OSTs? (1.42.12-wc1)<br/> Then try to fix the quota files by:</p> <ul class="alternate" type="square"> <li>umount the OST;</li> <li>tune2fs -O ^quota $ostdev;</li> <li>tune2fs -O quota $ostdev;</li> <li>mount OST;</li> </ul> <p>Aggregated logs from all MDS/OSS, before and after 2.4.3 -&gt; 2.5.3 upgrade.<br/> Sometime between 3pm and 4pm on that day was when we mounted OST/MDT.</p> <p>On all OSTs or just the ones reporting problem?</p> <p>thanks,<br/> Haisong</p> <p>By the way....</p> <p><span class="error">&#91;root@meerkat-oss-11-4 log&#93;</span># rpm -aq | grep e2fsprog<br/> e2fsprogs-libs-1.42.12.wc1-7.el6.x86_64<br/> e2fsprogs-1.42.12.wc1-7.el6.x86_64</p> <p><span class="error">&#91;root@meerkat-oss-11-4 log&#93;</span># umount /meerkat/ost_sdi<br/> <span class="error">&#91;root@meerkat-oss-11-4 log&#93;</span># tune2fs -O ^quota /dev/ost_sdi<br/> tune2fs 1.42.12.wc1 (15-Sep-2014)<br/> <span class="error">&#91;root@meerkat-oss-11-4 log&#93;</span># tune2fs -O quota /dev/ost_sdi<br/> tune2fs 1.42.12.wc1 (15-Sep-2014)<br/> <span class="error">&#91;root@meerkat-oss-11-4 log&#93;</span># mount -t lustre /dev/ost_sdi /meerkat/ost_sdi<br/> mount.lustre: set /sys/block/sdi/queue/max_sectors_kb to 280</p> <p><span class="error">&#91;root@meerkat-oss-11-4 log&#93;</span># </p> <p>Testing and will let you know if it fix the problem or not.</p> <p>I'm not sure if the latest e2fsprogs has been installed when you enable quota feature for the OST/MDT devices (when upgrading from 2.1 to 2.4). If it wasn't the latest version, I suggest you to run it for all MDT/OSTs. Of course, you'd choose the problematic one first to see if it can fix the problem.</p> <p>Niu,</p> <p>A quick test on one the problematic OSTs show that fix the write problem. <br/> We are going to initialize all OSTs and MDT now. </p> <p>Haisong </p> <p>Haisong, anything else need we do on this ticket? Can we close this one?</p> Development End date Thu, 23 Jul 2015 05:01:06 +0000 Rank 1|hzxc2v: Rank (Obsolete) 9223372036854775807 Severity Start date Wed, 29 Apr 2015 05:01:06 +0000