https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7352 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/fe434a72-7d5b-11e5-bca9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fe434a72-7d5b-11e5-bca9-5254006e85c2</a>.</p> <p>The sub-test test_78 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1048576 bytes (1.0 MB) copied, 0.140863 s, 7.4 MB/s dd: opening `/mnt/lustre/d78.conf-sanity/f78.conf-sanity-99': No space left on device conf-sanity test_78: @@@@@@ FAIL: (4) create /mnt/lustre/d78.conf-sanity/f78.conf-sanity-99 failed </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: conf-sanity 78<br/> Info required for matching: conf-sanity 79</p> LU-7352

conf-sanity test_78: no space left on device

Bug Critical Resolved Fixed WC Triage Maloo Wed, 28 Oct 2015 18:34:53 +0000 Mon, 10 Apr 2017 19:58:53 +0000 Mon, 6 Jun 2016 17:51:43 +0000 Lustre 2.9.0 Lustre 2.9.0 0 13 <p>This may relate to <a href="https://jira.whamcloud.com/browse/LU-7309" title="replay-single test_70b: no space left on device" class="issue-link" data-issue-key="LU-7309"><del>LU-7309</del></a>.</p> <p>More failures on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/fe434a72-7d5b-11e5-bca9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fe434a72-7d5b-11e5-bca9-5254006e85c2</a><br/> 2015-12-15 02:54:42 - <a href="https://testing.hpdd.intel.com/test_sets/04050e8a-a2fc-11e5-9b3d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/04050e8a-a2fc-11e5-9b3d-5254006e85c2</a><br/> 2015-12-30 07:03:26 - <a href="https://testing.hpdd.intel.com/test_sets/0df22316-aef4-11e5-aa1f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0df22316-aef4-11e5-aa1f-5254006e85c2</a></p> <p>The same failure still occurred on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/94f9085c-fd3c-11e5-8750-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/94f9085c-fd3c-11e5-8750-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/1d2bb940-090e-11e6-9e5d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1d2bb940-090e-11e6-9e5d-5254006e85c2</a></p> <p>Another failure on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/834acc42-090f-11e6-b5f1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/834acc42-090f-11e6-b5f1-5254006e85c2</a></p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20166" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20166</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7352" title="conf-sanity test_78: no space left on device" class="issue-link" data-issue-key="LU-7352"><del>LU-7352</del></a> tests: add debugging to conf-sanity test_78<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 46e82d0c3f2d94b5e579ca291df3137ef38c6612</p> <p>Looking at the client debug log from a recent failure it appears that the ENOSPC is coming from the MDS, possibly if there are no precreated objects on the OSTs yet since the filesystem was just formatted?<br/> <a href="https://testing.hpdd.intel.com/sub_tests/eadbd53e-17f8-11e6-b5f1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/eadbd53e-17f8-11e6-b5f1-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(mdc_locks.c:536:mdc_finish_enqueue()) Process entered (mdc_locks.c:590:mdc_finish_enqueue()) @@@ op: 3 disposition: 17, status: -28 req@ffff88004e7ca400 x1534075851521040/t0(0) o101-&gt;lustre-MDT0000-mdc-ffff88005ac84800@10.9.4.123@tcp:12/10 lens 888/544 e 0 to 0 dl 1463009294 ref 1 fl Complete:R/0/0 rc 301/301 (mdc_locks.c:719:mdc_finish_enqueue()) Process leaving (rc=0 : 0 : 0) (mdc_locks.c:980:mdc_finish_intent_lock()) D_IT dentry f78.conf-sanity-97 intent: open|creat status -28 disp 17 rc 0 </pre> </div></div> <p>The MDS log seems to agree with that to some extent, but it is getting 0 available blocks back from the OST, so it is generating the ENOSPC locally:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(osp_precreate.c:129:osp_statfs_interpret()) Process entered (osp_precreate.c:969:osp_pre_update_status()) lustre-OST0000-osc-MDT0000: status: 36034 blocks, 9261 free, 26 used, 0 avail -&gt; -28: rc = 0 (osp_precreate.c:972:osp_pre_update_status()) non-committed changes: 0, in progress: 0 (osp_precreate.c:151:osp_statfs_interpret()) updated statfs ffff88004f038800 (osp_precreate.c:153:osp_statfs_interpret()) Process leaving (rc=0 : 0 : 0) </pre> </div></div> <p>The OST was formatted with 45000 blocks = 175MB for the test:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=10.9.4.123@tcp sys.timeout=20 device size = 12472MB formatting backing filesystem ldiskfs on /dev/lvm-Role_OSS/P1 target name lustre:OST0000 4k blocks 45000 options -I 256 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -I 256 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init -F /dev/lvm-Role_OSS/P1 45000 </pre> </div></div> <p>but there are 7 OSTs ~= 1225MB so it is strange that there isn't enough space available on this one OST. Hopefully the extra debugging patch will show what is happening.</p> <p>The actual problem here is that there is only a single OST in the filesystem, and it is very close to running out of space, so when the MDT is trying to precreate objects it is seeing that bavail = 0 and skipping that OST, which is the only one. It isn't taking into account that the client may have reserved space on the OST that it wants to write to a new file. That said, if the MDT checked bfree it may continue to create files on that OST even though the client does not have any grant there.</p> <p>I thought many years ago that the client should be able to send a bitmap of "preferred" OSTs and/or "forbidden" OSTs during create RPCs to the MDS. This would allow the client to select OSTs that are closer to itself by network topology, skip OSTs that it does not have a connection to, or that it doesn't have any grant for.</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/638fcf44-21b3-11e6-a909-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/638fcf44-21b3-11e6-a909-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/e899791e-24bb-11e6-a8f9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e899791e-24bb-11e6-a8f9-5254006e85c2</a></p> <p>Let's think this problem more.</p> <p> Now lod_qos_dev_is_full() is always checking using bavail which will except<br/> Granting space, but MDS pre-create will only try to create empty objects on OST,<br/> which dose not eat any free blocks here.</p> <p> So if we skip granted space here and just use bfree blocks to check if we can<br/> go on, that should be fine, considering when clients connect to OST, it should<br/> have tried it best to get some grant space or just fall back to sync/direct write.</p> <p> Another thing that I notcied is now it has following comments:<br/> /* the minimum of 0.1% used blocks and 1GB bytes. */<br/> I think this still not perfect, think carefull, with a big OST, we might lost 1GB space,<br/> and I am wondering there exist a case, Users just want empty or small files, But it can<br/> not go on because of this check.</p> <p>Hmm..one point is MDS is trying to balance OST objects allocation between different<br/> OST, trying to allocate new objects on those with more free blocks.</p> <p>Whatever way, users will still fail to create empty files, when all OST<br/> are running out of this check..</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="http://review.whamcloud.com/20166/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20166/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7352" title="conf-sanity test_78: no space left on device" class="issue-link" data-issue-key="LU-7352"><del>LU-7352</del></a> tests: don't fail conf-sanity test_78 on ENOSPC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b666d02e13e0890a244910221c5833d2af39aec9</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20558" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20558</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7352" title="conf-sanity test_78: no space left on device" class="issue-link" data-issue-key="LU-7352"><del>LU-7352</del></a> tests: conf-sanity 78 don't check missing files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d6a01df3686ce965e2ff268a8f667f926f9bd1e8</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20558/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20558/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7352" title="conf-sanity test_78: no space left on device" class="issue-link" data-issue-key="LU-7352"><del>LU-7352</del></a> tests: conf-sanity 78 don't check missing files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 46319ab0903ccd1a28a3011eef995e7f5cfe334e</p> Duplicate Related LU-7309 LU-7871 Development Rank 1|hzxrpj: Rank (Obsolete) 9223372036854775807 Severity