https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1236 Lustre <p>I'm benchmarking a SSD device using <tt>mds-survey</tt>. If I run with:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$ dir_count=1 thrlo=8 thrhi=10000 file_count=100000 sh mds-survey </pre> </div></div> <p>The following is seen in <tt>dmesg</tt> immediatly before the machine hangs:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">... 1332181376 206.76.192.63 6665 Directory (ino: 130547715) index full, reach max htree level :2 1332181376 206.76.192.63 6665 LDISKFS-fs warning (device rsXX0): ldiskfs_dx_add_entry: 1332181376 206.76.192.63 6665 Large directory feature isnot enabled on <span class="code-keyword">this</span> filesystem 1332181376 206.76.192.63 6665 LDISKFS-fs warning (device rsXX0): ldiskfs_dx_add_entry: 1332181376 206.76.192.63 6665 Directory (ino: 130547715) index full, reach max htree level :2 1332181376 206.76.192.63 6665 LDISKFS-fs warning (device rsXX0): ldiskfs_dx_add_entry: 1332181376 206.76.192.63 6665 Large directory feature isnot enabled on <span class="code-keyword">this</span> filesystem 1332181398 206.76.192.63 6665 LustreError: 5111:0:(echo_client.c:1768:echo_lookup_object()) Can not lookup child 436236872: rc = -2 1332181398 206.76.192.63 6665 LustreError: 5111:0:(echo_client.c:1768:echo_lookup_object()) Skipped 1 previous similar message 1332181399 206.76.192.63 6665 LustreError: 5144:0:(echo_client.c:1768:echo_lookup_object()) Can not lookup child 989887001: rc = -2 1332181400 206.76.192.63 6665 LustreError: 5087:0:(echo_client.c:1768:echo_lookup_object()) Can not lookup child 33586437: rc = -2 1332181400 206.76.192.63 6665 LustreError: 5087:0:(echo_client.c:1768:echo_lookup_object()) Skipped 9 previous similar messages 1332181402 206.76.192.63 6665 LustreError: 5197:0:(echo_client.c:1768:echo_lookup_object()) Can not lookup child 1879083690: rc = -2 1332181402 206.76.192.63 6665 LustreError: 5197:0:(echo_client.c:1768:echo_lookup_object()) Skipped 57 previous similar messages 1332181970 206.76.192.63 6665 LustreError: 5486:0:(echo_client.c:1592:echo_md_lookup()) lookup 18446744071562098618: rc = -2 1332181970 206.76.192.63 6665 LustreError: 5486:0:(echo_client.c:1592:echo_md_lookup()) Skipped 127 previous similar messages 1332181970 206.76.192.63 6665 LustreError: 5486:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 18446744071562098618: rc = -2 1332181970 206.76.192.63 6665 LustreError: 5486:0:(echo_client.c:1706:echo_getattr_object()) Skipped 4 previous similar messages 1332181974 206.76.192.63 6665 LustreError: 5571:0:(echo_client.c:1592:echo_md_lookup()) lookup 18446744072988161445: rc = -2 1332181974 206.76.192.63 6665 LustreError: 5571:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 18446744072988161445: rc = -2 1332181988 206.76.192.63 6665 LustreError: 5538:0:(echo_client.c:1592:echo_md_lookup()) lookup 18446744072434514081: rc = -2 1332181988 206.76.192.63 6665 LustreError: 5538:0:(echo_client.c:1592:echo_md_lookup()) Skipped 1 previous similar message 1332181988 206.76.192.63 6665 LustreError: 5538:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 18446744072434514081: rc = -2 1332181988 206.76.192.63 6665 LustreError: 5538:0:(echo_client.c:1706:echo_getattr_object()) Skipped 1 previous similar message 1332182004 206.76.192.63 6665 LustreError: 5531:0:(echo_client.c:1592:echo_md_lookup()) lookup 18446744072317071604: rc = -2 1332182004 206.76.192.63 6665 LustreError: 5531:0:(echo_client.c:1592:echo_md_lookup()) Skipped 1 previous similar message 1332182004 206.76.192.63 6665 LustreError: 5531:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 18446744072317071604: rc = -2 1332182004 206.76.192.63 6665 LustreError: 5531:0:(echo_client.c:1706:echo_getattr_object()) Skipped 1 previous similar message 1332182023 206.76.192.63 6665 LustreError: 5554:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 18446744072702952620: rc = -2 1332182023 206.76.192.63 6665 LustreError: 5554:0:(echo_client.c:1706:echo_getattr_object()) Skipped 3 previous similar messages 1332182038 206.76.192.63 6665 LustreError: 5589:0:(echo_client.c:1592:echo_md_lookup()) lookup 18446744073290153182: rc = -2 1332182038 206.76.192.63 6665 LustreError: 5589:0:(echo_client.c:1592:echo_md_lookup()) Skipped 20 previous similar messages 1332182049 206.76.192.63 6665 LustreError: 5451:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 1560309174: rc = -2 1332182049 206.76.192.63 6665 LustreError: 5451:0:(echo_client.c:1706:echo_getattr_object()) Skipped 18 previous similar messages 1332182081 206.76.192.63 6665 LustreError: 5436:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 1308658240: rc = -2 1332182081 206.76.192.63 6665 LustreError: 5436:0:(echo_client.c:1706:echo_getattr_object()) Skipped 110 previous similar messages 1332182105 206.76.192.63 6665 LustreError: 5500:0:(echo_client.c:1592:echo_md_lookup()) lookup 18446744071796981746: rc = -2 1332182105 206.76.192.63 6665 LustreError: 5500:0:(echo_client.c:1592:echo_md_lookup()) Skipped 226 previous similar messages </pre> </div></div> <p>This log message originates from:<br/> <tt>./ldiskfs/kernel_patches/patches/ext4_pdirop-rhel6.patch</tt></p> LU-1236

Large directory feature isnot enabled on this filesystem

Bug Blocker Resolved Fixed Liang Zhen Richard Henwood Mon, 19 Mar 2012 15:17:25 +0000 Tue, 20 Mar 2012 11:36:00 +0000 Tue, 20 Mar 2012 10:38:08 +0000 Lustre 2.2.0 Lustre 2.3.0 0 4 <p>The <tt>mds-survey</tt> command creates a total of 100000 files in a single directory. It steps through a thread count from 8 to 10000, timing how long with a given number of threads, Create, Lookup, getAttr and setxAttr take.</p> <p>Seems the problem occurs at the 128 thread mark:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">~# dir_count=1 thrlo=8 thrhi=10000 file_count=100000 sh /usr/bin/mds-survey Mon Mar 19 14:24:18 CDT 2012 /usr/bin/mds-survey from mds51.ls4.tacc.utexas.edu mdt 1 file 100000 dir 1 thr 8 create 49207.37 [34994.58,55993.90] lookup 560559.72 [560559.72,560559.72] md_getattr 589089.04 [589089.04,589089.04] setxattr 341067.03 [359985.60,359985.60] destroy 132062.28 [124984.88,158982.04] mdt 1 file 100000 dir 1 thr 16 create 90094.03 [57996.58,165989.21] lookup 445579.31 [442979.62,442979.62] md_getattr 437745.60 [436985.14,436985.14] setxattr 158132.21 [123991.94,169978.24] destroy 72703.35 [53996.65,97988.54] mdt 1 file 100000 dir 1 thr 32 create 82619.38 [62995.15,106993.37] lookup 429201.01 [419986.14,435964.25] md_getattr 405683.99 [401974.68,406984.13] setxattr 102304.06 [82989.54,153975.52] destroy 49015.01 [35995.10,64992.07] mdt 1 file 100000 dir 1 thr 64 create 57911.35 [43997.01,74995.13] lookup 411225.64 [404976.92,410975.75] md_getattr 127483.38 [ 0.00,414142.72] setxattr 110405.23 [90977.98,136990.41] destroy 43722.84 [31996.03,59969.24] mdt 1 file 100000 dir 1 thr 128 create ERROR lookup ERROR md_getattr </pre> </div></div> <p>The host was rest during the crash, but the kernel messages again looked like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">1332185881 206.76.192.63 6665 LustreError: 3209:0:(echo_client.c:1451:echo_md_create_internal()) Can not create child [0x2000004c6:0x123f7:0x0]: rc = -28 1332185881 206.76.192.63 6665 LustreError: 3209:0:(echo_client.c:1562:echo_create_md_object()) Can not create child 18446744072199676837: rc = -28 1332185881 206.76.192.63 6665 LustreError: 3218:0:(echo_client.c:1451:echo_md_create_internal()) Can not create child [0x20000047a:0x120a9:0x0]: rc = -28 1332185881 206.76.192.63 6665 LustreError: 3218:0:(echo_client.c:1562:echo_create_md_object()) Can not create child 18446744072501665870: rc = -28 1332185881 206.76.192.63 6665 LustreError: 3218:0:(echo_client.c:1562:echo_create_md_object()) Skipped 1 previous similar message 1332185882 206.76.192.63 6665 LustreError: 3236:0:(echo_client.c:1451:echo_md_create_internal()) Can not create child [0x200000498:0x1237f:0x0]: rc = -28 1332185882 206.76.192.63 6665 LustreError: 3236:0:(echo_client.c:1451:echo_md_create_internal()) Skipped 6 previous similar messages 1332185882 206.76.192.63 6665 LustreError: 3236:0:(echo_client.c:1562:echo_create_md_object()) Can not create child 18446744073105646354: rc = -28 1332185882 206.76.192.63 6665 LustreError: 3236:0:(echo_client.c:1562:echo_create_md_object()) Skipped 5 previous similar messages 1332185883 206.76.192.63 6665 LustreError: 3213:0:(echo_client.c:1451:echo_md_create_internal()) Can not create child [0x2000004d4:0x124d0:0x0]: rc = -28 1332185883 206.76.192.63 6665 LustreError: 3234:0:(echo_client.c:1562:echo_create_md_object()) Can not create child 18446744073038537898: rc = -28 1332185883 206.76.192.63 6665 LustreError: 3234:0:(echo_client.c:1562:echo_create_md_object()) Skipped 15 previous similar messages 1332185883 206.76.192.63 6665 LustreError: 3213:0:(echo_client.c:1451:echo_md_create_internal()) Skipped 20 previous similar messages 1332185885 206.76.192.63 6665 LustreError: 3153:0:(echo_client.c:1451:echo_md_create_internal()) Can not create child [0x200000491:0x128c2:0x0]: rc = -28 1332185885 206.76.192.63 6665 LustreError: 3153:0:(echo_client.c:1451:echo_md_create_internal()) Skipped 63 previous similar messages 1332185885 206.76.192.63 6665 LustreError: 3230:0:(echo_client.c:1562:echo_create_md_object()) Can not create child 18446744072904321222: rc = -28 1332185885 206.76.192.63 6665 LustreError: 3230:0:(echo_client.c:1562:echo_create_md_object()) Skipped 69 previous similar messages 1332185909 206.76.192.63 6665 LustreError: 3505:0:(echo_client.c:1768:echo_lookup_object()) Can not lookup child 18446744072669436954: rc = -2 1332185909 206.76.192.63 6665 LustreError: 3487:0:(echo_client.c:1768:echo_lookup_object()) Can not lookup child 18446744072065458648: rc = -2 1332185910 206.76.192.63 6665 LustreError: 3485:0:(echo_client.c:1768:echo_lookup_object()) Can not lookup child 18446744071998350893: rc = -2 1332185910 206.76.192.63 6665 LustreError: 3485:0:(echo_client.c:1768:echo_lookup_object()) Skipped 5 previous similar messages 1332185911 206.76.192.63 6665 LustreError: 3413:0:(echo_client.c:1768:echo_lookup_object()) Can not lookup child 167846909: rc = -2 1332185911 206.76.192.63 6665 LustreError: 3413:0:(echo_client.c:1768:echo_lookup_object()) Skipped 17 previous similar messages 1332186150 206.76.192.63 6665 LustreError: 3585:0:(echo_client.c:1592:echo_md_lookup()) lookup 1140923015: rc = -2 1332186150 206.76.192.63 6665 LustreError: 3585:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 1140923015: rc = -2 1332186150 206.76.192.63 6665 LustreError: 3599:0:(echo_client.c:1592:echo_md_lookup()) lookup 1610686777: rc = -2 1332186150 206.76.192.63 6665 LustreError: 3599:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 1610686777: rc = -2 1332186151 206.76.192.63 6665 LustreError: 3556:0:(echo_client.c:1592:echo_md_lookup()) lookup 167846909: rc = -2 1332186151 206.76.192.63 6665 LustreError: 3556:0:(echo_client.c:1592:echo_md_lookup()) Skipped 13 previous similar messages 1332186151 206.76.192.63 6665 LustreError: 3556:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 167846909: rc = -2 1332186151 206.76.192.63 6665 LustreError: 3556:0:(echo_client.c:1706:echo_getattr_object()) Skipped 13 previous similar messages 1332186152 206.76.192.63 6665 LustreError: 3664:0:(echo_client.c:1592:echo_md_lookup()) lookup 18446744073206311328: rc = -2 1332186152 206.76.192.63 6665 LustreError: 3664:0:(echo_client.c:1592:echo_md_lookup()) Skipped 33 previous similar messages 1332186152 206.76.192.63 6665 LustreError: 3583:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 1073819696: rc = -2 1332186152 206.76.192.63 6665 LustreError: 3583:0:(echo_client.c:1706:echo_getattr_object()) Skipped 33 previous similar messages 1332186154 206.76.192.63 6665 LustreError: 3643:0:(echo_client.c:1592:echo_md_lookup()) lookup 18446744072501665870: rc = -2 1332186154 206.76.192.63 6665 LustreError: 3582:0:(echo_client.c:1706:echo_getattr_object()) Can't find child 1040263450: rc = -2 1332186154 206.76.192.63 6665 LustreError: 3582:0:(echo_client.c:1706:echo_getattr_object()) Skipped 65 previous similar messages 1332186154 206.76.192.63 6665 LustreError: 3643:0:(echo_client.c:1592:echo_md_lookup()) Skipped 66 previous similar messages </pre> </div></div> <p>Are you sure it isn't creating 10k files per thread? That would make this a 1B file test, which is definitely beyond the default limits for htree directories.</p> <p>The reason that we implemented the check/limit for directories over 2 levels is because the e2fsck support for &gt;2 htree levels and &gt;2GB size is not implemented yet (<a href="https://jira.whamcloud.com/browse/LU-896" title="change e2fsprogs to make it allow directories over 2GB" class="issue-link" data-issue-key="LU-896"><del>LU-896</del></a>).</p> <p>That functionality needs to be implemented before this can be enabled in production, but it isn't a huge problem in most cases since the current limit of ~10-15M files is enough for most users.</p> <p>As Andreas said, it will be enabled in <a href="https://jira.whamcloud.com/browse/LU-896" title="change e2fsprogs to make it allow directories over 2GB" class="issue-link" data-issue-key="LU-896"><del>LU-896</del></a>, so we can just close it.</p> <p>The file_count is per threads. make sure you have enough inode for $thr * $file_count</p> Development Rank 1|hzvh5r: Rank (Obsolete) 6429 Severity