Details
-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
3
-
9223372036854775807
Description
Testing this build: https://build.hpdd.intel.com/job/lustre-reviews/32021/
In AWS environment with 64 MDTs (8 MDS * 8 MDT each).
- cd /mnt/lustre
- lfs mkdir -c 8 8stripedir
- lfs mkdir -c 64 64stripedir
<hang>
On MDS0LustreError: 1291:0:(llog_cat.c:319:llog_cat_add_rec()) ASSERTION( rec->lrh_len <= 8192 ) failed: LustreError: 1291:0:(llog_cat.c:319:llog_cat_add_rec()) LBUG Pid: 1291, comm: mdt00_002 Call Trace: [<ffffffffa00f2875>] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [<ffffffffa00f2e77>] lbug_with_loc+0x47/0xb0 [libcfs] [<ffffffffa0207848>] llog_cat_add_rec+0x3e8/0x450 [obdclass] [<ffffffffa01ff039>] llog_add+0x89/0x1c0 [obdclass] [<ffffffffa187b6f4>] sub_updates_write+0x154/0x600 [ptlrpc] [<ffffffffa187c247>] top_trans_stop+0x6a7/0xb40 [ptlrpc] [<ffffffffa1d8cd21>] lod_trans_stop+0x61/0x70 [lod] [<ffffffffa1e3149a>] mdd_trans_stop+0x1a/0xac [mdd] [<ffffffffa1e20909>] mdd_create+0x13a9/0x1750 [mdd] [<ffffffffa1cdb65c>] ? mdt_version_save+0x8c/0x1a0 [mdt] [<ffffffffa1cdf9ec>] mdt_reint_create+0xbbc/0xcc0 [mdt] [<ffffffffa1cdab1d>] mdt_reint_rec+0x5d/0x200 [mdt] [<ffffffffa1cbffcb>] mdt_reint_internal+0x4cb/0x7a0 [mdt] [<ffffffffa1cc073b>] mdt_reint+0x6b/0x120 [mdt] [<ffffffffa1868e8e>] tgt_request_handle+0x8be/0xfe0 [ptlrpc] [<ffffffffa1818aa1>] ptlrpc_main+0xe41/0x1970 [ptlrpc] [<ffffffff81060c3f>] ? finish_task_switch+0x4f/0xf0 [<ffffffffa1817c60>] ? ptlrpc_main+0x0/0x1970 [ptlrpc] [<ffffffff8109e71e>] kthread+0x9e/0xc0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b [<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6 [<ffffffff8100c200>] ? child_rip+0x0/0x20 Kernel panic - not syncing: LBUG Pid: 1291, comm: mdt00_002 Not tainted 2.6.32-504.16.2.el6_lustre.gd805a88.x86_64 #1 Call Trace: [<ffffffff81529fbc>] ? panic+0xa7/0x16f [<ffffffffa00f2ecb>] ? lbug_with_loc+0x9b/0xb0 [libcfs] [<ffffffffa0207848>] ? llog_cat_add_rec+0x3e8/0x450 [obdclass] [<ffffffffa01ff039>] ? llog_add+0x89/0x1c0 [obdclass] [<ffffffffa187b6f4>] ? sub_updates_write+0x154/0x600 [ptlrpc] [<ffffffffa187c247>] ? top_trans_stop+0x6a7/0xb40 [ptlrpc] [<ffffffffa1d8cd21>] ? lod_trans_stop+0x61/0x70 [lod] [<ffffffffa1e3149a>] ? mdd_trans_stop+0x1a/0xac [mdd] [<ffffffffa1e20909>] ? mdd_create+0x13a9/0x1750 [mdd] [<ffffffffa1cdb65c>] ? mdt_version_save+0x8c/0x1a0 [mdt] [<ffffffffa1cdf9ec>] ? mdt_reint_create+0xbbc/0xcc0 [mdt] [<ffffffffa1cdab1d>] ? mdt_reint_rec+0x5d/0x200 [mdt] [<ffffffffa1cbffcb>] ? mdt_reint_internal+0x4cb/0x7a0 [mdt] [<ffffffffa1cc073b>] ? mdt_reint+0x6b/0x120 [mdt] [<ffffffffa1868e8e>] ? tgt_request_handle+0x8be/0xfe0 [ptlrpc] [<ffffffffa1818aa1>] ? ptlrpc_main+0xe41/0x1970 [ptlrpc] [<ffffffff81060c3f>] ? finish_task_switch+0x4f/0xf0 [<ffffffffa1817c60>] ? ptlrpc_main+0x0/0x1970 [ptlrpc] [<ffffffff8109e71e>] ? kthread+0x9e/0xc0 [<ffffffff8100c20a>] ? child_rip+0xa/0x20 [<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b [<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6 [<ffffffff8100c200>] ? child_rip+0x0/0x20
After each reboot/recovery cycle the MDS would LBUG again with same error right after recovery completed. Presumably the client was resending the mkdir. Once I killed lfs, the crashes stopped.
Attachments
Issue Links
Activity
I just upgrade the patch http://review.whamcloud.com/#/c/13942
With the patch, I can create 512 stripes striped dir locally.
== sanity test 300k: test large striped directory == 01:30:04 (1433320204) fail_loc=0x1703 fail_loc=0 /mnt/lustre/d300k.sanity/striped_dir lmv_stripe_count: 512 lmv_stripe_offset: 0 mdtidx FID[seq:oid:ver] 0 [0x200000400:0x104:0x0] 1 [0x240000403:0x104:0x0] 2 [0x280000403:0x104:0x0] 3 [0x2c0000403:0x104:0x0] 0 [0x200000400:0x105:0x0] 1 [0x240000403:0x105:0x0] 2 [0x280000403:0x105:0x0] 3 [0x2c0000403:0x105:0x0] 0 [0x200000400:0x106:0x0] 1 [0x240000403:0x106:0x0] 2 [0x280000403:0x106:0x0] 3 [0x2c0000403:0x106:0x0] 0 [0x200000400:0x107:0x0] 1 [0x240000403:0x107:0x0] 2 [0x280000403:0x107:0x0] 3 [0x2c0000403:0x107:0x0] 0 [0x200000400:0x108:0x0] 1 [0x240000403:0x108:0x0] 2 [0x280000403:0x108:0x0] 3 [0x2c0000403:0x108:0x0] 0 [0x200000400:0x109:0x0] 1 [0x240000403:0x109:0x0] 2 [0x280000403:0x109:0x0] 3 [0x2c0000403:0x109:0x0] 0 [0x200000400:0x10a:0x0] 1 [0x240000403:0x10a:0x0] 2 [0x280000403:0x10a:0x0] 3 [0x2c0000403:0x10a:0x0] 0 [0x200000400:0x10b:0x0] 1 [0x240000403:0x10b:0x0] 2 [0x280000403:0x10b:0x0] 3 [0x2c0000403:0x10b:0x0] 0 [0x200000400:0x10c:0x0] 1 [0x240000403:0x10c:0x0] 2 [0x280000403:0x10c:0x0] 3 [0x2c0000403:0x10c:0x0] 0 [0x200000400:0x10d:0x0] 1 [0x240000403:0x10d:0x0] 2 [0x280000403:0x10d:0x0] 3 [0x2c0000403:0x10d:0x0] 0 [0x200000400:0x10e:0x0] 1 [0x240000403:0x10e:0x0] 2 [0x280000403:0x10e:0x0] 3 [0x2c0000403:0x10e:0x0] 0 [0x200000400:0x10f:0x0] 1 [0x240000403:0x10f:0x0] 2 [0x280000403:0x10f:0x0] 3 [0x2c0000403:0x10f:0x0] 0 [0x200000400:0x110:0x0] 1 [0x240000403:0x110:0x0] 2 [0x280000403:0x110:0x0] 3 [0x2c0000403:0x110:0x0] 0 [0x200000400:0x111:0x0] 1 [0x240000403:0x111:0x0] 2 [0x280000403:0x111:0x0] 3 [0x2c0000403:0x111:0x0] 0 [0x200000400:0x112:0x0] 1 [0x240000403:0x112:0x0] 2 [0x280000403:0x112:0x0] 3 [0x2c0000403:0x112:0x0] 0 [0x200000400:0x113:0x0] 1 [0x240000403:0x113:0x0] 2 [0x280000403:0x113:0x0] 3 [0x2c0000403:0x113:0x0] 0 [0x200000400:0x114:0x0] 1 [0x240000403:0x114:0x0] 2 [0x280000403:0x114:0x0] 3 [0x2c0000403:0x114:0x0] 0 [0x200000400:0x115:0x0] 1 [0x240000403:0x115:0x0] 2 [0x280000403:0x115:0x0] 3 [0x2c0000403:0x115:0x0] 0 [0x200000400:0x116:0x0] 1 [0x240000403:0x116:0x0] 2 [0x280000403:0x116:0x0] 3 [0x2c0000403:0x116:0x0] 0 [0x200000400:0x117:0x0] 1 [0x240000403:0x117:0x0] 2 [0x280000403:0x117:0x0] 3 [0x2c0000403:0x117:0x0] 0 [0x200000400:0x118:0x0] 1 [0x240000403:0x118:0x0] 2 [0x280000403:0x118:0x0] 3 [0x2c0000403:0x118:0x0] 0 [0x200000400:0x119:0x0] 1 [0x240000403:0x119:0x0] 2 [0x280000403:0x119:0x0] 3 [0x2c0000403:0x119:0x0] 0 [0x200000400:0x11a:0x0] 1 [0x240000403:0x11a:0x0] 2 [0x280000403:0x11a:0x0] 3 [0x2c0000403:0x11a:0x0] 0 [0x200000400:0x11b:0x0] 1 [0x240000403:0x11b:0x0] 2 [0x280000403:0x11b:0x0] 3 [0x2c0000403:0x11b:0x0] 0 [0x200000400:0x11c:0x0] 1 [0x240000403:0x11c:0x0] 2 [0x280000403:0x11c:0x0] 3 [0x2c0000403:0x11c:0x0] 0 [0x200000400:0x11d:0x0] 1 [0x240000403:0x11d:0x0] 2 [0x280000403:0x11d:0x0] 3 [0x2c0000403:0x11d:0x0] 0 [0x200000400:0x11e:0x0] 1 [0x240000403:0x11e:0x0] 2 [0x280000403:0x11e:0x0] 3 [0x2c0000403:0x11e:0x0] 0 [0x200000400:0x11f:0x0] 1 [0x240000403:0x11f:0x0] 2 [0x280000403:0x11f:0x0] 3 [0x2c0000403:0x11f:0x0] 0 [0x200000400:0x120:0x0] 1 [0x240000403:0x120:0x0] 2 [0x280000403:0x120:0x0] 3 [0x2c0000403:0x120:0x0] 0 [0x200000400:0x121:0x0] 1 [0x240000403:0x121:0x0] 2 [0x280000403:0x121:0x0] 3 [0x2c0000403:0x121:0x0] 0 [0x200000400:0x122:0x0] 1 [0x240000403:0x122:0x0] 2 [0x280000403:0x122:0x0] 3 [0x2c0000403:0x122:0x0] 0 [0x200000400:0x123:0x0] 1 [0x240000403:0x123:0x0] 2 [0x280000403:0x123:0x0] 3 [0x2c0000403:0x123:0x0] 0 [0x200000400:0x124:0x0] 1 [0x240000403:0x124:0x0] 2 [0x280000403:0x124:0x0] 3 [0x2c0000403:0x124:0x0] 0 [0x200000400:0x125:0x0] 1 [0x240000403:0x125:0x0] 2 [0x280000403:0x125:0x0] 3 [0x2c0000403:0x125:0x0] 0 [0x200000400:0x126:0x0] 1 [0x240000403:0x126:0x0] 2 [0x280000403:0x126:0x0] 3 [0x2c0000403:0x126:0x0] 0 [0x200000400:0x127:0x0] 1 [0x240000403:0x127:0x0] 2 [0x280000403:0x127:0x0] 3 [0x2c0000403:0x127:0x0] 0 [0x200000400:0x128:0x0] 1 [0x240000403:0x128:0x0] 2 [0x280000403:0x128:0x0] 3 [0x2c0000403:0x128:0x0] 0 [0x200000400:0x129:0x0] 1 [0x240000403:0x129:0x0] 2 [0x280000403:0x129:0x0] 3 [0x2c0000403:0x129:0x0] 0 [0x200000400:0x12a:0x0] 1 [0x240000403:0x12a:0x0] 2 [0x280000403:0x12a:0x0] 3 [0x2c0000403:0x12a:0x0] 0 [0x200000400:0x12b:0x0] 1 [0x240000403:0x12b:0x0] 2 [0x280000403:0x12b:0x0] 3 [0x2c0000403:0x12b:0x0] 0 [0x200000400:0x12c:0x0] 1 [0x240000403:0x12c:0x0] 2 [0x280000403:0x12c:0x0] 3 [0x2c0000403:0x12c:0x0] 0 [0x200000400:0x12d:0x0] 1 [0x240000403:0x12d:0x0] 2 [0x280000403:0x12d:0x0] 3 [0x2c0000403:0x12d:0x0] 0 [0x200000400:0x12e:0x0] 1 [0x240000403:0x12e:0x0] 2 [0x280000403:0x12e:0x0] 3 [0x2c0000403:0x12e:0x0] 0 [0x200000400:0x12f:0x0] 1 [0x240000403:0x12f:0x0] 2 [0x280000403:0x12f:0x0] 3 [0x2c0000403:0x12f:0x0] 0 [0x200000400:0x130:0x0] 1 [0x240000403:0x130:0x0] 2 [0x280000403:0x130:0x0] 3 [0x2c0000403:0x130:0x0] 0 [0x200000400:0x131:0x0] 1 [0x240000403:0x131:0x0] 2 [0x280000403:0x131:0x0] 3 [0x2c0000403:0x131:0x0] 0 [0x200000400:0x132:0x0] 1 [0x240000403:0x132:0x0] 2 [0x280000403:0x132:0x0] 3 [0x2c0000403:0x132:0x0] 0 [0x200000400:0x133:0x0] 1 [0x240000403:0x133:0x0] 2 [0x280000403:0x133:0x0] 3 [0x2c0000403:0x133:0x0] 0 [0x200000400:0x134:0x0] 1 [0x240000403:0x134:0x0] 2 [0x280000403:0x134:0x0] 3 [0x2c0000403:0x134:0x0] 0 [0x200000400:0x135:0x0] 1 [0x240000403:0x135:0x0] 2 [0x280000403:0x135:0x0] 3 [0x2c0000403:0x135:0x0] 0 [0x200000400:0x136:0x0] 1 [0x240000403:0x136:0x0] 2 [0x280000403:0x136:0x0] 3 [0x2c0000403:0x136:0x0] 0 [0x200000400:0x137:0x0] 1 [0x240000403:0x137:0x0] 2 [0x280000403:0x137:0x0] 3 [0x2c0000403:0x137:0x0] 0 [0x200000400:0x138:0x0] 1 [0x240000403:0x138:0x0] 2 [0x280000403:0x138:0x0] 3 [0x2c0000403:0x138:0x0] 0 [0x200000400:0x139:0x0] 1 [0x240000403:0x139:0x0] 2 [0x280000403:0x139:0x0] 3 [0x2c0000403:0x139:0x0] 0 [0x200000400:0x13a:0x0] 1 [0x240000403:0x13a:0x0] 2 [0x280000403:0x13a:0x0] 3 [0x2c0000403:0x13a:0x0] 0 [0x200000400:0x13b:0x0] 1 [0x240000403:0x13b:0x0] 2 [0x280000403:0x13b:0x0] 3 [0x2c0000403:0x13b:0x0] 0 [0x200000400:0x13c:0x0] 1 [0x240000403:0x13c:0x0] 2 [0x280000403:0x13c:0x0] 3 [0x2c0000403:0x13c:0x0] 0 [0x200000400:0x13d:0x0] 1 [0x240000403:0x13d:0x0] 2 [0x280000403:0x13d:0x0] 3 [0x2c0000403:0x13d:0x0] 0 [0x200000400:0x13e:0x0] 1 [0x240000403:0x13e:0x0] 2 [0x280000403:0x13e:0x0] 3 [0x2c0000403:0x13e:0x0] 0 [0x200000400:0x13f:0x0] 1 [0x240000403:0x13f:0x0] 2 [0x280000403:0x13f:0x0] 3 [0x2c0000403:0x13f:0x0] 0 [0x200000400:0x140:0x0] 1 [0x240000403:0x140:0x0] 2 [0x280000403:0x140:0x0] 3 [0x2c0000403:0x140:0x0] 0 [0x200000400:0x141:0x0] 1 [0x240000403:0x141:0x0] 2 [0x280000403:0x141:0x0] 3 [0x2c0000403:0x141:0x0] 0 [0x200000400:0x142:0x0] 1 [0x240000403:0x142:0x0] 2 [0x280000403:0x142:0x0] 3 [0x2c0000403:0x142:0x0] 0 [0x200000400:0x143:0x0] 1 [0x240000403:0x143:0x0] 2 [0x280000403:0x143:0x0] 3 [0x2c0000403:0x143:0x0] 0 [0x200000400:0x144:0x0] 1 [0x240000403:0x144:0x0] 2 [0x280000403:0x144:0x0] 3 [0x2c0000403:0x144:0x0] 0 [0x200000400:0x145:0x0] 1 [0x240000403:0x145:0x0] 2 [0x280000403:0x145:0x0] 3 [0x2c0000403:0x145:0x0] 0 [0x200000400:0x146:0x0] 1 [0x240000403:0x146:0x0] 2 [0x280000403:0x146:0x0] 3 [0x2c0000403:0x146:0x0] 0 [0x200000400:0x147:0x0] 1 [0x240000403:0x147:0x0] 2 [0x280000403:0x147:0x0] 3 [0x2c0000403:0x147:0x0] 0 [0x200000400:0x148:0x0] 1 [0x240000403:0x148:0x0] 2 [0x280000403:0x148:0x0] 3 [0x2c0000403:0x148:0x0] 0 [0x200000400:0x149:0x0] 1 [0x240000403:0x149:0x0] 2 [0x280000403:0x149:0x0] 3 [0x2c0000403:0x149:0x0] 0 [0x200000400:0x14a:0x0] 1 [0x240000403:0x14a:0x0] 2 [0x280000403:0x14a:0x0] 3 [0x2c0000403:0x14a:0x0] 0 [0x200000400:0x14b:0x0] 1 [0x240000403:0x14b:0x0] 2 [0x280000403:0x14b:0x0] 3 [0x2c0000403:0x14b:0x0] 0 [0x200000400:0x14c:0x0] 1 [0x240000403:0x14c:0x0] 2 [0x280000403:0x14c:0x0] 3 [0x2c0000403:0x14c:0x0] 0 [0x200000400:0x14d:0x0] 1 [0x240000403:0x14d:0x0] 2 [0x280000403:0x14d:0x0] 3 [0x2c0000403:0x14d:0x0] 0 [0x200000400:0x14e:0x0] 1 [0x240000403:0x14e:0x0] 2 [0x280000403:0x14e:0x0] 3 [0x2c0000403:0x14e:0x0] 0 [0x200000400:0x14f:0x0] 1 [0x240000403:0x14f:0x0] 2 [0x280000403:0x14f:0x0] 3 [0x2c0000403:0x14f:0x0] 0 [0x200000400:0x150:0x0] 1 [0x240000403:0x150:0x0] 2 [0x280000403:0x150:0x0] 3 [0x2c0000403:0x150:0x0] 0 [0x200000400:0x151:0x0] 1 [0x240000403:0x151:0x0] 2 [0x280000403:0x151:0x0] 3 [0x2c0000403:0x151:0x0] 0 [0x200000400:0x152:0x0] 1 [0x240000403:0x152:0x0] 2 [0x280000403:0x152:0x0] 3 [0x2c0000403:0x152:0x0] 0 [0x200000400:0x153:0x0] 1 [0x240000403:0x153:0x0] 2 [0x280000403:0x153:0x0] 3 [0x2c0000403:0x153:0x0] 0 [0x200000400:0x154:0x0] 1 [0x240000403:0x154:0x0] 2 [0x280000403:0x154:0x0] 3 [0x2c0000403:0x154:0x0] 0 [0x200000400:0x155:0x0] 1 [0x240000403:0x155:0x0] 2 [0x280000403:0x155:0x0] 3 [0x2c0000403:0x155:0x0] 0 [0x200000400:0x156:0x0] 1 [0x240000403:0x156:0x0] 2 [0x280000403:0x156:0x0] 3 [0x2c0000403:0x156:0x0] 0 [0x200000400:0x157:0x0] 1 [0x240000403:0x157:0x0] 2 [0x280000403:0x157:0x0] 3 [0x2c0000403:0x157:0x0] 0 [0x200000400:0x158:0x0] 1 [0x240000403:0x158:0x0] 2 [0x280000403:0x158:0x0] 3 [0x2c0000403:0x158:0x0] 0 [0x200000400:0x159:0x0] 1 [0x240000403:0x159:0x0] 2 [0x280000403:0x159:0x0] 3 [0x2c0000403:0x159:0x0] 0 [0x200000400:0x15a:0x0] 1 [0x240000403:0x15a:0x0] 2 [0x280000403:0x15a:0x0] 3 [0x2c0000403:0x15a:0x0] 0 [0x200000400:0x15b:0x0] 1 [0x240000403:0x15b:0x0] 2 [0x280000403:0x15b:0x0] 3 [0x2c0000403:0x15b:0x0] 0 [0x200000400:0x15c:0x0] 1 [0x240000403:0x15c:0x0] 2 [0x280000403:0x15c:0x0] 3 [0x2c0000403:0x15c:0x0] 0 [0x200000400:0x15d:0x0] 1 [0x240000403:0x15d:0x0] 2 [0x280000403:0x15d:0x0] 3 [0x2c0000403:0x15d:0x0] 0 [0x200000400:0x15e:0x0] 1 [0x240000403:0x15e:0x0] 2 [0x280000403:0x15e:0x0] 3 [0x2c0000403:0x15e:0x0] 0 [0x200000400:0x15f:0x0] 1 [0x240000403:0x15f:0x0] 2 [0x280000403:0x15f:0x0] 3 [0x2c0000403:0x15f:0x0] 0 [0x200000400:0x160:0x0] 1 [0x240000403:0x160:0x0] 2 [0x280000403:0x160:0x0] 3 [0x2c0000403:0x160:0x0] 0 [0x200000400:0x161:0x0] 1 [0x240000403:0x161:0x0] 2 [0x280000403:0x161:0x0] 3 [0x2c0000403:0x161:0x0] 0 [0x200000400:0x162:0x0] 1 [0x240000403:0x162:0x0] 2 [0x280000403:0x162:0x0] 3 [0x2c0000403:0x162:0x0] 0 [0x200000400:0x163:0x0] 1 [0x240000403:0x163:0x0] 2 [0x280000403:0x163:0x0] 3 [0x2c0000403:0x163:0x0] 0 [0x200000400:0x164:0x0] 1 [0x240000403:0x164:0x0] 2 [0x280000403:0x164:0x0] 3 [0x2c0000403:0x164:0x0] 0 [0x200000400:0x165:0x0] 1 [0x240000403:0x165:0x0] 2 [0x280000403:0x165:0x0] 3 [0x2c0000403:0x165:0x0] 0 [0x200000400:0x166:0x0] 1 [0x240000403:0x166:0x0] 2 [0x280000403:0x166:0x0] 3 [0x2c0000403:0x166:0x0] 0 [0x200000400:0x167:0x0] 1 [0x240000403:0x167:0x0] 2 [0x280000403:0x167:0x0] 3 [0x2c0000403:0x167:0x0] 0 [0x200000400:0x168:0x0] 1 [0x240000403:0x168:0x0] 2 [0x280000403:0x168:0x0] 3 [0x2c0000403:0x168:0x0] 0 [0x200000400:0x169:0x0] 1 [0x240000403:0x169:0x0] 2 [0x280000403:0x169:0x0] 3 [0x2c0000403:0x169:0x0] 0 [0x200000400:0x16a:0x0] 1 [0x240000403:0x16a:0x0] 2 [0x280000403:0x16a:0x0] 3 [0x2c0000403:0x16a:0x0] 0 [0x200000400:0x16b:0x0] 1 [0x240000403:0x16b:0x0] 2 [0x280000403:0x16b:0x0] 3 [0x2c0000403:0x16b:0x0] 0 [0x200000400:0x16c:0x0] 1 [0x240000403:0x16c:0x0] 2 [0x280000403:0x16c:0x0] 3 [0x2c0000403:0x16c:0x0] 0 [0x200000400:0x16d:0x0] 1 [0x240000403:0x16d:0x0] 2 [0x280000403:0x16d:0x0] 3 [0x2c0000403:0x16d:0x0] 0 [0x200000400:0x16e:0x0] 1 [0x240000403:0x16e:0x0] 2 [0x280000403:0x16e:0x0] 3 [0x2c0000403:0x16e:0x0] 0 [0x200000400:0x16f:0x0] 1 [0x240000403:0x16f:0x0] 2 [0x280000403:0x16f:0x0] 3 [0x2c0000403:0x16f:0x0] 0 [0x200000400:0x170:0x0] 1 [0x240000403:0x170:0x0] 2 [0x280000403:0x170:0x0] 3 [0x2c0000403:0x170:0x0] 0 [0x200000400:0x171:0x0] 1 [0x240000403:0x171:0x0] 2 [0x280000403:0x171:0x0] 3 [0x2c0000403:0x171:0x0] 0 [0x200000400:0x172:0x0] 1 [0x240000403:0x172:0x0] 2 [0x280000403:0x172:0x0] 3 [0x2c0000403:0x172:0x0] 0 [0x200000400:0x173:0x0] 1 [0x240000403:0x173:0x0] 2 [0x280000403:0x173:0x0] 3 [0x2c0000403:0x173:0x0] 0 [0x200000400:0x174:0x0] 1 [0x240000403:0x174:0x0] 2 [0x280000403:0x174:0x0] 3 [0x2c0000403:0x174:0x0] 0 [0x200000400:0x175:0x0] 1 [0x240000403:0x175:0x0] 2 [0x280000403:0x175:0x0] 3 [0x2c0000403:0x175:0x0] 0 [0x200000400:0x176:0x0] 1 [0x240000403:0x176:0x0] 2 [0x280000403:0x176:0x0] 3 [0x2c0000403:0x176:0x0] 0 [0x200000400:0x177:0x0] 1 [0x240000403:0x177:0x0] 2 [0x280000403:0x177:0x0] 3 [0x2c0000403:0x177:0x0] 0 [0x200000400:0x178:0x0] 1 [0x240000403:0x178:0x0] 2 [0x280000403:0x178:0x0] 3 [0x2c0000403:0x178:0x0] 0 [0x200000400:0x179:0x0] 1 [0x240000403:0x179:0x0] 2 [0x280000403:0x179:0x0] 3 [0x2c0000403:0x179:0x0] 0 [0x200000400:0x17a:0x0] 1 [0x240000403:0x17a:0x0] 2 [0x280000403:0x17a:0x0] 3 [0x2c0000403:0x17a:0x0] 0 [0x200000400:0x17b:0x0] 1 [0x240000403:0x17b:0x0] 2 [0x280000403:0x17b:0x0] 3 [0x2c0000403:0x17b:0x0] 0 [0x200000400:0x17c:0x0] 1 [0x240000403:0x17c:0x0] 2 [0x280000403:0x17c:0x0] 3 [0x2c0000403:0x17c:0x0] 0 [0x200000400:0x17d:0x0] 1 [0x240000403:0x17d:0x0] 2 [0x280000403:0x17d:0x0] 3 [0x2c0000403:0x17d:0x0] 0 [0x200000400:0x17e:0x0] 1 [0x240000403:0x17e:0x0] 2 [0x280000403:0x17e:0x0] 3 [0x2c0000403:0x17e:0x0] 0 [0x200000400:0x17f:0x0] 1 [0x240000403:0x17f:0x0] 2 [0x280000403:0x17f:0x0] 3 [0x2c0000403:0x17f:0x0] 0 [0x200000400:0x180:0x0] 1 [0x240000403:0x180:0x0] 2 [0x280000403:0x180:0x0] 3 [0x2c0000403:0x180:0x0] 0 [0x200000400:0x181:0x0] 1 [0x240000403:0x181:0x0] 2 [0x280000403:0x181:0x0] 3 [0x2c0000403:0x181:0x0] 0 [0x200000400:0x182:0x0] 1 [0x240000403:0x182:0x0] 2 [0x280000403:0x182:0x0] 3 [0x2c0000403:0x182:0x0] 0 [0x200000400:0x183:0x0] 1 [0x240000403:0x183:0x0] 2 [0x280000403:0x183:0x0] 3 [0x2c0000403:0x183:0x0] Resetting fail_loc on all nodes...done. PASS 300k (7s) == sanity test complete, duration 9 sec == 01:30:11 (1433320211)
Oh, yes, this multiple stripes is only for testing purpose, IMHO. i.e. verifying big stripes on local environment.
I had 128 MDTs though, so it didn't need to create more than one stripe on a single MDT.
Ah, I probably already fixed this problem last night. I just updated the patch to make DNE create multiple stripes on single MDT as Andreas suggested. Right now, I am able to create the striped directory with 512 stripes on my local environment. I will push the patch soon. Thanks for testing.
With build 32529, I was able to create striped directories up to 84, but at 96 I hit another assertion on an MDS:
Jun 3 18:22:19 mdt03 kernel: LustreError: 1745:0:(llog_osd.c:147:llog_osd_pad()) ASSERTION( len >= (24) && (len & 0x7) == 0 ) failed: Jun 3 18:22:19 mdt03 kernel: LustreError: 1745:0:(llog_osd.c:147:llog_osd_pad()) LBUG Jun 3 18:22:19 mdt03 kernel: Pid: 1745, comm: mdt01_002 Jun 3 18:22:19 mdt03 kernel: Jun 3 18:22:19 mdt03 kernel: Call Trace: Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0145875>] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0145e77>] lbug_with_loc+0x47/0xb0 [libcfs] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa023d48e>] llog_osd_write_rec+0x167e/0x17a0 [obdclass] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa022d220>] llog_write_rec+0xb0/0x270 [obdclass] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0235a6f>] llog_cat_add_rec+0x9f/0x460 [obdclass] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa022d039>] llog_add+0x89/0x1c0 [obdclass] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa051b17c>] sub_updates_write+0x9dc/0x1380 [ptlrpc] Jun 3 18:22:19 mdt03 kernel: [<ffffffff8115cdbc>] ? __vunmap+0x9c/0x120 Jun 3 18:22:19 mdt03 kernel: [<ffffffffa051c26c>] top_trans_stop+0x74c/0xb30 [ptlrpc] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0c5b0ff>] ? lod_attr_set+0x12f/0xab0 [lod] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa029ce40>] ? lu_ucred+0x20/0x30 [obdclass] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0c449bf>] lod_trans_stop+0x2bf/0x330 [lod] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0ce867d>] mdd_trans_stop+0x1d/0xb0 [mdd] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0cd7a8c>] mdd_create+0x13ac/0x1760 [mdd] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0b9080c>] ? mdt_version_save+0x8c/0x1a0 [mdt] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0b94bec>] mdt_reint_create+0xbcc/0xce0 [mdt] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa029ce40>] ? lu_ucred+0x20/0x30 [obdclass] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0b70dd5>] ? mdt_ucred+0x15/0x20 [mdt] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0b8baec>] ? mdt_root_squash+0x2c/0x3f0 [mdt] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa04cbd42>] ? __req_capsule_get+0x162/0x6e0 [ptlrpc] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0b8fb9d>] mdt_reint_rec+0x5d/0x200 [mdt] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0b74fab>] mdt_reint_internal+0x4cb/0x7a0 [mdt] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa0b7571b>] mdt_reint+0x6b/0x120 [mdt] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa05074be>] tgt_request_handle+0x95e/0x10b0 [ptlrpc] Jun 3 18:22:19 mdt03 kernel: [<ffffffffa04b6c31>] ptlrpc_main+0xe41/0x1970 [ptlrpc] Jun 3 18:22:19 mdt03 kernel: [<ffffffff81014959>] ? sched_clock+0x9/0x10 Jun 3 18:22:19 mdt03 kernel: [<ffffffff81060c3f>] ? finish_task_switch+0x4f/0xf0 Jun 3 18:22:19 mdt03 kernel: [<ffffffffa04b5df0>] ? ptlrpc_main+0x0/0x1970 [ptlrpc] Jun 3 18:22:19 mdt03 kernel: [<ffffffff8109e71e>] kthread+0x9e/0xc0 Jun 3 18:22:19 mdt03 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20 Jun 3 18:22:19 mdt03 kernel: [<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b Jun 3 18:22:19 mdt03 kernel: [<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6 Jun 3 18:22:19 mdt03 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20 Jun 3 18:22:19 mdt03 kernel:
Actually you can increase the RPC to 4MB so in theory it can be 8000 stripes. It is very possible that we might have 16MB RPCs a few years for now.
Andreas: the maximum stripes is now limited by 1MB RPC size, which includes the updates to create the single stripe (about 4k, at most 8k, plus some RPC overhead) and the update logs (1016k at most, let's say 1000k for safe).
As we calculated before, each stripe costs about 390 bytes, and extra update records is about 2k. so (1000k - 2k)/390 = 2620 stripes.
There might be some other extra overhead, for example if there are ACL or default LOV etc, then each stripe creation might include more updates. let's say 3 more xattr set updates.(so it is 32 * 3 = 96 bytes), then the total bytes of creation single stripe will be 390 + 96 = 486 (Note: in setxattr, EA itself will be in common params area, because it will be the same for all stripes, so it will not repeat). Then the total stripes will be (1000k - 2k) / 486 = 2102 stripes.
So I would think the maximum stripes will be 2000 in theory. It would be interesting, if we can create 512 or 1024 stripes to see if it can works.
Andreas: Sorry, no independent patch yet. I was waiting Oleg to land some of my patches, (he is testing them right now, hopefully tomorrow they can land) then I will push those patches independently, which actually includes 3 patches.
Di, I was going to look at your patch to change the OUT logging change, but I only see the combined change 13942. Is there a different patch that contains only the changes for the redo logs?
wangdi (di.wang@intel.com) uploaded a new patch: http://review.whamcloud.com/15161
Subject:
LU-6602llog: increase llog chunk sizeProject: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 99b0ee58d13a032feb4d580b23dc46771add65d2