Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Lustre 2.8.0, Lustre 2.9.0
-
3
-
9223372036854775807
Description
Recently we have some Intel Xeon-Phi Knights Landing boxes up and running, but we have applications crashing with SIGBUS.
Steps to reproduce:
Put the Xeon-Phi MCDRAM to flat mode
[15:10:32 root@r3641:~] # cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 87 model name : Intel(R) Xeon Phi(TM) CPU 7230 @ 1.30GHz stepping : 1 microcode : 0x130 cpu MHz : 1177.769 cache size : 1024 KB physical id : 0 siblings : 256 core id : 0 cpu cores : 64 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms avx512f rdseed adx avx512pf avx512er avx512cd xsaveopt bugs : bogomips : 2600.01 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: [...thousands of lines omitted...] [15:10:36 root@r3641:~] # numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 node 0 size: 193304 MB node 0 free: 190405 MB node 1 cpus: node 1 size: 16125 MB node 1 free: 15889 MB node distances: node 0 1 0: 10 31 1: 31 10
We have two numa nodes here, node 0 has all the cpus and memory,
node 1 is the MCDRAM.
Run the stream benchmark: https://www.cs.virginia.edu/stream/
on numa node 1 with 64 threads, the stream binary itself is on the lustre filesystem.
[15:12:45 root@r3641:~] # export OMP_NUM_THREADS=64 [15:15:10 root@r3641:~] # numactl -m 1 /system/Benchmarks/stream/5.10-Intel-Opt/stream-nci-icc.exe ------------------------------------------------------------- STREAM version $Revision: 5.10 $ ------------------------------------------------------------- This system uses 8 bytes per array element. ------------------------------------------------------------- Array size = 6405100800 (elements), Offset = 0 (elements) Memory per array = 48867.0 MiB (= 47.7 GiB). Total memory required = 146601.1 MiB (= 143.2 GiB). Each kernel will be executed 10 times. The *best* time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth. ------------------------------------------------------------- Number of Threads requested = 64 Number of Threads counted = 64 Bus error
and it got killed with SIGBUS
[ 1472.651904] potentially unexpected fatal signal 7. [ 1472.656784] CPU: 28 PID: 10226 Comm: stream-nci-icc. Tainted: G O 4.4.16-raijin.el6.x86_64 #1 [ 1472.666492] Hardware name: Supermicro C2112-4KL1/MBD-K1SPI-P, BIOS 1.0a 06/11/2016 [ 1472.674120] task: ffff883421770000 ti: ffff882ed0f6c000 task.ti: ffff882ed0f6c000 [ 1472.681655] RIP: 0033:[<00000000004023ee>] [<00000000004023ee>] 0x4023ee [ 1472.688527] RSP: 002b:00007fec4fbfcd80 EFLAGS: 00010206 [ 1472.693892] RAX: 00000017dcc1b600 RBX: 000000000000001c RCX: 0000000000a33518 [ 1472.701087] RDX: 0000000bee910e00 RSI: 00000000a706a930 RDI: 0000000000606600 [ 1472.708275] RBP: 00007fec4fbfd0f0 R08: 0000000005f71850 R09: 0000000000000000 [ 1472.715469] R10: 00000000a706a930 R11: 00000000a7a9de40 R12: 0000000005f71854 [ 1472.722656] R13: 0000000000404880 R14: 0000000000000000 R15: 000000000000001c [ 1472.729843] FS: 00007fec4fbfe500(0000) GS:ffff882f7e180000(0000) knlGS:0000000000000000 [ 1472.738133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1472.744090] CR2: 00000000004023ee CR3: 0000003420bdf000 CR4: 00000000001406e0
It was handling a page fault on addr 4023ee which is inside the binary itself.
Turning off fast_read will make the problem go away:
lctl set_param llite.*.fast_read=0
In ll_fault0, with fast_read enabled, it calls filemap_fault() right away,
if (ll_sbi_has_fast_read(ll_i2sbi(file_inode(vma->vm_file)))) { /* do fast fault */ ll_cl_add(vma->vm_file, env, NULL, LCC_MMAP); fault_ret = filemap_fault(vma, vmf); ll_cl_remove(vma->vm_file, env); /* - If there is no error, then the page was found in cache and * uptodate; * - If VM_FAULT_RETRY is set, the page existed but failed to * lock. It will return to kernel and retry; * - Otherwise, it should try normal fault under DLM lock. */ if ((fault_ret & VM_FAULT_RETRY) || !(fault_ret & VM_FAULT_ERROR)) GOTO(out, result = 0); fault_ret = 0;b }
filemap_fault() calls find_get_page() which fails to find the page in page cache because we are under memory pressure, so filemap_fault() tries page_cache_read() which allocate a page, add it to page cache lru and then calls ll_readpage(). ll_readpage will fail for fast_read, so we fall back to normal fault in ll_fault0().
So we init a cl_io and does cl_io_loop, then we reach filemap_fault() for the second time, from vvp_io_kernel_fault()
This time we can find the page in the page cache, because we just created it in the first call to filemap_fault(). But the page is not up to date,
so we call ll_readpage() again, then sleep on the page until the page is unlocked.
page_not_uptodate: /* * Umm, take care of errors if the page isn't up-to-date. * Try to re-read it _once_. We do this synchronously, * because there really aren't any performance issues here * and we need to check for errors. */ ClearPageError(page); error = mapping->a_ops->readpage(file, page); if (!error) { wait_on_page_locked(page); if (!PageUptodate(page)) error = -EIO; } put_page(page); if (!error || error == AOP_TRUNCATED_PAGE) goto retry_find; /* Things didn't work out. Return zero to tell the mm layer so. */ shrink_readahead_size_eio(file, ra); return VM_FAULT_SIGBUS;
When ptlrpcd thread finishes the io, calls cl_page_export() to export the page and then unlocks the page, which will wake up the page fault thread waiting on the page. Before the page fault thread wakes up and continues, osc_lru_shrink kicks in because of memory pressure, it picks the just completed page, and kicks it out. when the page fault thread finally got a chance, it looks at the page and it's not up to date anymore. This leads page handler to fail with a SIGBUS.
The reason turning off fast_read works is without fast_read,
we only call filemap_fault() doing normal fault, where the first call to ll_readpage() will succeed and we will not reach the code label page_not_uptodate which does the waiting and checking for uptodate.
IMHO we should fix osc_lru_shrink, it's ok for fiemap_fault wait on the page and check if it's up to date. we need to make sure the page doesn't go away in the whole process.