Well, both the failure and suspected cause look surprising.
Do you mean that the fld.ko module load simply hangs on a fresh system when running "modprobre lustre"?
Module load time before was about 2-5mins, because we have large ntx values.
(options ko2iblnd ntx=125536 credits=62768 fmr_pool_size=31385)
But after the patch it takes >20mins
The fix is correct and it fixes a real bug. What this change did is exposed another problem in the ko2iblnd driver. I have to ask is your system really consuming all those credits? I don't think the IB driver queue pair depth is big enough to handle all those credits.
Hi Doug,
Can you please have a look into the issue since it relates to the LU-8057 change?
Thanks.
Joe
@Bruno Faccini: Yes, I can reproduce the problem on our freshly rebooted lustre servers by doing 'modprobe fld.'
we have >12,000 clients. We do see some servers consume all the credits.
perf top showed during module load all the time is spent in __vmalloc_node.
Samples: 748K of event 'cycles', Event count (approx.): 53812402443 Overhead Shared Object Symbol 96.21% [kernel] [k] __vmalloc_node 0.91% [kernel] [k] read_hpet 0.28% [kernel] [k] get_vmalloc_info 0.26% [kernel] [k] __write_lock_failed 0.25% [kernel] [k] __read_lock_failed 0.05% [kernel] [k] apic_timer_interrupt 0.05% [kernel] [k] _spin_lock 0.04% perf [.] dso__find_symbol 0.03% [kernel] [k] find_busiest_group 0.03% [kernel] [k] clear_page_c 0.03% [kernel] [k] page_fault 0.03% [kernel] [k] memset 0.02% [kernel] [k] rcu_process_gp_end 0.02% perf [.] perf_evsel__parse_sample 0.02% [kernel] [k] sha_transform 0.02% [kernel] [k] native_write_msr_safe
I know exactly what your problem is. We saw this problem in the lustre core some time ago and changed the OBD_ALLOC macros. The libcfs/LNet layer uses it own LIBCFS_ALLOC macros which means when the allocations are more than 2 pages in size they hit the vmalloc spinlock serialization issue. We need a fix for libcfs much like lustre had.
James, can we do that fix under this ticket?
Why not. The problem is the LIBCFS_ALLOC and FREE macros. Looking at the macros gave me a headache so no patch from me. I need to get into the right mental state to tackle it 
Any updates?
As best we can figure out, the change in LU-8057 has causes a little more memory to be used per connection. That is pushing your system over the edge.
As James has indicated, a proper fix will be to change how to allocate memory in LNet. That is going to take some time to get right as the potential to break all of LNet is pretty good.
I don't believe the fix for LU-8057 is critical for your setup. If this needs to be fixed quickly, then removing LU-8085 from your build will be the best approach. As ORNL needs LU-8085, we cannot remove it from master.
Doug,
You wrote in previous commet:
" If this needs to be fixed quickly, then removing LU-8085 from your build will be the best approach. As ORNL needs LU-8085, we cannot remove it from master."
Did you actually mean to write "LU-8057"? We do not have LU-8057 in our git repo...
Yes, sorry about that. LU-8057 is what I was referring to.
Is this still a problem
This case can be closed. Thanks.
ok thanks!