|
From the symptom you described, the root cause of this problem is that the pages were still in the kernel's LRU cache after lustre dropped those pages and kernel should be able to move the page from active -> inactive -> free by zone scanning - however, this is done by a single threaded kernel daemon so the speed of generating free pages is limited.
Can you please tell me the write speed of IOR job you saw on the node?
One way to fix this problem is to remove those pages out of kernel LRU list and drop them voluntarily. I will work out a patch soon.
|
|
Right, it does appear Lustre is dropping the pages and they are sitting in the kernel pages cache. I'm unsure why the kernel cannot then free these pages through it's direct reclaim code paths. Does the speed at which we are moving pages affect this? I had assumed that GFP_KERNEL allocation would block until the direct reclaim paths had run, at which point the kernel should have been able to reclaim all of these "inactive" file pages.
Voluntarily removing pages from the kernel LRU seems like it is just dodging the real issue at hand; which is the fact that these "inactive" pages are not getting reclaimed in the direct memory reclamation code paths. Or perhaps I don't fully understand something?
I believe we've been seeing roughly 1.1 GB/s per IO node as reported by IOR for reads and writes.
Also, voluntarily removing pages doesn't address the fact that, from my understanding, we shouldn't be seeing any "active" file pages from Lustre during this test load.
|
|
It's interesting that why it couldn't get any free pages even there was lots of inactive pages in the cache. Can you please monitor the CPU usage and /proc/meminfo when you're running it next time? Also, it would be a good idea to track ll_releasepage() to see the successful rate of releasing pages by kswapd(returning 1 means success).
I also worked out a patch to verify if removing pages from kernel LRU helps. Please give it a try.
|
|
patch to remove pages from kernel LRU voluntarily.
|
|
Sure. I'll add some monitoring when I dig into this some more next week. Also, I don't really like applying arbitrary patches to our branches.. So if you don't mind, when I apply the debug patch I'll also push it to to Gerrit. It makes it easier for us to track exactly what version of patches we are running.
|
|
Pushed the debug patch here: http://review.whamcloud.com/4012
|
|
Thank you Parakash, please notice that this is a debug patch which can only help understand the problem better.
|
|
Jinshan, what are you expecting to learn from the debug patch? I installed it this morning and was still able to hit the OOM. I didn't have any scripts monitoring the CPU or memory usage, though. So the only information about the state of memory when the OOM is from the OOM message it prints to the console:
Keep in mind, we're explicitly setting min_free_kbytes to 1G as a temporary workaround for atomic page allocation failures we are seeing elsewhere.
Also, there are many console messages as a result of the debug patch:
It prints a backtrace too, but I don't think it's necessary to paste it here. My guess is the message is a result of calling page_cache_release twice on the same page, once in vvp_page_fini and again in vvp_page_fini_common..?
|
|
No, that's not due to page_release() in vvp_page_fini() as the comment said it just frees the page refcount from isolate_lru_page(). Despite the page state error, it can control the LRU page number for cache right now, as you can see it from here:
2012-09-17 11:10:35.890389
{DefaultControlEventListener}
[mmcs]
{43}
.7.0: active_file:31999 inactive_file:31929
the total memory used for file cache is around 4G.
Please post the backtrace here as I want to know if the badpage is seen at allocation or free stage.
|
|
I guess the bad page is because of page active bit is still set so I pushed patch set 2 for this. I don't know why I didn't see the problem while I was testing the patch.
Please use vmstat to monitor the memory state and top to monitor who is using most of cpu time. THanks
|
|
Here is the backtrace from the console:
|
I guess the bad page is because of page active bit is still set so I pushed patch set 2 for this. I don't know why I didn't see the problem while I was testing the patch.
This is part (if the the root cause) of the problem. We shouldn't be seeing the active bit getting set for any Lustre pages for this test load.
|
- vulcanio18.tar.gz - Logs of vmstat and meminfo of vulcanio18 over time. The script dumped those files once per second until it hit the OOM.
- vulcanio18-ib0-2012-09-17-0.eps - A graph of the free memory, active files, and inactive files from meminfo vs. time.
|
|
I need to set up the job to run on a single IO node in order to effectively watch "top" to view the CPU usage. As an alternative, I was thinking about running oprofile on the IO nodes.
|
|
Hi Parakash, I can't open the file eps and failed to convert it to jpg with an online tool. Do you run the test with patch set 2 and can you still see the bad_page message?
|
|
Here is the same image in PNG format.
|
|
The attachments are from running with patch set 2, and I did not see the bad_page messages.
|
|
from the picture, it looks all right until at time 60ish seconds, but the amount memory of active and inactive file were still around 4G. I don't know what happened at that time, maybe application is requesting a large amount of anon page - there are already large amount of anon pages in the system. Another strange thing is that the total size of using pages is far behind the system memory size, I don;t know where it goes.
After checking the kernel source code, it looks like it can't shrink inactive pages if pte referenced bit is still set on the page, but this can only happen if those pages are mmapped.
Can you please set llite.*.max_cached_mb to a lower value, says 1G or 2G and try again. I don't know if my algorithm has some problem with a PPC node. It works great on Ihara's machine.
|
from the picture, it looks all right until at time 60ish seconds, but the amount memory of active and inactive file were still around 4G. I don't know what happened at that time, maybe application is requesting a large amount of anon page - there are already large amount of anon pages in the system. Another strange thing is that the total size of using pages is far behind the system memory size, I don;t know where it goes.
Well I don't think anything is changing in regards to the anon pages. Looking at the meminfo logs, I don't see a spike in the anon pages throughout the duration of the test. I also noticed the sum of the fields in meminfo did not add up to the total amount of memory in the system. Is this normal?
After checking the kernel source code, it looks like it can't shrink inactive pages if pte referenced bit is still set on the page, but this can only happen if those pages are mmapped.
This is the interesting part.. kswapd will set the referenced bit on the page only if page_mapped returns true. It will then move the page to the active list in a later iteration if the reference bit is set.
Can you please set llite.*.max_cached_mb to a lower value, says 1G or 2G and try again. I don't know if my algorithm has some problem with a PPC node. It works great on Ihara's machine.
Sure. I'm not sure what this will tell us though.
|
|
I added another debug patch which dumps more of the page flags from /proc/fs/lustre/llite/*/dump_page_cache: http://review.whamcloud.com/4035. None of the pages are reporting as being "mapped", so I'm curious how the reference bit is getting set without page_mapped returning true.
|
|
This is the same IOR test but with "max_cache_mb=1024" as opposed to the previous setting of 4096.
- vulcanio3-ib0-2012-09-18-0.png: A graph of inactive and active file fields from /proc/meminfo vs time.
- vulcanio3-ib0-2012-09-18-0.tar.gz: The data used to generate the graph, along with vmstat data over time.
|
Well I don't think anything is changing in regards to the anon pages. Looking at the meminfo logs, I don't see a spike in the anon pages throughout the duration of the test. I also noticed the sum of the fields in meminfo did not add up to the total amount of memory in the system. Is this normal?
I think this is abnormal as I usually can sum up cache and free memory to total memory.
I added another debug patch which dumps more of the page flags from /proc/fs/lustre/llite/*/dump_page_cache: http://review.whamcloud.com/4035. None of the pages are reporting as being "mapped", so I'm curious how the reference bit is getting set without page_mapped returning true.
I was just guessing and it verified my guest was wrong, that's it.
Actually your latest run tells us lots of information, it looks like 6G was used up in 8 seconds (from 1348011717 to 1348011725) and I don;t know where it was used. Please compare the output of
{vmstat|meminfo}
_
{1348011717|1348011725}
.log where other fields are close to each other the only exception is that free_mem was dropped from 7G to 1G. The most interesting thing is that there is NO attempt to free lru pages directly before hitting OOM.
We need help from vm guru.
|
|
Hi Andreas, can you please take a look at the problem I mentioned above?
|
|
I'm not sure if /proc/
{meminfo,vmstat}
can be trusted. I ran the test again, with the "Remove pages from kernel LRU" patch reverted. The values printed on the console as part of the OOM message are vastly different than what is reported in the proc files.
For example, I installed a systemtap script to dump meminfo, vmstat, and lustre's page cache with the out_of_memory function is called. Here is a portion of those files compared against the console dump:
/proc/vmstat contents:
Console Log:
My assumption was that the values should be roughly the same, but this is not the case. Specifically, "inactive_file" and "nr_inactive_file" are vastly different (two orders of magnitude). Theres a similar thing with "free" and "nr_free_pages". Perhaps I'm incorrectly assuming these values are reporting the same information?
A similar discrepancy is found with the meminfo file.
/proc/meminfo contents:
Console log:
The most interesting thing is that there is NO attempt to free lru pages directly before hitting OOM.
I have been assuming the VM was going through the reclaim logic, but was just not able to free anyting because of the "active" bit being set on the Lustre pages. But I'm no longer so sure this is happening. Perhaps the real issue is the reclaim logic isn't occurring at all? I should try and verify whether or not this is happening.
|
... Perhaps I'm incorrectly assuming these values are reporting the same information?
I just checked the code, they should fetch the data from the same source(vmstat.c and __show_free_areas()). Does it mean it was changed in that short time?
|
|
I'm not sure if this is relevant, but pages just written to are taking many seconds to be reclaimed when I "echo 3 > /proc/sys/vm/drop_caches" on the system. Is this normal?
For example, I have a systemtap script monitoring ll_readpage, and that function returns 0 for many seconds before returning 1. My test case is a simple "dd" of a 1M file followed by multiple invocations of drop_caches..
In one shell:
And in the other is my systemtap monitoring:
So in this test case, it took nearly 7 seconds for the page to be reclaimed. Should I be alarmed by that? Is it because of network latency, waiting for the page to safely reach the server?
Reads on the other hand don't suffer this "lag"
|
So in this test case, it took nearly 7 seconds for the page to be reclaimed. Should I be alarmed by that? Is it because of network latency, waiting for the page to safely reach the server?
The writeback interval should be 5 seconds by default so the result is not abnormal to me.
We can verify this by running a `sync' before dropping cache, or we can print the page state in systemtap script. in ll_releasepage(), the page won't be dropped in the following scenarios:
1. Page is dirty or being written back;
2. still have extra users, that's the refcount is greater than 3(caller, page private and page cache);
3. can't allocate lu_env and keys - but this is unlikely in your case.
Do you monitor all pages and just show me the vmpage @ 0xc0000000014edaa0, or the script could only see one page?
To simplify the case, can you please try some other performance benchmark tool, for example iozone or even dd, to see if we can still hit the same issue? Moreover, we can try to revert patch LU-744 and see how things are going. Sorry I have no idea about why the memory was used up in a short period so let's try to narrow the problem.
|
|
Actualy, 7s is the interval for data written asynchronously on the OST to be reported committed on the client, if there is no more IO inflight which would report the commit.
To be specific, data written with async journal commit on the OST will schedule an OBD_PING for 7s after the write is completed, so that if no more RPCs are sent it will send the "short ping" to get back an updated last_committed value from the OST (after the 5s journal commit) and can free the RPC(s) that are waiting in case of replay and pinning the client pages.
However, this 7s interval only applies to data written in the previous few seconds and uncommitted on the OST. Writes should typically commit earlier tan this, and will piggyback the last_committed update on other in-flight writes.
If there is only a single client writing to the OST, this can be problematic. We have discussed some potential mechanisms to allow the client to request earlier transaction commits, but nothing has been implemented yet. It does seen that the Sequoia configuration will be running in a mode with only a single client per OST at any time, so this needs to be addressed.
Oleg should be able to point at the existing bug for this issue.
|
Do you monitor all pages and just show me the vmpage @ 0xc0000000014edaa0, or the script could only see one page?
I limited the output of the systemtap script to the first page it saw. Otherwise it would show ll_releasepage called all on 16 pages for the 1M file, as expected.
|
Actualy, 7s is the interval for data written asynchronously on the OST to be reported committed on the client, if there is no more IO inflight which would report the commit.
To be specific, data written with async journal commit on the OST will schedule an OBD_PING for 7s after the write is completed, so that if no more RPCs are sent it will send the "short ping" to get back an updated last_committed value from the OST (after the 5s journal commit) and can free the RPC(s) that are waiting in case of replay and pinning the client pages.
However, this 7s interval only applies to data written in the previous few seconds and uncommitted on the OST. Writes should typically commit earlier tan this, and will piggyback the last_committed update on other in-flight writes.
OK. Thanks for the clarification. In that case this doesn't appear to be related, since there will be other IO going on in the OOM case unlike my simple "dd" test above.
|
|
Good point on async commit, I realize this may be the root cause of this problem - this causes OSC LRU can't really free those pages as they are pinned by ptlrpc.
Can you please tell me how many OSTs are involved in your test, and what's the configuration of max pages per rpc?
As a workaround fix, you can try to disable async commit on the OSS and I guess it will survive your client nodes.
|
Can you please tell me how many OSTs are involved in your test, and what's the configuration of max pages per rpc?
128 OSTs for the mounted filesystem, and 16 pages per RPC.
|
To simplify the case, can you please try some other performance benchmark tool, for example iozone or even dd, to see if we can still hit the same issue?
I have been able to hit the same issue running FIO on the IO node itself, removing out the BGQ IO forwarding "stuff". I also tried "dd", but wasn't able to hit the OOM.
|
|
Here are monitoring dumps from a couple runs of FIO which hit the OOM. Each test successfully ran the first read/write iteration, and hit the OOM during the second iteration (from what I can tell, as it was transitioning to the read phase).
The two tests differ in that, test "2012-09-20-3" had the rev-id 2 of "ORI-781 vvp: Remove pages from kernel LRU" applied, and "2012-09-20-2" had that patch reverted.
As you can see, the "Inactive" file pages varies significantly between the two tests.
It's interesting to see how the first r/w iteration in each test was able to eventually evict memory and get back into a "mostly free" memory state prior to starting the second iteration.
I'm curious if Lustre is just reclaiming too "slowly", maybe a result of the "async commit" mechanics? So..
1. Writes cause pages to accumulate in "inactive" pages
2. Reads start, pushing more pages into pagecache and causing a low memory situation
3. Reclaim starts, but Lustre retuns 0 from ll_releasepage for the recently written pages
4. No memory is reclaimed as a result of (3), resulting in an OOM even in direct reclaim code paths.
But that's mere speculation at this point..?
|
|
It would be interfering to try running the OSTs with a 1s commit interval, to see if this affects the problem on the client. There is a pool tunable for ZFS, and a mount option for ldiskfs.
It is my thought that the difficulty in reclaiming memory (especially if the write load is from a single client) is due to the asynchronous journal commits on the OST. This pins the dirty pages in the client cache for up to 5s at a time.
Cheers, Andreas
|
16 pages per RPC.
Is that correct? Do you have large pages on the PPC clients, or should this be 256 pages per RPC?
|
It would be interfering to try running the OSTs with a 1s commit interval, to see if this affects the problem on the client. There is a pool tunable for ZFS, and a mount option for ldiskfs.
It is my thought that the difficulty in reclaiming memory (especially if the write load is from a single client) is due to the asynchronous journal commits on the OST. This pins the dirty pages in the client cache for up to 5s at a time.
Brian and I were thinking the same thing. I should get a chance to run that test later today.
Is that correct? Do you have large pages on the PPC clients, or should this be 256 pages per RPC?
The PPC clients have 64K pages, so in total it's the same (16*64K=1M, 256*4K=1M).
|
The two tests differ in that, test "2012-09-20-3" had the rev-id 2 of "ORI-781 vvp: Remove pages from kernel LRU" applied, and "2012-09-20-2" had that patch reverted.
As you can see, the "Inactive" file pages varies significantly between the two tests.
From the test results, I think the patch of removing pages from kernel LRU worked great as it can control the memory usage of Lustre more accurate.
For the OOM problem, this is a problem of async commit. This problem can be easily hit at the end of test, because if there is no upcoming RPCs to the OSTs, the client has to wait up to 7 seconds until an OBD_PING RPC is sent the OSTs and piggyback committed transno and unpin the caching pages. This is why ll_releasepage() couldn't free those pages because ptlrpc took an extra reference on them.
1. Writes cause pages to accumulate in "inactive" pages
2. Reads start, pushing more pages into pagecache and causing a low memory situation
3. Reclaim starts, but Lustre retuns 0 from ll_releasepage for the recently written pages
4. No memory is reclaimed as a result of (3), resulting in an OOM even in direct reclaim code paths.
I guess the following happened, at the end of writing, it accumulated lots of inactive pages but they couldn't be free as they were currently pinned by ptlrpc. After a while, another iteration started which sent RPCs and piggybacked the committed transno. This is why the free memory was restored at the beginning of test.
In theory this should be a common problem and can be easily seen on performance test. I think I still missed something in my analysis.
To verify it, can you please try to disable async commit on the OST and see if this can survive the clients?
Is that correct? Do you have large pages on the PPC clients, or should this be 256 pages per RPC?
This is an PPC node and using 64KB page size, iirc.
|
|
> To verify it, can you please try to disable async commit on the OST and see if this can survive the clients?
Yes, we can try this today. However, I expect that making the I/O synchronous will significantly degrade the OSS performance. It might be so bad that we simply don't hit the issue due to the reduced throughput at the client.
I think the more interesting test is to reduce the txg sync frequency from 5 seconds to 1 second. This should minimize the performance impact and reduce the number of pinned pages on the client. The default 5 second interval means we can easily pin 8.5GB (5s * 1.7GB/s = 8.5GB) of clean pages on the client. Dropping this to one second should limit it to 1.7GB (or so). That leaves reclaim on the client with far more eligible inactive pages to work with.
This may even improve performance by reducing the number ll_releasepage() calls which fail. In fact, it might be best to proactively informed the VM that these pages aren't reclaimable. One way to do this would be for Lustre to automatically move the pages on to the active LRU when they are submitted for writeback. This would prevent kswapd and direct reclaim from initially considering them during reclaim. They could then be moved to the inactive LRU (assuming there are no mmap() users) when Lustre releases the ptlrpc reference. Alternately, Lustre might just set PG_referenced on the page and let kswapd automatically promote it to the active LRU. As a bonus you would be able to glance at Active(file) in /proc/meminfo and get a rough idea of how many pages are in this state. This would even be consistent with the kernels definition of an active page.
This wouldn't fix this issue but it might be a nice performance improvement.
> For the OOM problem, this is a problem of async commit. This problem can be easily hit at the end of test, because if there is no upcoming RPCs to the OSTs, the client has to wait up to 7 seconds until an OBD_PING RPC is sent the OSTs and piggyback committed transno and unpin the caching pages. This is why ll_releasepage() couldn't free those pages because ptlrpc took an extra reference on them.
Yes, and no. It does seem like this is an async commit issue, although the problem doesn't occur at the end of the test. It occurs when IOR is transitioning from the write to the read phase. This is where the inactive file page spike occurs in the fio-vulcanio65-ib0-2012-09-20-2 graph.
The strange thing is that I would have expected the LU-744 changes to prevent this. If we're properly tracking all of the Lustre pages, then the right thing to do is block in ll_file_read() until one of these pinned pages (or another) gets released. We should not be allocating read pages past the defined limit, although we clearly are as shown by the spike.
Jinshan can you review the latest LU-744 patch to determine how this can happen?
Hopefully, in the short term we'll be able to just reduce the txg sync thread frequency until LU-744 can be fixed to limit this case.
|
Yes, we can try this today. However, I expect that making the I/O synchronous will significantly degrade the OSS performance. It might be so bad that we simply don't hit the issue due to the reduced throughput at the client.
well at least you can use it while we're fixing the issue.
... to reduce the txg sync frequency from 5 seconds to 1 second
Set txg sync merely won;t fix the problem, you also need to reduce the interval of PING_INTERVAL_SHORT to 1 second for example.
This may even improve performance by reducing the .... This would even be consistent with the kernels definition of an active page.
This is a good idea. The only way to exclude from kswap page scanning is to take them out of kernel LRU and mark them unevictable(so that they won't disappear from page accounting), the only problem right now is that isolate_lru_page() is not exported by kernel and this is the only interface to operate kernel LRU.
Jinshan can you review the latest LU-744 patch to determine how this can happen?
http://review.whamcloud.com/4012 is a start to solve this problem. I don't know why this didn't happen on Ihara's X86 node even it can do write much faster. Do you have a X86 node to run the same test set?
|
|
> well at least you can use it while we're fixing the issue.
Absolutely, if we need too! 
> Set txg sync merely won;t fix the problem, you also need to reduce the interval of PING_INTERVAL_SHORT to 1 second for example.
Are you sure. My expectation was that reducing the short ping time wouldn't be needed since we have a constant stream of writes/reads to the server for the last committed ID to piggyback on. This should keep the client memory usage constrained. The OOM happens after the writes complete and we are starting the reads.
> The only way to exclude from kswap page scanning is to take them out of kernel LRU and mark them unevictable
That's unfortunate. After looking at the code I do see that if shrink_page_list->try_to_release_page->ll_releasepage() fails PG_Active will get set on the page. I think this nicely explains the accumulation of pages on the active list we initially reported. So it seems the VFS should only be scanning each of these pages once. Still it would be better if we could find a way to exempt them entirely from the scan.
> Do you have a X86 node to run the same test set
Yes, in fact we regularly run the test on our x86_64 machines. The key difference here is those machines have 64GB of total memory and the ppc64 machines only have 16GB. Pinning 8GB on the x86_64 box isn't a huge deal, there's still lots of memory available. On the ppc64 box it's much more significant.
|
|
When setting 'sync_journal=1' on our OSSs we hit a kernel BUG, this was filed as ORI-783.
|
|
I was able to roll a build of ZFS with the 'zfs_txg_timeout = 1' setting and still had OOM troubles running both FIO and IOR. I only had a chance to run a single IOR, but I hit the OOM with that one run. Things were a bit better on the single IO node FIO test. I was able to run the test 5 or so iterations, but eventually hit the OOM there too. So, this setting might help, judging by the FIO test, but it doesn't appear to be an outright fix. It also didn't seem to hurt performance from what I could tell, which was an open question.
The intent of LU-744 is a good way to tackle this, by limiting the amount of data cached by Lustre, but I don't think the patch is working in its current form. Even with http://review.whamcloud.com/4012, we still see the OOMs, the only change being "Inactive" pages are no longer showing up in meminfo and vmstat (which makes me think rev-id 4012 only affects the accounting and not the actual memory usage).
|
My expectation was that reducing the short ping time wouldn't be needed since we have a constant stream of writes/reads to the server for the last committed ID to piggyback on. This should keep the client memory usage constrained. The OOM happens after the writes complete and we are starting the reads.
I think this is exactly when the problem will manifest itself. When the writes stop, there will no longer be a stream of write RPCs to update the last_committed value, so those pages just sent will be pinned in the client memory until the first ping (or other RPC) is completed after the commit.
It would be interesting to see if adding a ~10s pause between the read and write phases (for the ping to complete and pages to be freed) avoids the OOM on the client?
Oleg and I had previously discussed a potential workaround for this memory pinning problem on the client. Basically, the client would send some kind of "soft sync" flag to the OST when doing write RPCs as it begins to get low on memory (e.g. 1/2 of memory is pinned). The OST will determine whether to begin a journal commit based on the number of clients requesting the soft sync (e.g. immediately if this is the only client doing IO), and the time since its last commit.
This will dynamically shorten the sync interval on the server to avoid pinning pages longer, and the client must also reduce its ping interval while pages are pinned in its memory.
With a hard 1s commit interval on the OST, and short ping interval of 1s this should simulate the above optimization to determine if it would actually fix the problem if implemented.
|
I think this is exactly when the problem will manifest itself. When the writes stop, there will no longer be a stream of write RPCs to update the last_committed value, so those pages just sent will be pinned in the client memory until the first ping (or other RPC) is completed after the commit.
But wouldn't the write RPC piggyback the last_committed value too? Even though the writes stopped, we will then send read RPCs (at least on my write/read FIO test).
It would be interesting to see if adding a ~10s pause between the read and write phases (for the ping to complete and pages to be freed) avoids the OOM on the client?
I tried this and still saw the OOMs. But, with that said, I was able to see the OOMs under a write only test case as well. For some reason (maybe "fsync on close"?), there is a large spike in inactive file pages at the end of the write portion whether reads immediately follow it or not.
With a hard 1s commit interval on the OST, and short ping interval of 1s this should simulate the above optimization to determine if it would actually fix the problem if implemented.
I changed the sync interval in ZFS to 1s and still hit the issue. I didn't change any settings in Lustre though. Are there some tunables I can tweak to set the commit interval on the OSTs (is ZFS sync every 1s enough?) and to set the 1s short ping on the client?
|
But wouldn't the write RPC piggyback the last_committed value too? Even though the writes stopped, we will then send read RPCs (at least on my write/read FIO test).
As you can see, there is a large spike of free memory between the tests. Sometimes it can fail really depending on when read starts. Setting the interval of write and read to 10s or longer can make it easier to hit this issue.
I changed the sync interval in ZFS to 1s and still hit the issue. I didn't change any settings in Lustre though. Are there some tunables I can tweak to set the commit interval on the OSTs (is ZFS sync every 1s enough?) and to set the 1s short ping on the client?
You need to patch the client. But it's really easy, in file obi_support.h, change PING_INTERVAL_SHORT to 1 and that's it. I have no idea for zfs txg write back interval.
|
|
Jinshan, in the current state of LU-744, does it remove pages from the LRU that are waiting to be committed on the server? My expectation of the patch is given a 4G max, if 4G of write RPCs are issued to the server but uncommitted, the client would not be able to dirty any more pages until the inflight requests are committed and then removed from the client LRU. Is this not the case? It appears like the inflight data is removed from the LRU, allowing more pages to be used even though the client's max has already been exceeded and the pages are sitting in an unreclaimable state awaiting it's commit completion.
|
You need to patch the client. But it's really easy, in file obi_support.h, change PING_INTERVAL_SHORT to 1 and that's it. I have no idea for zfs txg write back interval.
OK. I'll try this out.
|
Jinshan, in the current state of LU-744, does it remove pages from the LRU that are waiting to be committed on the server? My expectation of the patch is given a 4G max, if 4G of write RPCs are issued to the server but uncommitted, the client would not be able to dirty any more pages until the inflight requests are committed and then removed from the client LRU. Is this not the case?
No, it isn't. Actually this is not a good way to go if it lacks a way to notify the OSTs to commit the data ASAP because otherwise the client may have to wait up to 7 seconds before it can allocate a new page for IO.
|
|
I accidentally hit "i" and assigned the ticket to myself. Giving it back to Jinshan.
|
No, it isn't. Actually this is not a good way to go if it lacks a way to notify the OSTs to commit the data ASAP because otherwise the client may have to wait up to 7 seconds before it can allocate a new page for IO.
What benefit does the LU-744 patch provide, then, if the client can have an unlimited amount of uncommitted data in it's cache? I guess I'm misunderstanding the purpose of the limit. Sure, we're capping the amount of pages that are dirty or in writeback; but this doesn't limit on the total amount of data Lustre pins on the system, leading to the issue we see here.
|
|
Prakash, a few reasons we haven't seen this problem in the past:
- async journal commit is relatively new
- the client couldn't fill up all of the cache in a few seconds
- most testing is done with multiple clients (less per-client bandwidth, more journal commits)
|
async journal commit is relatively new
From my understanding we've essentially been running this way on ZFS the whole time. Not because Lustre specifically supported it, but because of the way ZFS defers its txg sync until the write limit is hit or the timeout is triggered. But of course, Lustre may be doing something new which I'm unaware of, making the situation worse than before.
the client couldn't fill up all of the cache in a few seconds
I think this is the main reason we haven't seen this previously. We saw OOMs while chasing ORI-768 sporadically running the same tests, and could never really nail down a cause for them.
Now on the rebased client code, we're seeing much better rates from a single client which I believe is compounding the issue (previously the best I remember seeing was around 1.7 GB/s per IO node, now I see 2.6-2.9 GB/s as reported by ltop). So that extra 5-7.5 GB of uncommitted data (1-1.5 GB/s increase * 5s for ZFS txg sync interval) is probably stressing the VM in ways it wasn't previously.
I want to try setting the short ping interval and the zfs sync interval each to 1s to see if that alleviates the situation, but I don't think I'll be able to test that out until later this week.
most testing is done with multiple clients (less per-client bandwidth, more journal commits)
Sure. Although our internal testing hasn't changed between testing the rebased vs. non-rebased orion branch, so I'm more inclined to pin this issue on the increased per IO node bandwidth.
|
|
FWIW, we used to work on a patch which accounted pages hold for replay under NR_UNSTABLE_NFS (might still be useful) and send OST_SYNC RPCs when short of memory (we probably don't want to do that any more with ZFS). See bugzilla 23529 (patch is against 1.8) for more information.
|
|
Interesting, thanks for the pointer to that bug. Did any of those patches land on an official 1.8 release? It's hard to tell from the comments.
|
|
No, none of the patches have been landed.
|
|
Johann, you say we probably wouldn't want to send OST_SYNC RPCs with ZFS.. Why is that?
|
|
Last week at LAD12 there was some discussion of this issue which is now well understood. This isn't a new issue but do to a variety reasons, such as...
- Relatively small client memory,
- Improved client performance
- Enabled async journal commits
we're now regularly able to reproduce this on Sequoia client I/O nodes. It has existed since the async journal feature was added. My understanding is that the preferred fix goes like this:
LU-744 currently only tracks clean pages and doesn't distinguish between unstable and stable pages. Limiting clean pages on a client can be useful and this change should be merged to master. This patch is working as expected for us and others, and it's going to add some of the basic infrastructure the following suggested changes need. However, it doesn't address our issue which caused by the accumulation of unstable clean pages on the clients.
- To resolve the root cause we must limit unstable clean pages on the client. We should put together a patch (or several) which do the following:
-
- Track all clean unstable pages. For the moment, it's probably easiest to track these per filesystem as was done with
LU-744. However, long term I do feel a global limit is preferable.
- For increased visibility these pages should additionally be tracked as NR_UNSTABLE_NFS which can be seen in /proc/meminfo
- A limit on the maximum number of unstable clean pages should be set (perhaps 1/4 of memory by default). All new writes must block until there is cache space available.
- Prior to reaching this limit (perhaps 1/2 way to the limit) the client should issue a 'soft sync' to all the servers with unstable clean pages. This soft sync can be handled asynchronously and the servers may optionally ignore it if they are already in the process of a sync. The ability to ignore the sync is particularly import for the ZFS OSDs to avoid thrashing.
Further optimization is possible but this is a good start to resolve our OOM issues.
|
|
Please see: http://review.whamcloud.com/4245
|
|
Hi Prakash, can you please take a look at LU-744 the comment at "Frederik Ferner added a comment - 13/Nov/12 8:44 AM", there was a strange behavior that IO only lasted for a couple of seconds and then stalled for ~20 seconds then restarted. I suspect that is related to our patches.
|
|
Jinshan, please see my comment on LU-744. What is described in your dialogue with Frederik looks normal and expected, details are provided in my comment.
|
|
Hi Prakash, Lustre doesn't rely on balance_dirty_pages() to write back dirty pages because it has its own IO engine; also your patch should work with old server anyways.
|
Lustre doesn't rely on balance_dirty_pages() to write back dirty pages because it has its own IO engine
OK, that's fine. The kernel still calls balance_dirty_pages when IO is submitted though. For example:
By properly accounting NR_UNSTABLE_NFS, we still run into dirty page limitation in that function.
also your patch should work with old server anyways
And it does work with old servers, AFAIK. It might bump up against the dirty page limits, thus stalling any outstanding IO, but that's better than an OOM event.
|
|
Regarding Brian's comment:
- Track all clean unstable pages. For the moment, it's probably easiest to track these per filesystem as was done with
LU-744. However, long term I do feel a global limit is preferable.
- For increased visibility these pages should additionally be tracked as NR_UNSTABLE_NFS which can be seen in /proc/meminfo
- A limit on the maximum number of unstable clean pages should be set (perhaps 1/4 of memory by default). All new writes must block until there is cache space available.
- Prior to reaching this limit (perhaps 1/2 way to the limit) the client should issue a 'soft sync' to all the servers with unstable clean pages. This soft sync can be handled asynchronously and the servers may optionally ignore it if they are already in the process of a sync. The ability to ignore the sync is particularly import for the ZFS OSDs to avoid thrashing.
We've done some preliminary testing on Sequoia at scale using my LU-2139 patch stack (4245, 4374, and 4375) and stability and performance are much better as a result. Write workloads are much more sustainable and predictable, and the number of unstable pages on the system as a result of Lustre fluctuates around 2G (the point at which soft syncs are sent to the OSTs). Prior to the patch stack being applied, the number of unstable pages resulting from Lustre IO would easily grow beyond 10G causing the OOMs.
|
|
Applied the patch stack (4245, 4374, 4375) to a 2.6.32-358.0.1.el6.x86_64 kernel. Getting soft lockups when unmounting. The servers run Lustre 2.1.5.
|
|
efocht Please make sure you are using patch-set 30 of http://review.whamcloud.com/4245. The earlier patch-sets had deficiencies in them, and patch-set 29 specifically had a bug in it causing umounts to hang in ll_put_super (which looks like the problem you are having).
|
|
Hit assertion introduced by first patch in the series:
"LustreError: 17404:0:(osc_cache.c:1774:osc_dec_unstable_pages()) ASSERTION( atomic_read(&cli->cl_cache->ccc_unstable_nr) >= 0 ) failed"
See LU-3274 for more details.
|
|
Lai
Could you please take care of refreshing Jinshan's patches so they are suitable for landing to master?
Thanks
Peter
|
|
This remains a support priority to get these patches refreshed and landed but the patches are not presently ready for consideration to include in 2.5.0
|
|
I've just refreshed these three patches onto HEAD of master:
1) http://review.whamcloud.com/6284
2) http://review.whamcloud.com/4374
3) http://review.whamcloud.com/4375
It would be nice to get some feedback on them, we've been running with previous versions of these patches on Sequoia/Grove for nearly a year now.
|
|
Prakash
Thanks for the refresh. Yes, we will review these patches shortly
Peter
|
|
Remaining:
http://review.whamcloud.com/5935
|
|
All patches now landed to master
|
|
Here are the back-ported patches for Lustre b2_5 branch:
- http://review.whamcloud.com/12604 (from http://review.whamcloud.com/6284)
- http://review.whamcloud.com/12605 (from http://review.whamcloud.com/4374)
- http://review.whamcloud.com/12606 (from http://review.whamcloud.com/4375)
- http://review.whamcloud.com/12612 (from http://review.whamcloud.com/5935)
|
Generated at Sat Feb 10 01:22:41 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
fio-vulcanio65-ib0-2012-09-20-2.png
fio-vulcanio65-ib0-2012-09-20-2.tar.gz