https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1328 Lustre <p>Customer running a script that calls a Java program (see attachments). Two clients panic'd.</p> <p>2012-04-13 19:21:49 +0000 <span class="error">&#91;26759.401564&#93;</span> -----------<del>[ cut here ]</del>-----------<br/> 2012-04-13 19:21:49 +0000 <span class="error">&#91;26759.424756&#93;</span> WARNING: at fs/libfs.c:363 simple_setattr+0x99/0xb0()<br/> 2012-04-13 19:21:49 +0000 <span class="error">&#91;26759.455860&#93;</span> Hardware name: ProLiant BL460c G7<br/> 2012-04-13 19:21:49 +0000 <span class="error">&#91;26759.477542&#93;</span> Modules linked in: lmv mgc lustre lquota lov osc mdc fid fld ksocklnd ptlrpc obdclass lnet lvfs libcfs ppdev 8021q garp bridge stp llc nfsd nfs lockd nfs_acl auth_rpcgss sunrpc netconsole configfs dm_crypt dm_mod crc32c ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi iptable_filter be2net ip_tables x_tables hpilo joydev hpwdt rtc_cmos psmouse rtc_core lp rtc_lib evdev parport mac_hid loop serio_raw tcp_scalable fuse virtio_blk virtio virtio_ring xenfs ext4 mbcache jbd2 xfs usbhid exportfs raid1 mptspi mptsas mptscsih mptbase mpt2sas raid_class arcmsr aic94xx libsas libata scsi_transport_sas aic7xxx aic79xx scsi_transport_spi megaraid_sas cciss sd_mod sg hpsa scsi_mod uhci_hcd ehci_hcd <span class="error">&#91;last unloaded: libcfs&#93;</span><br/> 2012-04-13 19:21:49 +0000 <span class="error">&#91;26759.830142&#93;</span> Pid: 11122, comm: java Not tainted 2.6.38.2-ts4 #11<br/> 2012-04-13 19:21:49 +0000 <span class="error">&#91;26759.860423&#93;</span> Call Trace:<br/> 2012-04-13 19:21:49 +0000 <span class="error">&#91;26759.872545&#93;</span> <span class="error">&#91;&lt;ffffffff8105d10f&gt;&#93;</span> ? warn_slowpath_common+0x7f/0xc0<br/> 2012-04-13 19:21:49 +0000 <span class="error">&#91;26759.903121&#93;</span> <span class="error">&#91;&lt;ffffffff8105d16a&gt;&#93;</span> ? warn_slowpath_null+0x1a/0x20<br/> 2012-04-13 19:21:49 +0000 <span class="error">&#91;26759.933790&#93;</span> <span class="error">&#91;&lt;ffffffff8115bea9&gt;&#93;</span> ? simple_setattr+0x99/0xb0<br/> 2012-04-13 19:21:50 +0000 <span class="error">&#91;26759.962234&#93;</span> <span class="error">&#91;&lt;ffffffff8115be10&gt;&#93;</span> ? simple_setattr+0x0/0xb0<br/> 2012-04-13 19:21:50 +0000 <span class="error">&#91;26759.990161&#93;</span> <span class="error">&#91;&lt;ffffffffa1200af6&gt;&#93;</span> ? ll_md_setattr+0x3e6/0x840 <span class="error">&#91;lustre&#93;</span><br/> 2012-04-13 19:21:50 +0000 <span class="error">&#91;26760.024040&#93;</span> <span class="error">&#91;&lt;ffffffffa12011b4&gt;&#93;</span> ? ll_setattr_raw+0x264/0xe40 <span class="error">&#91;lustre&#93;</span><br/> 2012-04-13 19:21:50 +0000 <span class="error">&#91;26760.056779&#93;</span> <span class="error">&#91;&lt;ffffffffa0ba3126&gt;&#93;</span> ? cfs_hash_del+0xa6/0x1d0 <span class="error">&#91;libcfs&#93;</span><br/> 2012-04-13 19:21:50 +0000 <span class="error">&#91;26760.089867&#93;</span> <span class="error">&#91;&lt;ffffffffa1201ded&gt;&#93;</span> ? ll_setattr+0x5d/0x100 <span class="error">&#91;lustre&#93;</span><br/> 2012-04-13 19:21:50 +0000 <span class="error">&#91;26760.120906&#93;</span> <span class="error">&#91;&lt;ffffffff81153761&gt;&#93;</span> ? notify_change+0x161/0x2c0<br/> 2012-04-13 19:21:50 +0000 <span class="error">&#91;26760.150011&#93;</span> <span class="error">&#91;&lt;ffffffff811383d1&gt;&#93;</span> ? do_truncate+0x61/0x90<br/> 2012-04-13 19:21:50 +0000 <span class="error">&#91;26760.176623&#93;</span> <span class="error">&#91;&lt;ffffffff8113862d&gt;&#93;</span> ? sys_ftruncate+0xdd/0xf0<br/> 2012-04-13 19:21:50 +0000 <span class="error">&#91;26760.205178&#93;</span> <span class="error">&#91;&lt;ffffffff8100bfc2&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b<br/> 2012-04-13 19:21:50 +0000 <span class="error">&#91;26760.235981&#93;</span> --<del>[ end trace 402d40ca74c5ea86 ]</del>--</p> <p>On one OSS, about an hour before the client crash, I saw this:</p> <p>Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: Lustre: Service thread pid 16292 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: Pid: 16292, comm: ll_ost_io_254<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel:<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: Call Trace:<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa026f1a9&gt;&#93;</span> ? LNetGet+0x3e9/0x830 <span class="error">&#91;lnet&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa02699c5&gt;&#93;</span> ? LNetMDBind+0x135/0x490 <span class="error">&#91;lnet&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff8104fdef&gt;&#93;</span> ? lock_timer_base+0x2b/0x4f<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff8104febd&gt;&#93;</span> ? try_to_del_timer_sync+0xaa/0xb7<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff814aa778&gt;&#93;</span> schedule_timeout+0x1c6/0x1ee<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff810500ce&gt;&#93;</span> ? process_timeout+0x0/0x10<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa0221351&gt;&#93;</span> cfs_waitq_timedwait+0x11/0x20 <span class="error">&#91;libcfs&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa03b417f&gt;&#93;</span> target_bulk_io+0x3af/0xa40 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa0221910&gt;&#93;</span> ? cfs_alloc+0x30/0x60 <span class="error">&#91;libcfs&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff81038bc6&gt;&#93;</span> ? default_wake_function+0x0/0x14<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa038752f&gt;&#93;</span> ost_brw_write+0x12af/0x2040 <span class="error">&#91;ost&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa03ef7cc&gt;&#93;</span> ? lustre_msg_set_timeout+0x9c/0x110 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff810c6edb&gt;&#93;</span> ? free_hot_page+0x3f/0x44<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff810c7081&gt;&#93;</span> ? __free_pages+0x5a/0x70<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa03b19c0&gt;&#93;</span> ? target_bulk_timeout+0x0/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa038e104&gt;&#93;</span> ost_handle+0x2604/0x57e0 <span class="error">&#91;ost&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa03ea29e&gt;&#93;</span> ? lustre_msg_get_opc+0x8e/0xf0 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa03f735c&gt;&#93;</span> ptlrpc_server_handle_request+0x4ec/0xfc0 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff81037e95&gt;&#93;</span> ? enqueue_task+0x7c/0x8b<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa02212ae&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa022deb0&gt;&#93;</span> ? lc_watchdog_touch+0x70/0x150 <span class="error">&#91;libcfs&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa022dfd7&gt;&#93;</span> ? lc_watchdog_disable+0x47/0x120 <span class="error">&#91;libcfs&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa03fa714&gt;&#93;</span> ? ptlrpc_wait_event+0xa4/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff81030fda&gt;&#93;</span> ? __wake_up+0x48/0x55<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa03fae7f&gt;&#93;</span> ptlrpc_main+0x53f/0x1670 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff81003ada&gt;&#93;</span> child_rip+0xa/0x20<br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffffa03fa940&gt;&#93;</span> ? ptlrpc_main+0x0/0x1670 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 13 18:00:53 ts-xxxxxxxx-04 kernel: <span class="error">&#91;&lt;ffffffff81003ad0&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>I don't know if it is related, or not.</p> <p>Attached files:</p> <p>Reproduce.java: Java program used when reproducing the problem<br/> reproduce.sh: Shell script that calls Reproduce.Java in a loop<br/> messages-mds: /var/log/messages from the MDS<br/> messages-oss-1 from the first OSS<br/> messages-oss-2 from the second OSS<br/> users*: Client output via netconsole.</p> Lustre servers are running 2.6.32-220.el6, with Lustre 2.1.1.rc4.<br/> Lustre clients are running 2.6.38.2, with special code created for this release, with <a href="http://review.whamcloud.com/#change,2170">http://review.whamcloud.com/#change,2170</a>. (patch 8) LU-1328

Failing customer's file creation test

Bug Major Resolved Fixed Lai Siyao Roger Spellman Mon, 16 Apr 2012 17:03:59 +0000 Thu, 27 Sep 2012 17:49:58 +0000 Sun, 1 Jul 2012 22:58:07 +0000 Lustre 2.1.1 0 3 <p>Lai</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Roger, the first backtrace is a warning, the panic information is lost (refer to usrs393.netconsole), is it possible to setup kdump and get crash data? or connect a serial console and get panic info? I'll try to reproduce in my setup too.</p> <p>The second backtrace shows a slow thread on OST, which should not be relative to the panic.</p> <p>/var/log/messages from system with the problem.</p> <p>From the customer:</p> <p>Attached is the appropriate portion of /var/log/messages from that date - it had more info then the netconsole because apparently the problem was in the softirq and the network driver couldn't send anymore. Unfortunately the serial connections very often don't have anything on them for whatever reason and we haven't been able to get kdump to work in our environment. I currently have another client that is locked up on probably the same issue.</p> <p>Yes, the log shows there are some network IO problem, but it's still a warning, is it possible to get the crash backtrace?</p> <p>The customer is unable to provide a crash backtrace. Are you able to reproduce the problem in-house with the provided test application?</p> <p>Not yet, I'll keep it running.</p> <p>This has been run for several days, but not produced in my test environment.</p> <p>I used `while true; do sh reproduce.sh || break; done`, and I saw a few mkdir failures (looks to be failed on -EEXIST, this may be okay), everything else looks fine.</p> <p>Customer reports:</p> <p>I was able to verify I still see this problem this morning, additionally I think I have distilled it down to an even smaller test case with a single client.</p> <p>I have a directory (generated from the test case I sent you) that has 34853 files in it. If I execute the following:</p> <p>$ lfs find &lt;dir&gt; -type f | sh -c 'while read line; do stat ${line} &lt; /dev/null &gt; /dev/null &amp; done'</p> <p>This will attempt to stat the files in parallel and I see a bunch of errors that it can't find the files. However if I stat the files sequentially, everything is ok:</p> <p>$ lfs find &lt;dir&gt; -type f | sh -c 'while read line; do stat ${line} &lt; /dev/null &gt; /dev/null; done'</p> <p>Lai, please see if you can reproduce using this method.</p> <p>Regarding the EXIST, customer responded:<br/> As far as I know we don't see EEXIST. Every time I've seen a problem it has been a FileNotFoundException caused by ENOENT. I ran the test case I gave you this morning using "strace -f" and here is a snippet of one of the logs that failed:</p> <p><span class="error">&#91;pid 22808&#93;</span> open("/mnt/lustre /filetest/usrs398/20120507-092516/d5/7/75.bin", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 79 <span class="error">&#91;pid 22808&#93;</span> fstat(79, </p> {st_mode=S_IFREG|0664, st_size=0, ...}) = 0<br/> <span class="error">&#91;pid 22808&#93;</span> fcntl(79, F_GETFD) = 0<br/> <span class="error">&#91;pid 22808&#93;</span> fcntl(79, F_SETFD, FD_CLOEXEC) = 0 <span class="error">&#91;pid 22808&#93;</span> write(79, "K\304\215f\361R\351\226\340Hh\32i\303\233\243t\20\301\34\313A\243\254\224\n\262\211^\202\215\20"..., 1024) = 1024 <span class="error">&#91;pid 22808&#93;</span> open("/mnt/lustre/ filetest/usrs398/20120507-092516/d5/7/76.bin", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 80 <span class="error">&#91;pid 22808&#93;</span> fstat(80, {st_mode=S_IFREG|0664, st_size=0, ...} <p>) = 0<br/> <span class="error">&#91;pid 22808&#93;</span> fcntl(80, F_GETFD) = 0<br/> <span class="error">&#91;pid 22808&#93;</span> fcntl(80, F_SETFD, FD_CLOEXEC) = 0 <span class="error">&#91;pid 22808&#93;</span> write(80, "K\304\215f\361R\351\226\340Hh\32i\303\233\243t\20\301\34\313A\243\254\224\n\262\211^\202\215\20"..., 1024) = 1024 <span class="error">&#91;pid 22808&#93;</span> open("/mnt/lustre/ filetest/usrs398/20120507-092516/d5/7/77.bin", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 ENOENT (No such file or directory) <span class="error">&#91;pid 22808&#93;</span> lseek(3, 49747007, SEEK_SET) = 49747007 <span class="error">&#91;pid 22808&#93;</span> read(3, "PK\3\4\n\0\0\0\0\0I\nj?0\341^$j\2\0\0j\2\0\0#\0\0\0", 30) = 30 <span class="error">&#91;pid 22808&#93;</span> lseek(3, 49747072, SEEK_SET) = 49747072 <span class="error">&#91;pid 22808&#93;</span> read(3, "\312\376\272\276\0\0\0001\0\"\10\0\4\10\0\5\10\0\n\1\0\0\1\0\2 (\1\0\24()"..., 618) = 618 <span class="error">&#91;pid 22808&#93;</span> write(2, "Exception in thread \"main\" ", 27Exception in thread "main" ) = 27 <span class="error">&#91;pid 22808&#93;</span> write(2, "java.io.FileNotFoundException: /"..., 127java.io.FileNotFoundException: /mnt/lustre/ filetest/usrs398/20120507-092516/d5/7/77.bin (No such file or directory)) = 127<br/> <span class="error">&#91;pid 22808&#93;</span> write(2, "\n", 1) = 1<br/> <span class="error">&#91;pid 22808&#93;</span> write(2, "\tat java.io.FileOutputStream.ope"..., 48 at java.io.FileOutputStream.open(Native Method)) = 48<br/> <span class="error">&#91;pid 22808&#93;</span> write(2, "\n", 1) = 1<br/> <span class="error">&#91;pid 22808&#93;</span> write(2, "\tat java.io.FileOutputStream.&lt;in"..., 62 at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:194)) = 62<br/> <span class="error">&#91;pid 22808&#93;</span> write(2, "\n", 1) = 1<br/> <span class="error">&#91;pid 22808&#93;</span> write(2, "\tat java.io.FileOutputStream.&lt;in"..., 62 at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:145)) = 62<br/> <span class="error">&#91;pid 22808&#93;</span> write(2, "\n", 1) = 1<br/> <span class="error">&#91;pid 22808&#93;</span> write(2, "\tat Reproduce.main(Reproduce.jav"..., 37 at Reproduce.main(Reproduce.java:31)) = 37<br/> <span class="error">&#91;pid 22808&#93;</span> write(2, "\n", 1) = 1</p> <p>You can see that 2 opens succeeded with writes and the third failed. They were all writing to the same directory so that means the directory existed, just for some reason the third open returned ENOENT instead of succeeding. open(2) says:</p> <p> ENOENT O_CREAT is not set and the named file does not exist. Or, a directory component in pathname does not exist or is a dan-<br/> gling symbolic link.</p> <p>Since O_CREAT is given and all directory components exist since previous writes succeeded, there must be a bug in Lustre somewhere. What about the simple script where I tried to stat the files in parallel. Did you pass that on to Whamcloud as (on the surface) it appears to be a similar problem.</p> <p>I can't reproduce with the 'stat' script yet; maybe my test machine is not powerful enough, and there isn't a machine installed with fc15 in testlab.</p> <p>With a small tweak of your script, it can dump debug log upon error:</p> <p>lfs find &lt;dir&gt; -type f | sh -c 'while read line; do stat ${line} &lt; /dev/null &gt; /dev/null || lctl dk /tmp/`basename $line`.log &amp; done'</p> <p>Could you test this script on your environment?</p> <p>I've setup 3 nodes on testlab, 1 is server installed el6, 2 clients with kernel 2.6.38, and both clients doing stat and reproduce.sh in a loop. I have run test for half a day, but haven't reproduced yet, and I'll keep them running, and will add another client tomorrow if not reproduce yet.</p> <p>I can't reproduce over night, now there are 300k files total in test system, could you tell me how many files are in your system when the error appeared? Could you upload messages files of both client and MDS at failure time?</p> <p>Results from requested script change</p> <p>lfs find &lt;dir&gt; -type f | sh -c 'while read line; do stat ${line} &lt; /dev/null &gt; /dev/null || lctl dk /tmp/`basename $line`.log &amp; done'</p> <p>This looks to be a MDS issue about OPEN lock, I have a patch at <a href="http://review.whamcloud.com/#change,2800" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2800</a>, will you compile it against master, update MDS and test again?</p> <p>Customer asks:<br/> &gt; What version of the kernel is Whamcloud testing with? Are they using 2.6.38.2 or a newer version of 2.6.38?</p> <p>I'm testing with 2.6.38.6-26.</p> <p>Please verify that the patch only touches one file.</p> <p>Please verify that this only affects the Lustre server, not the client.</p> <p>Yes, the patch affects Lustre MDS server only, and only touches one file.</p> <p>We applied the patch. The customer reports the following:</p> <p>$ lfs find . -type f | sh -c 'while read file; do stat $file &amp; done &gt;/dev/null'<br/> stat: cannot stat `./20120507-091222/d3/3/270.bin': No such file or directory<br/> stat: cannot stat `./20120507-091222/d5/32/681.bin': No such file or directory<br/> stat: cannot stat `./20120507-091222/d5/32/1525.bin': No such file or directory<br/> stat: cannot stat `./20120507-091222/d12/0/284.bin': No such file or directory<br/> stat: cannot stat `./20120507-091222/d12/0/1037.bin': No such file or directory<br/> stat: cannot stat `./20120507-091222/d12/0/917.bin': No such file or directory<br/> stat: cannot stat `./20120507-091222/d11/32/1807.bin': No such file or directory<br/> stat: cannot stat `./20120507-091222/d10/6/2587.bin': No such file or directory<br/> stat: cannot stat `./20120507-091222/d10/6/891.bin': No such file or directory<br/> stat: cannot stat `./20120507-091222/d10/6/750.bin': No such file or directory<br/> stat: cannot stat `./20120507-091222/d7/34/151.bin': No such file or directory</p> <p>Here are some additional details:</p> <p> This is a single client with 24 cores running the test.</p> <p> "ls -l' on each file succeeds - if I perform the stat sequentially it succeeds - it only fails when I launch the stat processes in the background thus making them parallel.</p> <p> Each time I run the test, I get different files that can't be found.</p> <p>Is there any error message on MDS server?</p> <p>I have no way to debug without debug logs and messages; it would really to useful to dump debuglog upon error on both client and MDS, and it's best to enable 'info' and 'vfstrace' debug trace, you can check it by `lctl get_param debug`, if it doesn't contain 'info' or 'vfstrace', you can enable it with `lctl set_param debug=+"info vfstrace"`.</p> <p>This is the messages file from the MDS.</p> <p>&gt; I have no way to debug without debug logs and messages; it would really to useful to dump debuglog upon error on both client and MDS, and it's best to enable 'info' and 'vfstrace' debug trace, you can check it by `lctl get_param debug`, if it doesn't contain 'info' or 'vfstrace', you can enable it with `lctl set_param debug=+"info vfstrace"`.</p> <p>I will ask the customer to enable these. Are these enabled on both the client and servers?</p> <p>What command is run to get view the debug data?</p> <p>Yes, it's better to enable debug trace on both client and server. Upon error, you can run `lctl dk /tmp/create.log`, then you can check debug trace in the log file, which contains recent debug traces, and this is why this command should be run immediately after error occurs, otherwise the old debug trace is dropped.</p> <p>This file contains client.out and mds.out with the debug info you requested.</p> <p>When is the test conducted? </p> <p>I saw two lines of error messages suspicious in messages.mds, but they are at May 21 23:00:01:<br/> May 21 23:00:01 ts-xxxxxxxx-01 kernel: LustreError: 6386:0:(mdt_open.c:1472:mdt_reint_open()) open <span class="error">&#91;0x200000400:0xa:0x0&#93;</span>/(logdates-&gt;<span class="error">&#91;0x2003acec9:0x1:0x0&#93;</span>) cr_flag=01102 mode=0042775 msg_flag=0x0 failed: -21<br/> May 21 23:00:01 ts-xxxxxxxx-01 kernel: LustreError: 6386:0:(mdt_open.c:1472:mdt_reint_open()) open <span class="error">&#91;0x200000400:0xa:0x0&#93;</span>/(logdates-&gt;<span class="error">&#91;0x2003acec9:0x2:0x0&#93;</span>) cr_flag=01102 mode=0042775 msg_flag=0x0 failed: -21</p> <p>Could you upload script output along with this test? And you can use `lfs fid2path <span class="error">&#91;0x2003acec9:0x1:0x0&#93;</span>` and `lfs fid2path <span class="error">&#91;0x2003acec9:0x2:0x0&#93;</span>` to check which two files are failed to open.</p> <p>I'm moving back to this thread to continue the file issue since it had gotten merged in with the OST failure...</p> <p>I ran the test yesterday so the lines from May 21 don't pertain to the test. The two fids Lai asks about (<span class="error">&#91;0x2003acec9:0x1:0x0&#93;</span> &amp; <span class="error">&#91;0x2003acec9:0x2:0x0&#93;</span>) don't exist in the file system, but I assume they were located under the directory indicated by <span class="error">&#91;0x200000400:0xa:0x0&#93;</span> which is owned by one of our users who is more tolerant of random failures and so he hasn't been telling me about any issues he has had since he knows it is "use at your own risk" at this time.</p> <p>I have uploaded a new tarball where I ran the test again. It contains the following files:</p> <p>client.out - debug_kernel output from the client <br/> mds.out - debug_kernel output from the mds <br/> fid_mapping - mapping of fids to files for the fids found in client.out<br/> error_fid_mapping - mapping of paths to fids and OSTs for the files that weren't found typescript - the output of "script"</p> <p>The tarball is enoent-20120523.tar.gz.</p> <p>The logs show that the -ENOENT files are never .lookup or .revalidate at both client and server side, so the -ENOENT error should happen in directory name lookup/revalidate. I'm a bit suspicious of 'CWD' handling in lustre: the test script calls 'stat' which calls syscall vfs_lstat() finally, and it starts pathname lookup from CWD (Current Working Directory) if filename doesn't start with '/'.</p> <p>One way to prove this is to test like this:<br/> lfs find /mnt/lustre/filetest -type f | sh -c 'while read file; do stat $file &amp; done &gt;/dev/null'</p> <p>Then all `stat` will lookup pathname from ROOT.</p> <p>BTW, please enable more debug trace in your test: `lctl set_param debug=+"vfstrace info dlmtrace dentry inode"`.</p> <p>Customer reports:</p> <p>It didn't make a difference - I didn't think it would since this seems to be an issue with parallel access since if I run the stats sequentially it works.</p> <p>I have uploaded enoent-20120524.tar.gz:</p> <p>"client.out" and "mds.out" contain the debug output.<br/> "typescript" contains the output from the run.<br/> "fids" contains the file names and fids for the files that weren't found.</p> <p>Customer reports</p> <p>If I limit the number of CPUs it can run on, it reduces the number of failures to the point where running it on 1 cpu makes it effectively sequential:</p> <p>$ lfs find . -type f | taskset -c 0 sh -c 'while read file; do stat $file &amp; done &gt; /dev/null'</p> <p>This command eventually can't fork any more processes (I reach that limit) but there are no ENOENT errors.</p> <p>I see the same scenario with "taskset -c 0,1", and I see my first failure with "taskset -c 0-2", but it takes some time. However using "taskset -c 0,2" after some time I do see the ENOENT error. CPU 0 and CPU 1 are in different sockets and CPU 0 and CPU 2 are in the same socket. I do not see the error with 0 &amp; 12 which are the hyperthreaded pairs.</p> <p>I can run more tests next week if you want me to.</p> <p>Hmm, it looks to be a bug in ll_splice_alias():<br/> Once ll_splice_alias() found a reusable dentry for current inode, it will tries to reuse it, and dput() the just allocated dentry, but before that d_rehash() is called to hash this newly allocated dentry into dcache, with dentry-&gt;d_inode set to NULL. This originates from a kernel bug (d_rehash() used to be called by d_splice_alias() too), and actually it's planned to be removed in another patch, but we just think this line is redundant, and don't see it will cause error.</p> <p>I'll update FC15 patchless client patch later to reflect this change.</p> <p>Patch is updated: <a href="http://review.whamcloud.com/#change,2170" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2170</a>.</p> <p>Please patch it against latest master branch, and this change is for client only.</p> <p>Due to recent commit, please use patch: <a href="http://review.whamcloud.com/#change,1865" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1865</a></p> <p>Is this change just for the clients, or the servers, or both?<br/> Do I apply both the patches from 2170 and 1865, or just 1865?</p> <p>It's for client only. 2170 has been abandoned now, 1865 is the only remaining patch for FC15 patchless client support code.</p> <p>The modules will not load correctly. I get the following errors:</p> <p>Lustre: Lustre: Build Version: 2.2.54-g2fcae7b-CHANGED-2.6.32-220.el6.x86_64<br/> ptlrpc: Unknown symbol lut_boot_epoch_update<br/> ptlrpc: Unknown symbol lut_mod_exit<br/> ptlrpc: Unknown symbol lut_mod_init</p> <p>grepping through the code, I find:</p> <p>lustre-release-with-patch-9/lustre/include/lu_target.h:void lut_boot_epoch_update(struct lu_target *lut);<br/> lustre-release-with-patch-9/lustre/ldlm/ldlm_lib.c: lut_boot_epoch_update(lut);<br/> lustre-release-with-patch-9/lustre/ldlm/ldlm_lib.c: lut_boot_epoch_update(lut);<br/> lustre-release-with-patch-9/lustre/ptlrpc/target.c:void lut_boot_epoch_update(struct lu_target *lut)<br/> lustre-release-with-patch-9/lustre/ptlrpc/target.c:EXPORT_SYMBOL(lut_boot_epoch_update);</p> <p>So, that function is called from ldlm_lib.c.</p> <p>BUT ... In the ptlrpc/Makefile, target.o is not included, i.e.</p> <p>#ptlrpc_objs += target.o</p> <p>you need to update to latest master branch, then patch latest Change,1865 (it's patch 21 now).</p> <p>The problem still persists, i.e.</p> <p>May 31 12:05:21 compute-01-32 kernel: Lustre: Lustre: Build Version: 2.2.54-g9567e22-CHANGED-2.6.32-220.el6.x86_64<br/> May 31 12:05:21 compute-01-32 kernel: ptlrpc: Unknown symbol lut_boot_epoch_update<br/> May 31 12:05:21 compute-01-32 kernel: ptlrpc: Unknown symbol lut_mod_exit<br/> May 31 12:05:21 compute-01-32 kernel: ptlrpc: Unknown symbol lut_mod_init</p> <p>I think I know the cause of the problem.</p> <p>In the file lustre/ptlrpc/Makefile.in is the line</p> <p>@SERVER_TRUE@ptlrpc_objs += target.o</p> <p>After running: </p> <p>./configure --with-kernel=/usr/src/kernels/2.6.32-220.el6.x86_64 \<br/> --with-linux=/usr/src/kernels/2.6.32-220.el6.x86_64 \<br/> --disable-liblustre \<br/> --without-sysio \<br/> --disable-server</p> <p>The file lustre/ptlrpc/Makefile contains</p> <p>#ptlrpc_objs += target.o</p> <p>The file lustre/ldlm/ldlm_lib.c calls lut_boot_epoch_update(), which is defined in target.c. But, since ldlm_lib.c is compiled, but target.c is not compiled, there is an unknown symbol.</p> <p>The commands I ran to get the build are:</p> <p>git clone git://git.whamcloud.com/fs/lustre-release.git # svn checkout url<br/> cd lustre-release<br/> git fetch <a href="http://review.whamcloud.com/p/fs/lustre-release" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/p/fs/lustre-release</a> refs/changes/65/1865/21 &amp;&amp; git checkout FETCH_HEAD</p> <p>I did the same as you, but all looks well.</p> <p>And where lut_boot_epoch_update() is called in ldlm_lib.c is inside #ifdef HAVE_SERVER_SUPPORT, so that these code won't be compiled, and won't cause the unknown symbol problem.</p> <p>My error. I copied the wrong modules to the target system. The modules load fine now.</p> <p>As I understand it, this code is being tracked for landing under <a href="https://jira.whamcloud.com/browse/LU-506" title="FC15 patchless client support." class="issue-link" data-issue-key="LU-506"><del>LU-506</del></a></p> <p>Customer reports:<br/> It appears that this patch has fixed the original bug as the researcher was not able to replicate the problem, however he was able to panic nine of the clients with an LBUG. </p> <p>See attachment for the netconsole output. The following is the output from one:</p> <p>2012-06-06 11:35:51 +0000 <span class="error">&#91;4026925.936063&#93;</span> Lustre: MGC192.168.185.35@tcp: Reactivating import<br/> 2012-06-06 11:35:52 +0000 <span class="error">&#91;4026926.153312&#93;</span> Lustre: Mounted xxxx-client<br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.176640&#93;</span> LustreError: 8473:0:(cl_page.c:1026:cl_page_assume()) page@ffff880f6107ad80<span class="error">&#91;4 ffff880d5f11e498:0 ^ (null)_ffff880f6107ae40 1 0 1 ffff88103efcc028 (null) 0x0&#93;</span><br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.261214&#93;</span> LustreError: 8473:0:(cl_page.c:1026:cl_page_assume()) page@ffff880f6107ae40<span class="error">&#91;1 ffff880fafbffda8:0 ^ffff880f6107ad80_ (null) 0 0 1 (null) (null) 0x0&#93;</span><br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.346779&#93;</span> LustreError: 8473:0:(cl_page.c:1026:cl_page_assume()) vvp-page@ffff880dbfdb9640(0:0:0) vm@ffffea0033482180 60000000000086d 4:0 ffff880f6107ad80 0 lru<br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.419949&#93;</span> LustreError: 8473:0:(cl_page.c:1026:cl_page_assume()) lov-page@ffff880dbfdbdcf0<br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.462875&#93;</span> LustreError: 8473:0:(cl_page.c:1026:cl_page_assume()) osc-page@ffff880f610795a8: 1&lt; 0x845fed 2 0 - - - &gt; 2&lt; 0 0 51 0x0 0x400 | (null) ffff880e893b1fb8 ffff88176f4b3e80 ffffffffa1088200 ffff880f610795a8 &gt; 3&lt; - ffff880967a58000 0 0 0 &gt; 4&lt; 0 0 48 532799488 - | - - + - &gt; 5&lt; - - - - | 0 - - | 0 - -&gt;<br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.604818&#93;</span> LustreError: 8473:0:(cl_page.c:1026:cl_page_assume()) end page@ffff880f6107ad80<br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.647677&#93;</span> LustreError: 8473:0:(cl_page.c:1026:cl_page_assume()) pg-&gt;cp_owner == NULL<br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.688726&#93;</span> LustreError: 8473:0:(cl_page.c:1026:cl_page_assume()) ASSERTION( 0 ) failed:<br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.731216&#93;</span> LustreError: 8473:0:(cl_page.c:1026:cl_page_assume()) LBUG<br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.764574&#93;</span> Pid: 8473, comm: java<br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.782718&#93;</span><br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.782720&#93;</span> Call Trace:<br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.803973&#93;</span> <span class="error">&#91;&lt;ffffffffa0b77865&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.841227&#93;</span> <span class="error">&#91;&lt;ffffffffa0b77d97&gt;&#93;</span> lbug_with_loc+0x47/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.872993&#93;</span> <span class="error">&#91;&lt;ffffffffa0d61b50&gt;&#93;</span> cl_page_own0+0x0/0x2c0 <span class="error">&#91;obdclass&#93;</span><br/> 2012-06-06 16:08:32 +0000 <span class="error">&#91;4043242.906060&#93;</span> <span class="error">&#91;&lt;ffffffffa1196726&gt;&#93;</span> ll_prepare_write+0x86/0x170 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043242.940052&#93;</span> <span class="error">&#91;&lt;ffffffffa11aa8e8&gt;&#93;</span> ll_write_begin+0x88/0x160 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043242.973265&#93;</span> <span class="error">&#91;&lt;ffffffffa11a51cb&gt;&#93;</span> ? ll_getxattr+0xfb/0x440 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.005911&#93;</span> <span class="error">&#91;&lt;ffffffff810e75fe&gt;&#93;</span> generic_file_buffered_write+0xfe/0x250<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.039902&#93;</span> <span class="error">&#91;&lt;ffffffff810ea1a0&gt;&#93;</span> __generic_file_aio_write+0x230/0x470<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.073917&#93;</span> <span class="error">&#91;&lt;ffffffff810ea442&gt;&#93;</span> generic_file_aio_write+0x62/0xd0<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.105326&#93;</span> <span class="error">&#91;&lt;ffffffffa11bb800&gt;&#93;</span> vvp_io_write_start+0xb0/0x1e0 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.139984&#93;</span> <span class="error">&#91;&lt;ffffffffa0d6a1d2&gt;&#93;</span> cl_io_start+0x72/0x100 <span class="error">&#91;obdclass&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.172847&#93;</span> <span class="error">&#91;&lt;ffffffffa0d6d774&gt;&#93;</span> cl_io_loop+0xd4/0x160 <span class="error">&#91;obdclass&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.203854&#93;</span> <span class="error">&#91;&lt;ffffffffa1175d3e&gt;&#93;</span> ll_file_io_generic+0x3be/0x4f0 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.238796&#93;</span> <span class="error">&#91;&lt;ffffffffa1175fa0&gt;&#93;</span> ll_file_aio_write+0x130/0x1f0 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.272727&#93;</span> <span class="error">&#91;&lt;ffffffffa117634c&gt;&#93;</span> ll_file_write+0x14c/0x250 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.306367&#93;</span> <span class="error">&#91;&lt;ffffffff811390a8&gt;&#93;</span> vfs_write+0xc8/0x190<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.332757&#93;</span> <span class="error">&#91;&lt;ffffffff811398d1&gt;&#93;</span> sys_write+0x51/0x90<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.359311&#93;</span> <span class="error">&#91;&lt;ffffffff8100bfc2&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.389972&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.399097&#93;</span> Kernel panic - not syncing: LBUG<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.421020&#93;</span> Pid: 8473, comm: java Tainted: G W 2.6.38.2-ts4 #11<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.455850&#93;</span> Call Trace:<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.468809&#93;</span> <span class="error">&#91;&lt;ffffffff8145a373&gt;&#93;</span> ? panic+0x91/0x19c<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.494377&#93;</span> <span class="error">&#91;&lt;ffffffffa0b77dfb&gt;&#93;</span> ? lbug_with_loc+0xab/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.527753&#93;</span> <span class="error">&#91;&lt;ffffffffa0d61b50&gt;&#93;</span> ? cl_page_own0+0x0/0x2c0 <span class="error">&#91;obdclass&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.561570&#93;</span> <span class="error">&#91;&lt;ffffffffa1196726&gt;&#93;</span> ? ll_prepare_write+0x86/0x170 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.595489&#93;</span> <span class="error">&#91;&lt;ffffffffa11aa8e8&gt;&#93;</span> ? ll_write_begin+0x88/0x160 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.629850&#93;</span> <span class="error">&#91;&lt;ffffffffa11a51cb&gt;&#93;</span> ? ll_getxattr+0xfb/0x440 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.661785&#93;</span> <span class="error">&#91;&lt;ffffffff810e75fe&gt;&#93;</span> ? generic_file_buffered_write+0xfe/0x250<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.697719&#93;</span> <span class="error">&#91;&lt;ffffffff810ea1a0&gt;&#93;</span> ? __generic_file_aio_write+0x230/0x470<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.734746&#93;</span> <span class="error">&#91;&lt;ffffffff810ea442&gt;&#93;</span> ? generic_file_aio_write+0x62/0xd0<br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.767215&#93;</span> <span class="error">&#91;&lt;ffffffffa11bb800&gt;&#93;</span> ? vvp_io_write_start+0xb0/0x1e0 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.803208&#93;</span> <span class="error">&#91;&lt;ffffffffa0d6a1d2&gt;&#93;</span> ? cl_io_start+0x72/0x100 <span class="error">&#91;obdclass&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.836585&#93;</span> <span class="error">&#91;&lt;ffffffffa0d6d774&gt;&#93;</span> ? cl_io_loop+0xd4/0x160 <span class="error">&#91;obdclass&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.870706&#93;</span> <span class="error">&#91;&lt;ffffffffa1175d3e&gt;&#93;</span> ? ll_file_io_generic+0x3be/0x4f0 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:33 +0000 <span class="error">&#91;4043243.907105&#93;</span> <span class="error">&#91;&lt;ffffffffa1175fa0&gt;&#93;</span> ? ll_file_aio_write+0x130/0x1f0 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043243.942842&#93;</span> <span class="error">&#91;&lt;ffffffffa117634c&gt;&#93;</span> ? ll_file_write+0x14c/0x250 <span class="error">&#91;lustre&#93;</span><br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043243.977049&#93;</span> <span class="error">&#91;&lt;ffffffff811390a8&gt;&#93;</span> ? vfs_write+0xc8/0x190<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.004041&#93;</span> <span class="error">&#91;&lt;ffffffff811398d1&gt;&#93;</span> ? sys_write+0x51/0x90<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.031568&#93;</span> <span class="error">&#91;&lt;ffffffff8100bfc2&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.063216&#93;</span> -----------<del>[ cut here ]</del>-----------<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.087602&#93;</span> WARNING: at arch/x86/kernel/smp.c:118 native_smp_send_reschedule+0x54/0x60()<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.129258&#93;</span> Hardware name: ProLiant BL460c G7<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.151613&#93;</span> Modules linked in: lmv mgc lustre lov osc mdc fid fld ksocklnd ptlrpc obdclass lnet lvfs libcfs parport_pc ppdev 8021q garp bridge stp llc nfsd netconsole configfs nfs lockd nfs_acl auth_rpcgss sunrpc dm_crypt dm_mod crc32c ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi iptable_filter ip_tables x_tables hpwdt psmouse be2net joydev hpilo evdev mac_hid serio_raw rtc_cmos rtc_core rtc_lib parport loop tcp_scalable fuse virtio_blk virtio virtio_ring xenfs ext4 mbcache jbd2 xfs exportfs raid1 usbhid mptspi mptsas mptscsih mptbase mpt2sas raid_class arcmsr aic94xx libsas libata scsi_transport_sas aic7xxx aic79xx scsi_transport_spi megaraid_sas cciss sg sd_mod hpsa ehci_hcd scsi_mod uhci_hcd <span class="error">&#91;last unloaded: libcfs&#93;</span><br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.501423&#93;</span> Pid: 15289, comm: ptlrpcd_6 Tainted: G W 2.6.38.2-ts4 #11<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.540816&#93;</span> Call Trace:<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.553799&#93;</span> <span class="error">&#91;&lt;ffffffff8105d10f&gt;&#93;</span> ? warn_slowpath_common+0x7f/0xc0<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.585974&#93;</span> <span class="error">&#91;&lt;ffffffff8105d16a&gt;&#93;</span> ? warn_slowpath_null+0x1a/0x20<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.616810&#93;</span> <span class="error">&#91;&lt;ffffffff81029c24&gt;&#93;</span> ? native_smp_send_reschedule+0x54/0x60<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.651789&#93;</span> <span class="error">&#91;&lt;ffffffff81045f06&gt;&#93;</span> ? resched_task+0x76/0x90<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.679720&#93;</span> <span class="error">&#91;&lt;ffffffff81056465&gt;&#93;</span> ? check_preempt_wakeup+0x1b5/0x280<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.714069&#93;</span> <span class="error">&#91;&lt;ffffffff81045fc4&gt;&#93;</span> ? check_preempt_curr+0x84/0xa0<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.745025&#93;</span> <span class="error">&#91;&lt;ffffffff81055ebb&gt;&#93;</span> ? try_to_wake_up+0x7b/0x410<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.775300&#93;</span> <span class="error">&#91;&lt;ffffffff81056262&gt;&#93;</span> ? default_wake_function+0x12/0x20<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.808168&#93;</span> <span class="error">&#91;&lt;ffffffff8107e046&gt;&#93;</span> ? autoremove_wake_function+0x16/0x40<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.841761&#93;</span> <span class="error">&#91;&lt;ffffffff810455c9&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.873316&#93;</span> <span class="error">&#91;&lt;ffffffff8104d998&gt;&#93;</span> ? __wake_up+0x48/0x70<br/> 2012-06-06 16:08:34 +0000 <span class="error">&#91;4043244.900189&#93;</span> <span class="error">&#91;&lt;ffffffffa0b7834a&gt;&#93;</span> ? cfs_waitq_signal+0x1a/0x20 <span class="error">&#91;libcfs&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043244.935008&#93;</span> <span class="error">&#91;&lt;ffffffffa1012227&gt;&#93;</span> ? ksocknal_queue_tx_locked+0x277/0x540 <span class="error">&#91;ksocklnd&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043244.973972&#93;</span> <span class="error">&#91;&lt;ffffffffa100d033&gt;&#93;</span> ? ksocknal_find_conn_locked+0xa3/0x230 <span class="error">&#91;ksocklnd&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.013276&#93;</span> <span class="error">&#91;&lt;ffffffffa101263b&gt;&#93;</span> ? ksocknal_launch_packet+0x14b/0x350 <span class="error">&#91;ksocklnd&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.053190&#93;</span> <span class="error">&#91;&lt;ffffffffa10129be&gt;&#93;</span> ? ksocknal_send+0x17e/0x410 <span class="error">&#91;ksocklnd&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.087409&#93;</span> <span class="error">&#91;&lt;ffffffffa0cce12b&gt;&#93;</span> ? lnet_ni_send+0x4b/0x100 <span class="error">&#91;lnet&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.119743&#93;</span> <span class="error">&#91;&lt;ffffffffa0cd294b&gt;&#93;</span> ? lnet_send+0x20b/0xa30 <span class="error">&#91;lnet&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.150531&#93;</span> <span class="error">&#91;&lt;ffffffffa0cce530&gt;&#93;</span> ? lnet_prep_send+0x50/0xb0 <span class="error">&#91;lnet&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.183883&#93;</span> <span class="error">&#91;&lt;ffffffffa0cd3a49&gt;&#93;</span> ? LNetPut+0x2a9/0x670 <span class="error">&#91;lnet&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.214255&#93;</span> <span class="error">&#91;&lt;ffffffffa0e9e4da&gt;&#93;</span> ? ptl_send_buf+0x18a/0x440 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.247512&#93;</span> <span class="error">&#91;&lt;ffffffffa0ea08a0&gt;&#93;</span> ? ptl_send_rpc+0x4e0/0xb10 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.280878&#93;</span> <span class="error">&#91;&lt;ffffffff8104bb3a&gt;&#93;</span> ? finish_task_switch+0x4a/0x100<br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.311703&#93;</span> <span class="error">&#91;&lt;ffffffffa0e97695&gt;&#93;</span> ? ptlrpc_send_new_req+0x3e5/0x720 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.348610&#93;</span> <span class="error">&#91;&lt;ffffffff8145d2bf&gt;&#93;</span> ? _raw_spin_lock_irqsave+0x2f/0x40<br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.381490&#93;</span> <span class="error">&#91;&lt;ffffffffa0e9a9a0&gt;&#93;</span> ? ptlrpc_check_set+0x340/0x1750 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.417485&#93;</span> <span class="error">&#91;&lt;ffffffff8106ca3a&gt;&#93;</span> ? del_timer_sync+0x3a/0x60<br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.447321&#93;</span> <span class="error">&#91;&lt;ffffffffa0ec4dcb&gt;&#93;</span> ? ptlrpcd_check+0x52b/0x550 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.480734&#93;</span> <span class="error">&#91;&lt;ffffffffa0ec50ab&gt;&#93;</span> ? ptlrpcd+0x2bb/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.514762&#93;</span> <span class="error">&#91;&lt;ffffffff81056250&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.546436&#93;</span> <span class="error">&#91;&lt;ffffffff8100cde4&gt;&#93;</span> ? kernel_thread_helper+0x4/0x10<br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.578542&#93;</span> <span class="error">&#91;&lt;ffffffffa0ec4df0&gt;&#93;</span> ? ptlrpcd+0x0/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.609487&#93;</span> <span class="error">&#91;&lt;ffffffff8100cde0&gt;&#93;</span> ? kernel_thread_helper+0x0/0x10<br/> 2012-06-06 16:08:35 +0000 <span class="error">&#91;4043245.640837&#93;</span> --<del>[ end trace 70a7f3071bb3c3f8 ]</del>--</p> <p>Jinshan, could you help check why this ASSERT is not true on 2.6.38 kernel?</p> <p>As Jinshan pointed out, it looks to be the same issue of <a href="https://jira.whamcloud.com/browse/LU-1421" title="Client LBUG in ll_file_write after filesystem expansion" class="issue-link" data-issue-key="LU-1421"><del>LU-1421</del></a>, the fix is at <a href="http://review.whamcloud.com/#change,3027" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3027</a>. Roger, could you apply this patch and try again?</p> <p>Customer reports:</p> <p>We have been running with the latest patch for several days with no problems other than the old slowpath warning:</p> <p>609631.154485] -----------<del>[ cut here ]</del>-----------<br/> <span class="error">&#91;609631.154499&#93;</span> WARNING: at fs/libfs.c:363 simple_setattr+0x99/0xb0()<br/> <span class="error">&#91;609631.154502&#93;</span> Hardware name: ProLiant BL460c G7<br/> <span class="error">&#91;609631.154504&#93;</span> Modules linked in: lmv mgc lustre lov osc mdc fid fld ksocklnd ptlrpc obdclass lnet lvfs libcfs parport_pc ppdev 8021q garp bridge stp llc nfsd nfs lockd nfs_acl auth_rpcgss sunrpc netconsole configfs dm_crypt dm_mod crc32c ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi iptable_filter ip_tables x_tables parport be2net psmouse evdev hpilo rtc_cmos joydev hpwdt serio_raw mac_hid loop rtc_core rtc_lib tcp_scalable fuse virtio_blk virtio virtio_ring xenfs ext4 mbcache jbd2 usbhid xfs exportfs raid1 mptspi mptsas mptscsih mptbase mpt2sas raid_class arcmsr aic94xx libsas libata scsi_transport_sas aic7xxx aic79xx scsi_transport_spi megaraid_sas cciss sd_mod sg hpsa scsi_mod ehci_hcd uhci_hcd <span class="error">&#91;last unloaded: libcfs&#93;</span><br/> <span class="error">&#91;609631.154578&#93;</span> Pid: 19765, comm: java Not tainted 2.6.38.2-ts4 #11<br/> <span class="error">&#91;609631.154580&#93;</span> Call Trace:<br/> <span class="error">&#91;609631.154588&#93;</span> <span class="error">&#91;&lt;ffffffff8105d10f&gt;&#93;</span> ? warn_slowpath_common+0x7f/0xc0<br/> <span class="error">&#91;609631.154592&#93;</span> <span class="error">&#91;&lt;ffffffff8105d16a&gt;&#93;</span> ? warn_slowpath_null+0x1a/0x20<br/> <span class="error">&#91;609631.154596&#93;</span> <span class="error">&#91;&lt;ffffffff8115bea9&gt;&#93;</span> ? simple_setattr+0x99/0xb0<br/> <span class="error">&#91;609631.154632&#93;</span> <span class="error">&#91;&lt;ffffffffa10f2cd6&gt;&#93;</span> ? ll_md_setattr+0x3e6/0x840 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;609631.154652&#93;</span> <span class="error">&#91;&lt;ffffffffa10f3394&gt;&#93;</span> ? ll_setattr_raw+0x264/0xe40 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;609631.154672&#93;</span> <span class="error">&#91;&lt;ffffffffa10f3fcd&gt;&#93;</span> ? ll_setattr+0x5d/0x100 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;609631.154677&#93;</span> <span class="error">&#91;&lt;ffffffff81153761&gt;&#93;</span> ? notify_change+0x161/0x2c0<br/> <span class="error">&#91;609631.154682&#93;</span> <span class="error">&#91;&lt;ffffffff811383d1&gt;&#93;</span> ? do_truncate+0x61/0x90<br/> <span class="error">&#91;609631.154687&#93;</span> <span class="error">&#91;&lt;ffffffff811beeec&gt;&#93;</span> ? security_inode_permission+0x1c/0x30<br/> <span class="error">&#91;609631.154692&#93;</span> <span class="error">&#91;&lt;ffffffff81144868&gt;&#93;</span> ? finish_open+0x138/0x1b0<br/> <span class="error">&#91;609631.154696&#93;</span> <span class="error">&#91;&lt;ffffffff81146003&gt;&#93;</span> ? do_last+0x83/0x360<br/> <span class="error">&#91;609631.154699&#93;</span> <span class="error">&#91;&lt;ffffffff81148706&gt;&#93;</span> ? do_filp_open+0x3d6/0x830<br/> <span class="error">&#91;609631.154704&#93;</span> <span class="error">&#91;&lt;ffffffff8110bc27&gt;&#93;</span> ? handle_mm_fault+0x157/0x250<br/> <span class="error">&#91;609631.154708&#93;</span> <span class="error">&#91;&lt;ffffffff8115493a&gt;&#93;</span> ? alloc_fd+0x10a/0x150<br/> <span class="error">&#91;609631.154713&#93;</span> <span class="error">&#91;&lt;ffffffff811373a9&gt;&#93;</span> ? do_sys_open+0x69/0x110<br/> <span class="error">&#91;609631.154717&#93;</span> <span class="error">&#91;&lt;ffffffff81137490&gt;&#93;</span> ? sys_open+0x20/0x30<br/> <span class="error">&#91;609631.154722&#93;</span> <span class="error">&#91;&lt;ffffffff8100bfc2&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b<br/> <span class="error">&#91;609631.154725&#93;</span> --<del>[ end trace e84ad085cd1d9abc ]</del>--</p> <p>Question: Is the best thing to do for this to lower the number of OST threads, as documented on the Lustre manual?</p> <p>This is just kernel warning which is a bit excessive, and it has nothing to do with real IO performance. Lustre uses a kernel exported function simple_setattr(), which is originally for simple filesystem (which doesn't implement truncate), so it gives a warning, but there is no real problem.</p> Development Rank 1|hzvh1z: Rank (Obsolete) 6412 Severity