Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-844

LBUG: ASSERTION(iobuf->dr_npages < iobuf->dr_max_pages) when run obdfilter_survey using rsz >= 2M

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • Lustre 2.2.0
    • None
    • None
    • On linux 2.6.18-238.12.1

    Description

      Ran obdfilter_survey(lustre-iokit-1.2-200709210921) for rsz>=2M against local disk using command:

      1. size=32768 case=disk sh obdfilter-survey

      I hit a LBUG:

      Nov 11 13:24:01 oss0 kernel: LustreError: 12359:0:(filter_io_26.c:293:filter_iobuf_add_page()) ASSERTION(iobuf->dr_npages < iobuf->dr_max_pages) failed
      Nov 11 13:24:01 oss0 kernel: LustreError: 12359:0:(filter_io_26.c:293:filter_iobuf_add_page()) LBUG
      Nov 11 13:24:01 oss0 kernel: Pid: 12359, comm: lctl
      Nov 11 13:24:01 oss0 kernel:
      Nov 11 13:24:01 oss0 kernel: Call Trace:
      Nov 11 13:24:01 oss0 kernel: [<ffffffff887786a1>] libcfs_debug_dumpstack+0x51/0x60 [libcfs]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff88778bda>] lbug_with_loc+0x7a/0xd0 [libcfs]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff88780ff0>] tracefile_init+0x0/0x110 [libcfs]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff88b3b650>] ldiskfs_bmap+0x0/0xf0 [ldiskfs]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff88c0291b>] filter_iobuf_add_page+0x3b/0x70 [obdfilter]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff88c05580>] filter_commitrw_write+0x4e0/0x2be0 [obdfilter]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff88bfe8ce>] filter_brw+0x52e/0x6e0 [obdfilter]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff8000f41d>] __alloc_pages+0x78/0x308
      Nov 11 13:24:01 oss0 kernel: [<ffffffff88c2d462>] echo_client_brw_ioctl+0x1032/0x3050 [obdecho]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff8003336f>] __tcp_push_pending_frames+0x766/0x840
      Nov 11 13:24:01 oss0 kernel: [<ffffffff80022185>] tcp_transmit_skb+0x644/0x67c
      Nov 11 13:24:01 oss0 kernel: [<ffffffff80064ae9>] _spin_lock_bh+0x9/0x14
      Nov 11 13:24:01 oss0 kernel: [<ffffffff80030fdc>] release_sock+0x13/0xc1
      Nov 11 13:24:01 oss0 kernel: [<ffffffff800ce73f>] zone_statistics+0x3e/0x6d
      Nov 11 13:24:01 oss0 kernel: [<ffffffff8000abd2>] get_page_from_freelist+0x378/0x43a
      Nov 11 13:24:01 oss0 kernel: [<ffffffff800ce73f>] zone_statistics+0x3e/0x6d
      Nov 11 13:24:01 oss0 kernel: [<ffffffff800ce73f>] zone_statistics+0x3e/0x6d
      Nov 11 13:24:01 oss0 kernel: [<ffffffff88c30583>] echo_client_iocontrol+0x1103/0x2420 [obdecho]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff8001aa2d>] vsnprintf+0x5df/0x627
      Nov 11 13:24:01 oss0 kernel: [<ffffffff80001c20>] _stext+0xc20/0x1000
      Nov 11 13:24:01 oss0 kernel: [<ffffffff800d3af5>] __vmalloc_area_node+0x12e/0x156
      Nov 11 13:24:01 oss0 kernel: [<ffffffff88804f07>] obd_ioctl_getdata+0x5b7/0xec0 [obdclass]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff88819cf0>] class_handle_ioctl+0x1de0/0x2190 [obdclass]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff8000dad0>] permission+0x8d/0xc8
      Nov 11 13:24:01 oss0 kernel: [<ffffffff801aedd8>] misc_open+0x16c/0x260
      Nov 11 13:24:01 oss0 kernel: [<ffffffff888043cb>] obd_class_ioctl+0x19b/0x220 [obdclass]
      Nov 11 13:24:01 oss0 kernel: [<ffffffff800423f3>] do_ioctl+0x21/0x6b
      Nov 11 13:24:01 oss0 kernel: [<ffffffff80030401>] vfs_ioctl+0x457/0x4b9
      Nov 11 13:24:01 oss0 kernel: [<ffffffff800b9630>] audit_syscall_entry+0x1a4/0x1cf
      Nov 11 13:24:01 oss0 kernel: [<ffffffff8004c8fb>] sys_ioctl+0x59/0x78
      Nov 11 13:24:01 oss0 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0

      I tested it using Lustre 1.8.5, 1.8.6-wc1, 1.8.7-wc1 and 2.1. All have the exact same problem. Especially, 1.8.5 and 2.1 will cause kernel panic. This problem happened on both real machine and virtual machine. It does not matter what other parameters I used, once I set rsc 1M to run obdfilter_survey, the problem happened immediately.

      Attachments

        Issue Links

          Activity

            [LU-844] LBUG: ASSERTION(iobuf->dr_npages < iobuf->dr_max_pages) when run obdfilter_survey using rsz >= 2M

            Integrated in lustre-master » i686,client,el5,ofa #404
            LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a)

            Result = SUCCESS
            Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a
            Files :

            • lustre-iokit/obdfilter-survey/obdfilter-survey
            hudson Build Master (Inactive) added a comment - Integrated in lustre-master » i686,client,el5,ofa #404 LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a) Result = SUCCESS Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a Files : lustre-iokit/obdfilter-survey/obdfilter-survey

            Integrated in lustre-master » i686,client,el5,inkernel #404
            LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a)

            Result = SUCCESS
            Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a
            Files :

            • lustre-iokit/obdfilter-survey/obdfilter-survey
            hudson Build Master (Inactive) added a comment - Integrated in lustre-master » i686,client,el5,inkernel #404 LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a) Result = SUCCESS Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a Files : lustre-iokit/obdfilter-survey/obdfilter-survey

            Integrated in lustre-master » i686,server,el5,inkernel #404
            LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a)

            Result = SUCCESS
            Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a
            Files :

            • lustre-iokit/obdfilter-survey/obdfilter-survey
            hudson Build Master (Inactive) added a comment - Integrated in lustre-master » i686,server,el5,inkernel #404 LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a) Result = SUCCESS Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a Files : lustre-iokit/obdfilter-survey/obdfilter-survey

            Integrated in lustre-master » i686,server,el5,ofa #404
            LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a)

            Result = SUCCESS
            Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a
            Files :

            • lustre-iokit/obdfilter-survey/obdfilter-survey
            hudson Build Master (Inactive) added a comment - Integrated in lustre-master » i686,server,el5,ofa #404 LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a) Result = SUCCESS Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a Files : lustre-iokit/obdfilter-survey/obdfilter-survey

            Integrated in lustre-master » i686,client,el6,inkernel #404
            LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a)

            Result = SUCCESS
            Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a
            Files :

            • lustre-iokit/obdfilter-survey/obdfilter-survey
            hudson Build Master (Inactive) added a comment - Integrated in lustre-master » i686,client,el6,inkernel #404 LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a) Result = SUCCESS Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a Files : lustre-iokit/obdfilter-survey/obdfilter-survey

            Integrated in lustre-master » x86_64,server,el6,inkernel #404
            LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a)

            Result = SUCCESS
            Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a
            Files :

            • lustre-iokit/obdfilter-survey/obdfilter-survey
            hudson Build Master (Inactive) added a comment - Integrated in lustre-master » x86_64,server,el6,inkernel #404 LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a) Result = SUCCESS Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a Files : lustre-iokit/obdfilter-survey/obdfilter-survey
            pjones Peter Jones added a comment -

            Landed for 2.2

            pjones Peter Jones added a comment - Landed for 2.2

            Integrated in lustre-master » x86_64,server,el5,inkernel #404
            LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a)

            Result = SUCCESS
            Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a
            Files :

            • lustre-iokit/obdfilter-survey/obdfilter-survey
            hudson Build Master (Inactive) added a comment - Integrated in lustre-master » x86_64,server,el5,inkernel #404 LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a) Result = SUCCESS Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a Files : lustre-iokit/obdfilter-survey/obdfilter-survey

            Integrated in lustre-master » x86_64,client,el5,ofa #404
            LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a)

            Result = SUCCESS
            Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a
            Files :

            • lustre-iokit/obdfilter-survey/obdfilter-survey
            hudson Build Master (Inactive) added a comment - Integrated in lustre-master » x86_64,client,el5,ofa #404 LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a) Result = SUCCESS Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a Files : lustre-iokit/obdfilter-survey/obdfilter-survey

            Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #404
            LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a)

            Result = SUCCESS
            Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a
            Files :

            • lustre-iokit/obdfilter-survey/obdfilter-survey
            hudson Build Master (Inactive) added a comment - Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #404 LU-844 test: limit max IO data size for obdfilter test (Revision 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a) Result = SUCCESS Oleg Drokin : 78b38388cc3aa8f2f9ed367bc812165f24cbdb6a Files : lustre-iokit/obdfilter-survey/obdfilter-survey

            People

              bobijam Zhenyu Xu
              sxu Sean Xu (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: