Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
None
-
3
-
9223372036854775807
Description
Customer sees repeatable MDS crashes with ASSERTION( atomic_read(&d->opd_sync_changes) > 0 . From the vmcore it is seen sync_changes indeed overflowed and turned into a negative number triggering the assertion failure.
crash> osp_device ffff9a6f5ff4e000
struct osp_device {
opd_dt_dev = {
dd_lu_dev = {
ld_ref = {
counter = 371136
},
ld_type = 0xffffffffc14d1620 <osp_device_type>,
ld_ops = 0xffffffffc14c74c0 <osp_lu_ops>,
ld_site = 0xffff9a6fac302138,
...
crash> osp_device.opd_sync_changes ffff9a6f5ff4e000
opd_sync_changes = {
counter = -2147477073
}
crash>
The whole OSP-sync system of adding unlink/setattr llog records to per OST llog files (two-tier , llog catalog + plain llogs) has no mechanism to prevent growing of llog catalogs/files and eventually overflow sync_updates counter . The counter is a signed integer, so exceeding 2bln turns it into a negative number. The llog catalog + llog files also has a limited capacity to store llog records (approximately 64k * 64k is the max).
On a slow system, I can reproduce an unlimited grow of sync_changes by running a simple program changing uid of an open file in a endless loop:
[root@cslmo2302 ~]# while sleep 10; do lctl get_param osp.*.sync_changes ; done
osp.testfs-OST0000-osc-MDT0000.sync_changes=106264
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=157168
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=206598
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=255955
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=305767
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
...
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=1225362
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=1221266
by running a single thread test program with chown(2) system calls and setting
lctl set_param osp.*.max_rpcs_in_progress=4096
Having 1) a faster system , 2) some additional load on OSTs , 3) network problems it is even more real to hit the assertion.
