[LU-15114] ASSERTION( atomic_read(&d->opd_sync_changes) > 0

Related — Sun, 20 Feb 2022 19:18:35 +0000

Customer sees repeatable MDS crashes with ASSERTION( atomic_read(&d->opd_sync_changes) > 0 . From the vmcore it is seen sync_changes indeed overflowed and turned into a negative number triggering the assertion failure.

crash> osp_device ffff9a6f5ff4e000
struct osp_device {
  opd_dt_dev = {
    dd_lu_dev = {
      ld_ref = {
        counter = 371136
      }, 
      ld_type = 0xffffffffc14d1620 <osp_device_type>, 
      ld_ops = 0xffffffffc14c74c0 <osp_lu_ops>, 
      ld_site = 0xffff9a6fac302138, 
...
crash> osp_device.opd_sync_changes ffff9a6f5ff4e000
  opd_sync_changes = {
    counter = -2147477073
  }
crash>

The whole OSP-sync system of adding unlink/setattr llog records to per OST llog files (two-tier , llog catalog + plain llogs) has no mechanism to prevent growing of llog catalogs/files and eventually overflow sync_updates counter . The counter is a signed integer, so exceeding 2bln turns it into a negative number. The llog catalog + llog files also has a limited capacity to store llog records (approximately 64k * 64k is the max).

On a slow system, I can reproduce an unlimited grow of sync_changes by running a simple program changing uid of an open file in a endless loop:

[root@cslmo2302 ~]# while sleep 10; do lctl get_param osp.*.sync_changes ; done
osp.testfs-OST0000-osc-MDT0000.sync_changes=106264
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=157168
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=206598
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=255955
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=305767
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
...
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=1225362
osp.testfs-OST0001-osc-MDT0000.sync_changes=0
osp.testfs-OST0000-osc-MDT0000.sync_changes=1221266
by running a single thread test program with chown(2) system calls and setting

lctl set_param osp.*.max_rpcs_in_progress=4096
Having 1) a faster system , 2) some additional load on OSTs , 3) network problems it is even more real to hit the assertion.

Whamcloud Community JIRA

[LU-15114] ASSERTION( atomic_read(&d->opd_sync_changes) > 0