-
Bug
-
Resolution: Unresolved
-
Blocker
-
Lustre 2.18.0
-
None
-
3
-
9223372036854775807
I am setting this to blocker for now as next timebomb is 2.18.0 and we must not forget about it. Will drop down to critical once that one is handled.
Backstory:
We often phase out of old features/code with a timed ifdefs that checl current version via LUSTRE_VERSION_CODE define and once pas a particular threshold, things get disabled.
The problem here is this new code shape is never tested nor is even easily testable, a tag is created and boom, exactly same code breaks.
This happened numerous time, last time as of the time of this writing was 2.17.58.
I dispatched Claude to catalogue remaining time bombs and when they will hit and what hte falout would be, here are the 8 that would break the build:
h2. Verified build breaks — what happens on each future tag day
| Fires at | Site | Failure |
|---|---|---|
| 2.18.0 | lustre/utils/lfs.c:5600 | #error "Remove LFS_EC_OK check for 2.18 release" — deliberate, guarded > 2.17.90. Build stops in lustre/utils. |
| 2.18.53 | lustre/ptlrpc/wiretest.c:4194 and lustre/utils/wiretest.c:4213 | struct mgs_send_param / MGS_PARAM_MAXLEN disappear from lustre_idl.h, but the two generated wiretest.c files reference them unconditionally: invalid application of 'sizeof' to incomplete type 'struct mgs_send_param', 'MGS_PARAM_MAXLEN' undeclared. Breaks the ptlrpc module and the userspace wiretest. wirecheck.c (the generator) is guarded — its output is not. |
| 3.0.0 | lustre/include/lustre_ioctl_old.h:60,65 via lustre/obdclass/class_obd.c:338 | #define OBD_IOC_CMD_GONE(cmd, name) takes two arguments but the #else branches call OBD_IOC_CMD_GONE(cmd): macro "OBD_IOC_CMD_GONE" requires 2 arguments, but only 1 given. This fallback branch has never been compiled since it was written. |
| 3.0.0 | lustre/llite/llite_lib.c:4258 | if (cmd == OBD_IOC_GETNAME_OLD || ...) is not #ifdef-guarded although the definition (and the case at line 3660) are: 'OBD_IOC_GETNAME_OLD' undeclared. |
| 3.0.53 | lustre/utils/obd.c:827 | struct obd_ioctl_data data; in jt_obd_no_transno() is declared outside the guard, used only inside it: unused variable 'data' [-Werror=unused-variable]. Exactly the again: label failure again. |
| 3.0.53 | lustre/mdt/mdt_internal.h:1560 | static unsigned int max_mod_rpcs_per_client lives in a header; once mdt_max_mod_rpcs_changed() becomes a #define ... false and the module_param_cb in mdt_handler.c:50 goes away, every TU gets defined but not used [-Werror=unused-variable]. |
| 3.1.53 | lustre/obdclass/llog_ioctl.c:23-26 | id, seq, end, ogen, rc in str2logid() are declared outside the guard: five -Werror=unused-variable. |
| > 3.2.53 | lustre/utils/mount_lustre.c:377 | #warn "remove MS_STRICTATIME override..." — #warn is not a directive: error: invalid preprocessing directive #warn; did you mean #warning?. And after correcting it to #warning, it is still fatal: error: #warning ... [-Werror=cpp]. The "just remind me later" idiom does not work at all in this build. |
3. Verified runtime / behaviour breaks
3.1 tgt_lastrcvd.c:2198 (3.5.53) — every server target becomes unmountable
include/uapi/linux/lustre/lustre_disk.h:220 still says:
#define LRH_MAGIC LRH_MAGIC_V1
so tgt_reply_data_init() writes a V1 header into a brand-new reply_data file (tgt_lastrcvd.c:2174, unguarded). But the case LRH_MAGIC_V1: that reads it back is inside #if LUSTRE_VERSION_CODE < OBD_OCD_VERSION(3, 5, 53, 0). At 3.5.53 the read falls to default: and returns -EINVAL.
Reproduced (3.5.53 build, existing ldiskfs MDT):
mount.lustre: mount /dev/vdb at /mnt/mdt0 failed: Invalid argument
LustreError: (tgt_lastrcvd.c:2228:tgt_reply_data_init()) lustre-MDT0000:
invalid reply_data magic: bdabda01 != bdabda01/bdabda02
LustreError: (tgt_mount.c:1887:server_start_targets()) failed to start server lustre-MDT0000: -22
Note the error message prints the magic as equal to LRH_MAGIC_V1 while declaring it invalid. This is not only an upgrade problem: a filesystem formatted and mounted by the 3.5.53 build itself fails on its second mount, because the first mount wrote a V1 header the same build then refuses:
# fresh mkfs + mount by the 3.5.53 build: OK # umount, mount again: mount.lustre_tgt: mount /dev/vdb at /mnt/mdt0 failed: Invalid argument LustreError: ... invalid reply_data magic: bdabda01 != bdabda01/bdabda02
This is the single most dangerous item in the tree. It is a total, fleet-wide server outage on the release that crosses 3.5.53, and nothing in the tree forces LRH_MAGIC to be bumped to LRH_MAGIC_V2 first.
3.2 lustre/utils/mount_lustre.c:1163 (2.18.53) — inverted comparison
#if LUSTRE_VERSION_CODE > OBD_OCD_VERSION(2, 18, 53, 0) /* Pre-2.13 Lustre without 'lustre_tgt' type? * Try with 'lustre' instead. Eventually this * can be removed (e.g. 2.18 or whenever). */
The comment describes backward compatibility that should last until 2.18; the test is >, so the code is compiled out now and switches on at 2.18.53 — precisely backwards. Reproduced by invoking the helper with the module autoloader disabled so the mount() returns ENODEV:
=== 2.18.53 (fallback compiled OUT — today's behaviour) === mount.lustre_tgt: mount -t lustre_tgt /dev/vdb at /mnt/mdt0 failed: No such device retries left: 0 mount.lustre_tgt: mount /dev/vdb at /mnt/mdt0 failed: No such device === 2.19.53 (fallback compiled IN) === mount.lustre_tgt: mount -t lustre_tgt /dev/vdb at /mnt/mdt0 failed: No such device retries left: 0 mount.lustre_tgt: mount -t lustre /dev/vdb at /mnt/mdt0 failed: No such device retries left: 0 mount.lustre_tgt: mount /dev/vdb at /mnt/mdt0 failed: No such device
The "mount an old target with a new mount.lustre" fallback is dead today and will resurrect itself later — the exact opposite of what was intended, and consistent with the LU-12514 "Can't find device name" reports.
3.3 lustre/utils/lfs.c:7920 (2.18.53) — a case label that wakes up and crashes
#if LUSTRE_VERSION_CODE >= OBD_OCD_VERSION(2, 18, 53, 0) case 'p': #endif case LFS_POOL_OPT: if (strlen(optarg) > LOV_MAXPOOLNAME) {
lfs find's option string is "-0A:b:B:...Pp..." — p is declared as taking no argument, so optarg is NULL. Reproduced:
=== 2.17.58 (today) === $ lfs find -p mypool /mnt/lustre/tdir lfs: unrecognized option '-p' (rc=4, prints usage) === 2.18.53 === $ lfs find -p mypool /mnt/lustre/tdir Segmentation fault (core dumped) (rc=139)
A clean error today becomes a NULL dereference in strlen() on the tag that crosses 2.18.53. --pool (the long option, with a required argument) works in both. This is "sleeping code that has never executed" in its purest form.
3.4 lustre/ofd/lproc_ofd.c:532,1302 (2.20.53) — sanity 56c breaks
The compatibility sysfs alias obdfilter.*.no_precreate (for no_create) is removed at 2.20.53. lustre/tests/sanity.sh:7386-7390 uses it gated only on OST1_VERSION >= 2.12.55 — a lower bound with no upper bound, the same mistake as 300l. Reproduced:
=== 2.17.58 === === 2.20.53 === obdfilter.*.no_create obdfilter.*.no_create obdfilter.*.no_precreate (gone) $ ONLY=56c bash sanity.sh # 2.20.53 error: set_param: setting : Invalid argument sanity test_56c: @@@@@@ FAIL: lustre-OST0000 status is 'Df', missing 'N'
conf-sanity.sh:10011 (test_112a) handles the same parameter correctly (>= 2.15.56.125 → no_create, else no_precreate), which shows the fix is known — it just was not applied everywhere.
And even more stuff:
E. Mis-scoped guard swallowing live code.
lustre/osd-ldiskfs/osd_handler.c:8556: the #if < 3.2.53 was clearly meant to cover the Lustre-1.x dirdata warning, but the #endif sits 18 lines further down and also swallows
/* enable large_dir on MDTs to avoid REMOTE_PARENT_DIR overflow,
* and on very large OSTs to avoid object directory overflow */
if (unlikely(!ldiskfs_has_feature_largedir(...)))
ldiskfs_set_feature_largedir(...);
which is not compat code at all. At 3.2.53 the runtime auto-enable of large_dir disappears. mkfs.lustre sets large_dir at format time (libmount_utils_ldiskfs.c:676), so freshly formatted targets are unaffected — but any target formatted by an older mkfs.lustre, or with an e2fsprogs that lacked the feature, silently stops being upgraded, and eventually overflows REMOTE_PARENT_DIR. A guard whose #endif is in the wrong place is the easiest kind of bug to write and the hardest to see in review.
F. Silent removal of a runtime interface with a test that cannot skip.
sanity 300l (already broken) and sanity 56c (breaks at 2.20.53) are the same bug: the test has a lower version bound and no upper bound, so when the feature is deleted on schedule the test converts to a permanent failure. Grep-able signature: a test that uses a parameter/ioctl/option that appears inside any #if LUSTRE_VERSION_CODE < block. Present candidates: obdfilter.*.no_precreate (sanity 56c, 2.20.53), lctl conf_param (test-framework.sh:568 sets PERM_CMD to it by default, 13 more uses in conf-sanity.sh — 3.0.53), lfs find -i (sanity.sh:29014, 3.0.53), lfs setquota -d (sanity-quota.sh:5255,5363, 2.22.53), and lfs find --ost-list (conf-sanity.sh:7648, 3.0.53).
G. On-disk / on-wire format guards that outlive their own default.
§3.1 is the extreme case: the reader of a format is deleted on a timer while the writer still emits it. The same shape exists (benignly, today) in ptlrpc/layout.c:1334 + pack_generic.c:3111 + lustre_idl.h:3812 (mdc_swap_layouts_217, 3.4.53) — those three are consistently guarded, but the pattern deserves a rule, not luck.