Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Lustre 2.14.0, Lustre 2.15.2
-
server (garter):
toss 4.5-6rc6
4.18.0-425.19.2.1toss.t4.x86_64
2.14.0_19.llnl
client (mutt):
toss 4.5-6
4.18.0-425.19.2.1toss.t4.x86_64
2.15.2_5.llnl
-
3
-
9223372036854775807
Description
Attempts to create directories are intermittently failing when using "mkdir".
So far, "lfs mkdir" succeeds every time, so does file creation using "touch".
Attachments
Activity
Hi Serguei or Peter,
Can you ask an appropriate Whamcloud-er to advise? The two test clusters involved will be taken away from us for other testing on Tuesday or Wednesday of next week.
thanks
It looks like this one needs attention of experts in higher-level Lustre.
For example, is LUDOC-289 relevant here?
I've uploaded debug logs for the attempted creation of the directory junk8
debug log which contains failed attempt for junk8 is
dk.mutt12.1683589208
It then succeeded (after failing about 10 times) in
dk.mutt12.1683589684
In this case some directories are created on the first try, but most fail
(mutt12):mdts_3_osts_3$ for x in {1..20}; do mkdir test_name$x; done
mkdir: cannot create directory ‘test_name1’: Object is remote
mkdir: cannot create directory ‘test_name2’: Object is remote
mkdir: cannot create directory ‘test_name3’: Object is remote
mkdir: cannot create directory ‘test_name4’: Object is remote
mkdir: cannot create directory ‘test_name5’: Object is remote
mkdir: cannot create directory ‘test_name6’: Object is remote
mkdir: cannot create directory ‘test_name8’: Object is remote
mkdir: cannot create directory ‘test_name9’: Object is remote
mkdir: cannot create directory ‘test_name10’: Object is remote
mkdir: cannot create directory ‘test_name11’: Object is remote
mkdir: cannot create directory ‘test_name12’: Object is remote
mkdir: cannot create directory ‘test_name13’: Object is remote
mkdir: cannot create directory ‘test_name14’: Object is remote
mkdir: cannot create directory ‘test_name15’: Object is remote
mkdir: cannot create directory ‘test_name16’: Object is remote
mkdir: cannot create directory ‘test_name18’: Object is remote
mkdir: cannot create directory ‘test_name19’: Object is remote
mkdir: cannot create directory ‘test_name20’: Object is remote
(mutt12):mdts_3_osts_3(1)$ for x in {1..20}; do mkdir test_name$x; done
mkdir: cannot create directory ‘test_name1’: Object is remote
mkdir: cannot create directory ‘test_name2’: Object is remote
mkdir: cannot create directory ‘test_name3’: Object is remote
mkdir: cannot create directory ‘test_name4’: Object is remote
mkdir: cannot create directory ‘test_name5’: Object is remote
mkdir: cannot create directory ‘test_name7’: File exists
mkdir: cannot create directory ‘test_name8’: Object is remote
mkdir: cannot create directory ‘test_name9’: Object is remote
mkdir: cannot create directory ‘test_name10’: Object is remote
mkdir: cannot create directory ‘test_name12’: Object is remote
mkdir: cannot create directory ‘test_name13’: Object is remote
mkdir: cannot create directory ‘test_name14’: Object is remote
mkdir: cannot create directory ‘test_name16’: Object is remote
mkdir: cannot create directory ‘test_name17’: File exists
mkdir: cannot create directory ‘test_name18’: Object is remote
mkdir: cannot create directory ‘test_name19’: Object is remote
mkdir: cannot create directory ‘test_name20’: Object is remote
The directories that are created inherit the striping (this was done after more mkdir attempts, so more directories exist than the 2 indicated above)
(mutt12):mdts_3_osts_3$ lfs getdirstripe test_name* lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none
However, using lfs mkdir seems to work fine
(mutt12):mdts_3_osts_3$ for x in {1..20}; do lfs mkdir -i $(($x % 4)) lfs_name$x; done
(mutt12):mdts_3_osts_3$ ls lfs_name*
lfs_name1:
lfs_name10:
lfs_name11:
lfs_name12:
lfs_name13:
lfs_name14:
lfs_name15:
lfs_name16:
lfs_name17:
lfs_name18:
lfs_name19:
lfs_name2:
lfs_name20:
lfs_name3:
lfs_name4:
lfs_name5:
lfs_name6:
lfs_name7:
lfs_name8:
lfs_name9:
Here's an example of a few failures followed by a successful create
(mutt12):mdts_3_osts_3$ pwd /p/lflood/defazio1/mdtest/mdts_3_osts_3 (mutt12):mdts_3_osts_3$ mkdir junk20 mkdir: cannot create directory ‘junk20’: Object is remote (mutt12):mdts_3_osts_3(1)$ mkdir junk20 mkdir: cannot create directory ‘junk20’: Object is remote (mutt12):mdts_3_osts_3(1)$ mkdir junk20 mkdir: cannot create directory ‘junk20’: Object is remote (mutt12):mdts_3_osts_3(1)$ mkdir junk20 (mutt12):mdts_3_osts_3$ ls -ld junk20 drwx------ 2 defazio1 defazio1 26624 May 8 17:53 junk20
The parent directory is /p/lflood/defazio1/mdtest/mdts_3_osts_3
(mutt12):mdts_3_osts_3$ stat . File: . Size: 26624 Blocks: 52 IO Block: 131072 directory Device: 11602e98h/291516056d Inode: 198161923999531372 Links: 16 Access: (0700/drwx------) Uid: (28153/defazio1) Gid: (28153/defazio1) Access: 2023-05-08 16:51:39.000000000 -0700 Modify: 2023-05-08 17:53:11.000000000 -0700 Change: 2023-05-08 17:53:11.000000000 -0700 Birth: 2023-05-08 11:43:54.000000000 -0700
it was created with the commands
lfs setdirstripe --mdt-index 3 /p/lflood/defazio1/mdtest/mdts_3_osts_3 lfs setstripe --ost 3 /p/lflood/defazio1/mdtest/mdts_3_osts_3 chown defazio1:defazio1 /p/lflood/defazio1/mdtest/mdts_3_osts_3
(mutt12):mdts_3_osts_3(4)$ lfs getdirstripe . lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none (mutt12):mdts_3_osts_3$ lfs getstripe -d . stripe_count: 1 stripe_size: 1048576 pattern: raid0 stripe_offset: 3
the llnl lustre versions are at
https://github.com/LLNL/lustre/tree/2.14.0_19.llnl
https://github.com/LLNL/lustre/tree/2.15.2_5.llnl
Could you collect MDS debug logs upon failure? I'm afraid some patch for backward compatibility is missing, maybe it's on server side.