[LU-3456] Remove or refactor "ost_connect failed" message

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.7.0
• Affects Version/s: Lustre 2.4.0
• Labels: shh
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Prakash Surya (Inactive)

Description

I see this message at startup time on the MDS. If it's safe to ignore, it should be removed. If it's important, it should be refactored to be understandable by an admin (I don't even know what it means, and it's a console message).

2013-06-11 12:53:52 LustreError: 11-0: lc2-OST0007-osc-MDT0000: Communicating with 10.1.1.48@o2ib9, operation ost_connect failed with -19.

Comments

[Andreas Dilger]

Patch landed to 2.7.0 to improve console message and quiet it down considerably.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/10057/
Subject: LU-3456 ptlrpc: quiet errors on initial connection
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d1cf226d04884a102e17a7d4109764c24572983f

[Zhenyu Xu]

Andreas,

It does not use the same request for initial connection, it changes the import status to DISCONN then CONNECTING and constructs new request (since it could possible select different connection for the import) for connection.

[Prakash Surya (Inactive)]

I think it would make sense to avoid printing this error if there are just a small number of failed initial connection attempts, but still print something if the connection is failing for a long time. It seems reasonable to only print out such messages when there are persistent problems on the connection.

Yes, I think I agree. Since we don't have any better infrastructure for reporting things like this, I'm more OK with the message if we just try and suppressed the "noise".

I still don't think the console is the "right" place for it, but that's all we have at the moment. It would be really cool to be able to, instead, post some sort of event that a another process (e.g. userspace daemon) could consume and then decide what to do (e.g. ignore, ping monitoring software, send email, etc). But that's a whole 'nother can of worms.

I think having some sort of timer (or number of resends) to suppress the message would go a long way in this particular case.

[Andreas Dilger]

Prakash, you are correct that this can happen if the MDS is started before the OSS. The message is printed to the console to alert the sysadmin in case the target OST is not starting up properly, but I agree it is a distraction if it is printed due to some transient condition.

That said, when Brian submitted the patch to update this console message he left in the printing of errors during the initial connection attempt. I think it would make sense to avoid printing this error if there are just a small number of failed initial connection attempts, but still print something if the connection is failing for a long time. It seems reasonable to only print out such messages when there are persistent problems on the connection.

I've pushed an RFC patch http://review.whamcloud.com/10057 but I haven't tested it at all. In particular, I'm not sure if the same request is used repeatedly for the initial connection (which means rq_nr_resends is properly incremented) or if a new request is used each time (which means my attempt at squashing the initial connect messages will fail). Bobijam, could you please take a look at this?

[Prakash Surya (Inactive)]

So then, that sounds like "normal" operation to me. I don't think it warrants a console message. It's probably an artifact of how I sometimes power cycle all server nodes in a test filesystem. If the MDS comes up before the OSS nodes, then this message will appear?