[LU-4560] Network error on samba share with lustre backend

• Status: Resolved
• Project: Lustre
• Type: Task
• Priority: Critical
• Reporter: Supporto Lustre Jnet2000 (Inactive)
• Assignee: Emoly Liu
• Resolution: Cannot Reproduce

Environment

Operating system
Red Hat Enterprise Linux Server release 5.7 (Tikanga)

Samba
samba-client-3.0.33-3.39.el5_8
samba-3.0.33-3.39.el5_8
samba-common-3.0.33-3.39.el5_8

Lustre
lustre-modules-1.8.7-wc1_2.6.18_274.3.1.el5xen
lustre-1.8.7-wc1_2.6.18_274.3.1.el5xen

Description

Our users with windows 7 64bit and Office 32bit, often receive the message "network is busy" saving .doc files over samba share (lustre backend). Sometimes it's sufficient to press try again, sometimes the file must be saved locally. Users with windows xp 32 bit and office 2007 32bit do not experience this problem. When the problem occur, samba seem to loose track of the opened files We attach files with samba log at level 4 and 5 and 10.

Regards

Comments

[Peter Jones]

Hi Emoly

Could you please look into this one?

Thanks

Peter

[Malcolm Cowe (Inactive)]

While we are not ruling out a potential issue with Lustre, there are a number of potential causes for this behaviour that relate to Samba or to Windows (sometimes both). I realise that you probably have far more experience with Samba and Windows networking than I do, but it would be helpful if we could narrow down the root cause by eliminating possible non-Lustre issues first.

There is a widely reported issue with Windows 7 and Office 2007/2010 that matches the symptoms described in the ticket. It is believed that disabling SMB2 protocol on both client and server may fix this. For example:

http://community.spiceworks.com/topic/203084-the-network-is-busy-in-excel

with proposed solution to disable SMB2:

http://social.technet.microsoft.com/Forums/office/en-US/46c082ad-2455-4120-beea-c69ff2219ed9/word-2010-slow-opening-files-on-network-share?forum=officeitproprevious

I also found that there is a bug with older builds of Samba on RHEL, as described here:

http://lists.samba.org/archive/samba/2009-March/146830.html

(Read through the whole thread – very interesting).

Other obvious things to look at are name service resolution (/etc/hosts – e.g. make sure the first line is the localhost entry, DNS, NetBIOS, Winbind, etc.) and date/time synchronisation. Slow networks and server loading can also lead to slowness symptoms. Are the files in question shared with many users, or heavily accessed by lots of individuals or clients?

Opportunistic locking has been known to create problems with share exports. I don't have a lot of experience with oplocks, but the Samba documentation is comprehensive:

http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/locking.html

I mention this only because the logs have some oplocks "permission denied" errors.

If you have security = server, there can be problems with the responsiveness of the Samba server when there are a lot of active users. If the connection is interrupted for any reason, then the client has to completely disconnect and reconnect to the server in order to reestablish the share. Security = server also adds significant overhead to the PDC because the connection must remain active for the duration of a user's session.

[Emoly Liu]

Malcolm, thanks for your information. I just get back to work after a vacation and start to look into this ticket.

Does this samba error only happen on windows+office specially? Is that possible to reproduce it on some linux system?

Thanks.

[Malcolm Cowe (Inactive)]

Emoly,

There may be many causes, but the Windows+Office combination is commonly reported. I am not sure if there is a linux-only reproducer but I think that a windows client is required.

[Emoly Liu]

Malcolm, I built my samba testing network, including one centos samba server and one windows7 client. Do you know how to reproduce this error in detail? I tried to save some office documents on lustre filesystem and modify them, but no errors happen.

[Malcolm Cowe (Inactive)]

Emoly, I don't have a reproducer for this error. When I was searching for a cause, there are many reports of similar symptoms but from a range of configurations. It might be related to locking or the oplocks feature, where multiple users try to access the same files at the same time. Or it could be one of the other causes I put in the comment. I suspect that the problem is most likely to occur at scale, when there is more than one client connected.

[Emoly Liu]

Hmm, as you said, this problem is most likely to occur at scale, so it's very hard for me to reproduce it in my simple local samba testing environment. BTW, I am not so familiar with windows and samba, so could anyone provide some lustre logs so that I can analyze if this issue is related to lustre?

[Gabriele Paciucci (Inactive)]

Upgrading to lustre 1.8.9 and samba 3.6, the problem seems to be fixed. Please close this issue.

[Peter Jones]

ok - thanks Gabriele!