Roundup Tracker - Issues

Issue 1063890

classification
portalocker.lock uses flock, does not work on NFS
Type: Severity: normal
Components: None Versions:
process
Status: closed fixed
:
: : moraes, pohly, richard
Priority: normal :

Created on 2004-11-10 15:32 by moraes, last changed 2005-01-13 05:35 by anonymous.

Files
File name Uploaded Description Edit Remove
DIFF-roundup-lockf moraes, 2004-11-10 15:33 Patch for roundup/backends/portalocker.py
Messages
msg1517 Author: [hidden] (moraes) Date: 2004-11-10 15:32
Trying the demo in an NFS directory (we use NetApps for
all our mass storage) causes an error because flock()
fails. Traceback from portalocker line 124 in lock says
"IOError: [Errno 37] No locks available".  Easy to
reproduce with:

$ python portalocker.py
Traceback (most recent call last):
  File "portalocker.py", line 136, in ?
    portalocker.lock(log, portalocker.LOCK_EX)
  File
"/tmp_mnt/home/moraes/roundup-0.7.8/roundup/backends/portalocker.py",
line 124, in lock
    fcntl.flock(file.fileno(), flags)
IOError: [Errno 37] No locks available
Exit 1

Switching to lockf() works fine (tested on Red
Enterprise 3 Linux, Python 2.3.4) and Solaris 8/Sparc
(also Python 2.3.4).
With the attached patch, users of portalocker can switch
from lockf to flock by setting portalocker.lockmethod =
fcntl.flock.

Regards,
Mark.



msg1518 Author: [hidden] (richard) Date: 2004-11-10 22:11
Logged In: YES 
user_id=6405

It is my understanding that flock is more portable - though yes, 
it doesn't work well with NFS. 
 
I don't know what the solution is here - fixing it for NFS may very 
well break it for other systems. 
msg1519 Author: [hidden] (moraes) Date: 2004-11-11 22:36
Logged In: YES 
user_id=390363

Perhaps a config file option?
NFSLOCKSUPPORT=1

It's a mess, but these days I'd say that fcntl()
locking (which Python slightly misleadingly calls
fcntl.lockf) is more portable than flock() (which
Python even more misleadingly calls fcntl.flock).

 * The only Unixes without fcntl() style locking
 are pre-4.4BSD, and pre-SysV. (fcntl() appeared
 in the SVID, was standardized by /usr/group,
 XPG4, and then Posix).
 http://www.opengroup.org/onlinepubs/009695399/functions/fcntl.html

 * flock() only appeared in 4.2BSD, it exists
 only on the BSD side of the universe.  Many
 systems tended to support both flock() and
 fcntl(), with dire warnings that never the twain
 should mix.  But not all: System V before SVR4
 (and some SVR4) don't support flock(), HPUX,
 RISCOS don't.  I'd bet that some AIX didn't
 either.

 * flock has never worked on NFS.

 * fcntl kind of worked on NFS, but had a bad
 reputation because of bugs and performance
 issues in rpc.lockd and rpc.statd.  In recent years,
 it seems to have improved, probably because the
 databases use this locking. (e.g. sqlite only supports
 fcntl locking)
http://www.washington.edu/imap/documentation/locking.txt.html
 has many of the ugly details.

 * Python implements fcntl.lockf with fcntl()
 locking everywhere.  It implements fcntl.flock
 with flock() if HAVE_FLOCK is defined, but
 otherwise uses fcntl() style locking.

Both flock() and fcntl() have long and ugly
histories, I dislike both and generally prefer
ancient-style hard-link or mkdir-style locking,
even though one has to write manual stale lock
detection/cleanup.
msg1520 Author: [hidden] (pohly) Date: 2004-12-03 10:10
Logged In: YES 
user_id=416927

We just switched our server from RH AS2.1 to EL3.0. Our database
is mounted via NFS. I don't know why flock() worked in the
previous
setup, but now it fails with "No locks available", as Mark
described.

I applied Mark's patch and can confirm that it solves the
problem.
+1 for applying a similar patch, with the default locking
method being the 
same as it ever was, but a config option to switch to the
NFS compatible
one...
msg1521 Author: [hidden] (anonymous) Date: 2005-01-13 05:35
Logged In: NO 

I'd strongly suggest you instead do a test at startup.

try flock, if it fails, do lockf, if that fails, use a
directory.

msg1522 Author: [hidden] (richard) Date: 2005-01-13 05:48
Logged In: YES 
user_id=6405

Hrm. I don't think there's going to be a fix in 0.8 for this (0.8 is way 
overdue as it is)
History
Date User Action Args
2004-11-10 15:32:16moraescreate