core: set O_NONBLOCK on every side. Closes #33

The last time I tested set_nonblock() as a fix for the rsync hang, I
used F_SETFD rather than F_SETFL, which resulted in no error, but also
did not set O_NONBLOCK. Turns out missing O_NONBLOCK was the problem.

The rsync hang was due to every context blocking in os.write() waiting
for either a parent or child buffer to empty, which was exacerbated by
rsync's own pipelining, that allows writes from both sides to proceed
even while reads aren't progressing. The hang was due to os.write() on a
blocking fd blocking until buffer space is available to complete the
write. Partial writes are only supported when O_NONBLOCK is enabled.

mitogen/core.py

@@ -29,7 +29,7 @@ FORWARD_LOG = 102
 ADD_ROUTE = 103
 ALLOCATE_ID = 104
 
-CHUNK_SIZE = 4096  # TODO: this was 16384, but that triggers an unfixed hang.
+CHUNK_SIZE = 16384
 
 
 if __name__ == 'mitogen.core':
@@ -125,6 +125,11 @@ def set_cloexec(fd):
     fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC)
 
 
+def set_nonblock(fd):
+    flags = fcntl.fcntl(fd, fcntl.F_GETFL)
+    fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
+
+
 def io_op(func, *args):
     try:
         return func(*args), False
@@ -461,6 +466,8 @@ class Side(object):
         #: active reader set to defer shutdown until the side is disconnected.
         self.keep_alive = keep_alive
 
+        set_nonblock(fd)
+
     def __repr__(self):
         return '<Side of %r fd %s>' % (self.stream, self.fd)