fixes #114 docs: Major differences and motivation vs. nanomsg

1 files changed, 260 insertions, 0 deletions

diff --git a/RATIONALE.adoc b/RATIONALE.adoc
new file mode 100644
index 00000000..9c21ce3f
--- /dev/null
+++ b/RATIONALE.adoc
@@ -0,0 +1,260 @@
+Rational: Or why am I bothering to rewrite nanomsg?
+===================================================
+Garrett D'Amore <garrett@damore.org>
+v0.1, October 18, 2017
+
+
+NOTE: You might want to review
+      http://nanomsg.org/documentation-zeromq.html[Martin Sustrik's rationale]
+      for nanomsg vs. ZeroMQ.
+
+
+Background
+----------
+
+I became involved in the nanomsg community back in 2014, when
+I wrote https://github.com/go-mangos/mangos[mangos] as a pure Golang
+implementation of the wire protocols behind nanomsg.  I did that work
+because I was dissatisfied with ZeroMQ's licensing and C++ baggage, and
+I needed something that would work with Golang on Solaris, which at the time
+lacked support for cgo (so I could not just use an FFI binding.)  This gave
+me a lot of detail about the internals of nanomsg and the SP protocols.
+At the time, it was the only alternate implementation those protocols.
+
+It would not be wrong to say that one of the goals of mangos was to teach
+me about golang.  It was my first non-trivial golang project.
+
+While working with mangos, I wound up implementing a number of additional
+features, such as a TLS transport, the ability to bind to wild card ports,
+and the ability to determine more information about the sender of a message.
+This was incredibly useful in a number of projects.
+
+I initially looked at nanomsg itself, as I wanted to add a TLS transport
+to it, and I needed to make some bug fixes (for protocol bugs for example),
+and so forth.
+
+
+State Machine Madness
+---------------------
+
+What I ran into was a challenging mess of state machines.
+nanomsg has dozens of state machines, many of which feed into others,
+such that tracking flow through the state machines is incredibly painful.
+
+Worse, these state machines are designed to be run from a single worker
+thread.  This means that a given socket is entirely single theaded; you
+could in theory have dozens, hundreds, or even thousands of connections
+open, but they would be serviced only by a single thread.  (Admittedly
+non-blocking I/O is used to let the OS kernel calls run asynchronously
+perhaps on multiple cores, but nanomsg itself runs all socket code on
+a single worker thread.)
+
+There is another problem too -- the inproc code that moves messages
+between one socket and another was incredibly racy.  This is because the
+two sockets have different locks, and so dealing with the different
+contexts was tricky (and consequently buggy).  (I've since, I think, fixed
+the worst of the bugs here, but not after many hours of pulling out hair.)
+
+The state machines also make fairly linear flow really difficult to follow.
+For example, there is a state machine to read the header information.  This
+may come a byte a time, and the state machine has to add the bytes, check
+for completion, and possibly change state, even if it is just reading a
+single 32-bit word.  This is a lot more complex than most programmers are
+used to, such as `read(fd, &val, 4)`.
+
+Now to be fair, Martin Sustrik had the best intentions when he created the
+state machine model around which nanomsg is built.  I do think that from
+experience this is one of the most dense and unapproachable parts of nanomsg,
+in spite of the fact that Martin's goal was precisely the opposite.  I
+consider this a "failed experiment" -- but hey failed experiments are the
+basis of all great science.
+
+
+Thread Challenges
+-----------------
+
+While nanomsg is mostly internally single threaded, I decided to try to
+emulate the simple architecture of mangos using system threads.  (mangos
+has the benefit of goroutines.)  Having been well and truly spoiled by
+Solaris/illumos threading (and especially Solaris/illumos kernel threads),
+I thought this would be a reasonable architecture.
+
+Sadly, this initial effort, while it worked, scaled incredibly poorly --
+even so-called "modern" operating systems like macOS 10.12 and Windows 8.1
+simply melted or failed entirely when creating any non-trivial number of
+threads.  (To me, creating 100 threads should be a no-brainer, especially if
+one limits the stack size appropriately.  I'm used to be able to create
+thousands of threads without concern.  As I said, I've been spoiled.
+If your system falls over at a mere 200 threads I consider it a toy
+implementation of threading. Unfortunately most of the mainstream operating
+systems are therefore toy implementations.)
+
+Chalk up another failed experiment.
+
+I did find another approach which is discussed further.
+
+
+File Descriptor Driven
+----------------------
+
+Most of the underlying I/O in nanomsg is built around file descriptors,
+and it's internal usock structure, which is also state machine driven.
+This means that implementing new transports which might need something
+other than a file descriptor, is really non-trivial.  This stymied my
+first attempt to add OpenSSL support to get TLS added -- OpenSSL has
+it's own struct BIO for this stuff, and I could not see an easy way to
+convert nanomsg's usock stuff to accomodate the struct BIO.
+
+
+Poll
+----
+
+In order to support use in event driven programming, asynchronous
+situations, etc. nanomsg offers non-blocking I/O.  In order to make
+this work for end-users, a notification mechanism is required, and
+nanomsg, in the spirit of following POSIX, offers a notification method
+based on poll(2) or select(2).
+
+In order for this to work, it offers up a selectable file descriptor
+for send and another one for receive.  When events occur, these are
+written to, and the user application "clears" these by reading from
+them.  (This is done on behalf of the application by nanomsg's API calls.)
+
+This means that in addition to the context switch code, there are not
+fewer than 2 extra system calls executed per message sent or received, and
+on a mostly idle system as many as 3.  This means that to send a message
+from one process to another you may have to execute up to 6 extra system
+calls, beyond the 2 required to actually send and receive the message.
+
+There are cases where this file descriptor logic is easier for existing
+applications to integrate into event loops (e.g. they already have a thread
+blocked in poll().)
+
+But for many cases this is not necessary.  A simple callback mechanism
+would be far better, with the FDs available only as an option for code
+that needs them.  This is the approach that we have taken with nng.
+
+As another consequence of our approach, we do not require file descriptors
+for sockets at all, so it is possible to create applications containing
+many thousands of inproc sockets with no files open at all.  (Obviously
+if you're going to perform real I/O to other processes or other systems,
+you're going to need to have the underlying transport file descriptors
+open.)
+
+
+POSIX APIs
+----------
+
+Another of Martin's goals, which seems worthwhile at first, was the
+attempt to provide a familiar BSD POSIX API.  As a C programmer, this
+really attracted me.
+
+The problem is that the POSIX APIs are actually really horrible.  In
+particular the semantics around cmsg are about as arcane and painful as
+one can imagine.  Largely, this has meant that extensions to the cmsg
+API simply have not occurred in nanomsg.
+
+The cmsg API specified by POSIX is as bad as it is because POSIX had
+requirements not to break APIs that already existed, and they needed to
+shim something that would work with existing implementations, including
+getting across a system call boundary.  nanomsg  has never had such
+constraints.
+
+Attempting to retain low numbered "socket descriptors" had its own
+problems -- a huge source of use-after-close bugs, which made the
+use of nn_close() incredibly dangerous for multithreaded sockets.
+(If one thread closes and opens a new socket, other threads still using
+the old socket might wind up accessing the "new" socket without realizing
+it.)
+
+The other thing is that BSD socket APIs are super familiar to UNIX C
+programmers -- but experience with nanomsg has taught us already that these
+are actually in the minority of nanomsg's users.  Most of our users are
+coming to us from C++ (object oriented), Java, and Python backgrounds.
+For them the BSD sockets API is frankly somewhat bizarre and alien.
+
+With nng, we realized that constraining ourselves to the mistakes of the
+POSIX API was hurting rather than helping. So nng provides a much friendlier
+interface for getting properties associated with messages.
+
+In nng we also generally try hard to avoid reusing
+an identifier until no other option exists.  This generally means most
+applications won't see socket reuse until billions of other sockets
+have been opened.
+
+
+Compatibility
+-------------
+
+Of course, there are a number of existing nanomsg consumers "in the wild"
+already.  It is important to continue to support them.  So I decided from
+the get go to implement a "compatibility" layer, that provides the same
+API, and as much as possible the same ABI, as legacy nanomsg.  However,
+new features and capabilities would not necessarily be exposed to the
+the legacy API.
+
+Today nng offers this.  You can relink an existing nanomsg binary against
+nng instead of nanomsg, and it usually Just Works.  Source compatibility
+is almost as easy, although the application code needs to be modified
+to use different header files.  (Note: I am considering changing this
+in the future.)
+
+
+Asynchronous IO
+---------------
+
+As a consequence of our experience with threads being so unscalable,
+we decided to create a new underlying abstraction modeled largely on
+Windows IO completion ports.  (As bad as so many of the Windows APIs
+are, the IO completion port stuff is actually pretty nice.)  Under the
+hood in nng all I/O is asynchronous, and we have `struct nni_aio` objects
+for each pending I/O.  These have an associated completion routine.
+
+The completion routines are _usually_ run on a separate worker thread
+(we have many such workers; in theory the number should be tuned to the
+available number of CPU cores to ensure that we never wait while a CPU
+core is available for work), but they can be run "synchronously" if
+the I/O provider knows it is safe to do so (for example the completion
+is occuring in a context where no locks are held.)
+
+There is still a lot of performance tuning work to do, and there is a
+real need to expose the AIO structures to user code, which is something
+that I will be doing in the coming days.  This should enable a much 
+faster and lighter weight I/O model, especially for event driven programs.
+
+
+New Transports (ZeroTier)
+-------------------------
+
+The other, most critical, motivation behind nng was to enable an easier
+creation of new transports.  In particular, one client (Capitar IT Group BV)
+contracted the creation of a ZeroTier transport for nanomsg.
+
+After beating my head against the state machines some more, I finally asked
+myself if it would not be easier just to rewrite nanomsg using the model
+I had created for mangos.
+
+In retrospect, I'm not sure that the answer was a clear and definite yes
+in favor of nng, but for the other things I want to do, it has enabled a
+lot of new work.  The ZeroTier transport was created with a relatively
+modest amount of effort, in spite of being based upon a connectionless
+transport.  I do not believe I could have done this easily in the existing
+nanomsg.
+
+In the coming weeks I expect to create transports for TLS, websocket,
+and websocket over TLS.  I expect the websocket transport to support
+multiple sockets with different path components bound to the same TCP port.
+This might be possible with nanomsg, but it wouldn't be easy.  With nng,
+it looks fairly straight-forward.
+
+
+Towards nanomsg 2.0
+-------------------
+
+It is my intention that nng ultimately replace nanomsg.  I do think of it
+as "nanomsg 2.0".  In fact "nng" stands for "nanomsg next generation" in
+my mind.  Some day before too long I'm hoping that the various website
+references to nanomsg my simply be updated to point at nng.  It is not
+clear to me whether at that time I will simply rename the existing
+code to nanomsg, nanomsg2, or leave it as nng.  (One possible issue is
+that Martin Sustrik retains the trademark for "nanomsg".)