From 87b26ce6dd00bec645af8702d99f832582fe2b9c Mon Sep 17 00:00:00 2001 From: Garrett D'Amore Date: Tue, 28 Jan 2020 18:55:19 -0800 Subject: Publishing site updates --- RATIONALE.html | 193 ++++++++++++++++++++++++++++++++++++++++++--------- _adoc/RATIONALE.adoc | 133 +++++++++++++++++++++++++---------- 2 files changed, 254 insertions(+), 72 deletions(-) diff --git a/RATIONALE.html b/RATIONALE.html index 3ccbbb88..d6a3595b 100644 --- a/RATIONALE.html +++ b/RATIONALE.html @@ -8,8 +8,8 @@ title: Rationale: Or why am I bothering to rewrite nanomsg?
Garrett D’Amore

-version 0.3, -April 10, 2018 +version 0.4, +January 28, 2020
@@ -73,7 +73,7 @@ and so forth.

Perhaps it might be better to state that there were a number of opportunities to learn from the lessons of nanomsg, as well as lessons we learned while -building nng itself.

+building NNG itself.

State Machine Madness

@@ -85,7 +85,7 @@ machines is incredibly painful.

Worse, these state machines are designed to be run from a single worker -thread. This means that a given socket is entirely single theaded; you +thread. This means that a given socket is entirely single threaded; you could in theory have dozens, hundreds, or even thousands of connections open, but they would be serviced only by a single thread. (Admittedly non-blocking I/O is used to let the OS kernel calls run asynchronously @@ -151,7 +151,7 @@ This means that implementing new transports which might need something other than a file descriptor, is really non-trivial. This stymied my first attempt to add OpenSSL support to get TLS added — OpenSSL has it’s own struct BIO for this stuff, and I could -not see an easy way to convert nanomsg's usock stuff to accomodate the +not see an easy way to convert nanomsg's usock stuff to accommodate the struct BIO.

@@ -206,7 +206,7 @@ blocked in poll().)

But for many cases this is not necessary. A simple callback mechanism would be far better, with the FDs available only as an option for code -that needs them. This is the approach that we have taken with nng.

+that needs them. This is the approach that we have taken with NNG.

As another consequence of our approach, we do not require file descriptors @@ -245,7 +245,7 @@ constraints.

Attempting to retain low numbered "socket descriptors" had its own problems — a huge source of use-after-close bugs, which made the -use of nn_close() incredibly dangerous for multithreaded sockets. +use of nn_close() incredibly dangerous for multi-threaded sockets. (If one thread closes and opens a new socket, other threads still using the old socket might wind up accessing the "new" socket without realizing it.)

@@ -258,12 +258,12 @@ coming to us from C++ (object oriented), Java, and Python
-

With nng, we realized that constraining ourselves to the mistakes of the -POSIX API was hurting rather than helping. So nng provides a much friendlier +

With NNG, we realized that constraining ourselves to the mistakes of the +POSIX API was hurting rather than helping. So NNG provides a much friendlier interface for getting properties associated with messages.

-

In nng we also generally try hard to avoid reusing +

In NNG we also generally try hard to avoid reusing an identifier until no other option exists. This generally means most applications won’t see socket reuse until billions of other sockets have been opened. There is little chance for accidental reuse.

@@ -275,15 +275,15 @@ have been opened. There is little chance for accidental reuse.

Compatibility

-

Of course, there are a number of existing nanomsg consumers "in the wild" +

Of course, there are a number of existing nanomsg consumers “in the wild” already. It is important to continue to support them. So I decided from -the get go to implement a "compatibility" layer, that provides the same +the get go to implement a “compatibility” layer, that provides the same API, and as much as possible the same ABI, as legacy nanomsg. However, new features and capabilities would not necessarily be exposed to the the legacy API.

-

Today nng offers this. You can relink an existing nanomsg binary against +

Today NNG offers this. You can relink an existing nanomsg binary against libnng instead of libnn, and it usually Just Works™. Source compatibility is almost as easy, although the application code needs to be modified to use different header files.

@@ -308,11 +308,11 @@ flag change would be needed.

Asynchronous IO

-

As a consequence of our experience with threads being so unscalable, +

As a consequence of our experience with threads being so un-scalable, we decided to create a new underlying abstraction modeled largely on Windows IO completion ports. (As bad as so many of the Windows APIs are, the IO completion port stuff is actually pretty nice.) Under the -hood in nng all I/O is asynchronous, and we have nni_aio objects +hood in NNG all I/O is asynchronous, and we have nni_aio objects for each pending I/O. These have an associated completion routine.

@@ -321,7 +321,7 @@ for each pending I/O. These have an associated completion routine.

available number of CPU cores to ensure that we never wait while a CPU core is available for work), but they can be run "synchronously" if the I/O provider knows it is safe to do so (for example the completion -is occuring in a context where no locks are held.)

+is occurring in a context where no locks are held.)

The nni_aio structures are accessible to user applications as well, which can @@ -340,7 +340,7 @@ but that work is already in progress.

Portability & Embeddability

-

A significant goal of nng is to be portable to many kinds of different +

A significant goal of NNG is to be portable to many kinds of different kinds of systems, and embedded in systems that do not support POSIX or Win32 APIs. To that end we have a clear platform portability layer. We do require that platforms supply entry points for certain networking, synchronization, @@ -372,7 +372,7 @@ If you want to write a coroutine-based platform, let me know!

New Transports

-

The other, most critical, motivation behind nng was to enable an easier +

The other, most critical, motivation behind NNG was to enable an easier creation of new transports. In particular, one client ( Capitar IT Group BV) contracted the creation of a ZeroTier transport for @@ -385,7 +385,7 @@ I had created for mangos.

In retrospect, I’m not sure that the answer was a clear and definite yes -in favor of nng, but for the other things I want to do, it has enabled a +in favor of NNG, but for the other things I want to do, it has enabled a lot of new work. The ZeroTier transport was created with a relatively modest amount of effort, in spite of being based upon a connectionless transport. I do not believe I could have done this easily in the existing @@ -394,12 +394,12 @@ transport. I do not believe I could have done this easily in the existing

I’ve since added a rich TLS transport, and have implemented a WebSocket transport that is far more capable than that in nanomsg, as it can -support TLS and sharing the TCP port across multiple nng sockets (using +support TLS and sharing the TCP port across multiple NNG sockets (using the path to discriminate) or even other HTTP services.

There are already plans afoot for other kinds of transports using QUIC -or KCP or SSH, as well as a pure UDP transport. The new nng transport +or KCP or SSH, as well as a pure UDP transport. The new NNG transport layer makes implementation of these all fairly straight-forward.

@@ -410,9 +410,9 @@ layer makes implementation of these all fairly straight-forward.

As part of implementing a real WebSocket transport, it was necessary to implement at least some HTTP capabilities. Rather than just settle for a toy -implementation, nng has a very capable HTTP server and client framework. +implementation, NNG has a very capable HTTP server and client framework. The server can be used to build real web services, so it becomes possible -for example to serve static content, REST API, and nng based services +for example to serve static content, REST API, and NNG based services all from the same TCP port using the same program.

@@ -421,13 +421,35 @@ a plethora of other kinds of transports and services.

There is also a portability layer — so some common services (threading, -timing, etc.) are provided in the nng library to help make writing -portable nng applications easier.

+timing, etc.) are provided in the NNG library to help make writing +portable NNG applications easier.

-

It will not surprise me if developers start finding uses for nng that +

It will not surprise me if developers start finding uses for NNG that have nothing to do with Scalability Protocols.

+
+
+
+

Actually, now in 2020, this has come to pass. There exists, for +example, a project that provides a full REST API framework built on top +NNG.

+
+
+
+
+
+
+

Streaming API

+
+
+

Along with the ability to use Scalability Protocols, and a generic +HTTP framework, we’ve provided a nice abstraction for byte-stream +oriented protocols. This makes it possible to build portable applications +that can use WebSockets, TCP, IPC, and TLS all in much the same way. +In 2019, we refactored much of the core Scalability Protocols code to +be based on top of this work.

+
@@ -437,7 +459,7 @@ have nothing to do with Scalability Protocols.

As part of working on a demo suite of applications, I realized that the requirement to use raw mode sockets for concurrent applications was rather onerous, forcing application developers to re-implement much of the -same logic that is already in nng.

+same logic that is already in NNG.

Thus was the born the idea of separating the context for protocols from @@ -457,7 +479,7 @@ always, if it has no need for extra concurrency.

One side benefit of this work was that we identified several places -to make nng perform more efficiently, reducing the number of context +to make NNG perform more efficiently, reducing the number of context switches and extra raw vs. cooked logic.

@@ -466,14 +488,117 @@ switches and extra raw vs. cooked logic.

Towards nanomsg 2.0

-

It is my intention that nng ultimately replace nanomsg. I do think of it -as "nanomsg 2.0". In fact "nng" stands for "nanomsg next generation" in -my mind. Some day soon I’m hoping that the various website -references to nanomsg my simply be updated to point at nng. It is not -clear to me whether at that time I will simply rename the existing -code to nanomsg, nanomsg2, or leave it as nng.

+

It is my intention that NNG ultimately replace nanomsg. I do think of it +as "nanomsg 2.0". In fact “NNG” stands for "nanomsg next generation" in mind. +Some day soon I’m hoping that the various website +references to nanomsg my simply be updated to point at NNG. +It is not clear to me whether at that time I will simply rename the existing +code to nanomsg, nanomsg2, or leave it as NNG.

+
+
+
+
+

My thinking in 2020, is that the nanomsg has significant value, but now +NNG has carved an identity out of it’s own, as has mangos. +The legacy nanomsg project is likely to thus be re-branded as libnanomsg, +and we will re-purpose the nanomsg brand to represent the entire family +of projects around these protocols. In fact NNG has already been +relocated to it’s own site under +nanomsg.org.

+
+
+

Future Directions

+
+
+

In the few years since I started the NNG project, it’s become apparent +that a few enhancements are sorely needed. A few things we have planned +as of January 2020 are:

+
+
+
    +
  • +

    support for pluggable TLS libraries (including FIPS and liberal licensed options)

    +
    • +
  • +

    better support for UDP (including multicast when appropriate)

    +
    • +
  • +

    new wire protocols to negotiate capabilities between peers

    +
    • +
  • +

    publisher side filtered PUB/SUB protocol

    +
    • +
  • +

    a true mesh protocol, including directed unicast and peer discovery

    +
    • +
  • +

    better support for QoS tuning in certain protocols

    +
    • +
  • +

    richer statistics and observability

    +
    • +
  • +

    support for HTTP/2

    +
    • +
  • +

    more platform ports to platforms like FreeRTOS

    +
    • +
+
+
+

Of course, we will also continue to make strides in improving the performance +and scalability of the library. In the past several months alone there have +been significant moves in this direction, and we’ve identified several other +major opportunities to improve performance and reduce latency.

+
+
+

Also, we’ve been working to improve our test coverage, which has uncovered +a raft of bugs (and we expect will continue to help us find and squash them), +as well as identifying some areas to tighten the code leading to both +simpler, and more efficient implementation.

+
+
+

The future of NNG is looking bright, indeed.

+
+
+
+
+

History

+
+ +++++ + + + + + + + + + + + + + + + + + + + + + + +

v0.4

January 28, 2020

Branding, updates for the new decade.

v0.3

April 12, 2018

Minor fixes, added contexts.

v0.2

February 23, 2018

Updates with new information, markup fixes.

v0.1

October 18, 2017

First version.

+
+
+
diff --git a/_adoc/RATIONALE.adoc b/_adoc/RATIONALE.adoc index 5a8dbb77..d02b4812 100644 --- a/_adoc/RATIONALE.adoc +++ b/_adoc/RATIONALE.adoc @@ -1,17 +1,12 @@ ---- -layout: default ---- = Rationale: Or why am I bothering to rewrite nanomsg? -:nofooter: Garrett D'Amore -v0.3, April 10, 2018 +v0.4, January 28, 2020 NOTE: You might want to review http://nanomsg.org/documentation-zeromq.html[Martin Sustrik's rationale] for nanomsg vs. ZeroMQ. - == Background I became involved in the @@ -45,7 +40,7 @@ and so forth. Perhaps it might be better to state that there were a number of opportunities to learn from the lessons of _nanomsg_, as well as lessons we learned while -building _nng_ itself. +building _NNG_ itself. === State Machine Madness @@ -55,7 +50,7 @@ many of which feed into others, such that tracking flow through the state machines is incredibly painful. Worse, these state machines are designed to be run from a single worker -thread. This means that a given socket is entirely single theaded; you +thread. This means that a given socket is entirely single threaded; you could in theory have dozens, hundreds, or even thousands of connections open, but they would be serviced only by a single thread. (Admittedly non-blocking I/O is used to let the OS kernel calls run asynchronously @@ -112,7 +107,7 @@ This means that implementing new transports which might need something other than a file descriptor, is really non-trivial. This stymied my first attempt to add http://www.openssl.org[OpenSSL] support to get TLS added -- _OpenSSL_ has it's own `struct BIO` for this stuff, and I could -not see an easy way to convert _nanomsg_'s `usock` stuff to accomodate the +not see an easy way to convert _nanomsg_'s `usock` stuff to accommodate the `struct BIO`. In retrospect, _OpenSSL_ wasn't the ideal choice for an SSL/TLS library, @@ -150,7 +145,7 @@ blocked in `poll()`.) But for many cases this is not necessary. A simple callback mechanism would be far better, with the FDs available only as an option for code -that needs them. This is the approach that we have taken with _nng_. +that needs them. This is the approach that we have taken with _NNG_. As another consequence of our approach, we do not require file descriptors for sockets at all, so it is possible to create applications containing @@ -182,7 +177,7 @@ Oh, and there was that whole "design by committee" aspect. Attempting to retain low numbered "socket descriptors" had its own problems -- a huge source of use-after-close bugs, which made the -use of `nn_close()` incredibly dangerous for multithreaded sockets. +use of `nn_close()` incredibly dangerous for multi-threaded sockets. (If one thread closes and opens a new socket, other threads still using the old socket might wind up accessing the "new" socket without realizing it.) @@ -193,11 +188,11 @@ are actually in the minority of _nanomsg_'s users. Most of our users are coming to us from {cpp} (object oriented), _Java_, and _Python_ backgrounds. For them the BSD sockets API is frankly somewhat bizarre and alien. -With _nng_, we realized that constraining ourselves to the mistakes of the -POSIX API was hurting rather than helping. So _nng_ provides a much friendlier +With _NNG_, we realized that constraining ourselves to the mistakes of the +POSIX API was hurting rather than helping. So _NNG_ provides a much friendlier interface for getting properties associated with messages. -In _nng_ we also generally try hard to avoid reusing +In _NNG_ we also generally try hard to avoid reusing an identifier until no other option exists. This generally means most applications won't see socket reuse until billions of other sockets have been opened. There is little chance for accidental reuse. @@ -205,14 +200,14 @@ have been opened. There is little chance for accidental reuse. == Compatibility -Of course, there are a number of existing _nanomsg_ consumers "in the wild" +Of course, there are a number of existing _nanomsg_ consumers "`in the wild`" already. It is important to continue to support them. So I decided from -the get go to implement a "compatibility" layer, that provides the same +the get go to implement a "`compatibility`" layer, that provides the same API, and as much as possible the same ABI, as legacy _nanomsg_. However, new features and capabilities would not necessarily be exposed to the the legacy API. -Today _nng_ offers this. You can relink an existing _nanomsg_ binary against +Today _NNG_ offers this. You can relink an existing _nanomsg_ binary against _libnng_ instead of _libnn_, and it usually Just Works(TM). Source compatibility is almost as easy, although the application code needs to be modified to use different header files. @@ -223,11 +218,11 @@ flag change would be needed. == Asynchronous IO -As a consequence of our experience with threads being so unscalable, +As a consequence of our experience with threads being so un-scalable, we decided to create a new underlying abstraction modeled largely on Windows IO completion ports. (As bad as so many of the Windows APIs are, the IO completion port stuff is actually pretty nice.) Under the -hood in _nng_ all I/O is asynchronous, and we have `nni_aio` objects +hood in _NNG_ all I/O is asynchronous, and we have `nni_aio` objects for each pending I/O. These have an associated completion routine. The completion routines are _usually_ run on a separate worker thread @@ -235,7 +230,7 @@ The completion routines are _usually_ run on a separate worker thread available number of CPU cores to ensure that we never wait while a CPU core is available for work), but they can be run "synchronously" if the I/O provider knows it is safe to do so (for example the completion -is occuring in a context where no locks are held.) +is occurring in a context where no locks are held.) The `nni_aio` structures are accessible to user applications as well, which can lead to much more efficient and easier to write asynchronous applications, @@ -248,7 +243,7 @@ but that work is already in progress. == Portability & Embeddability -A significant goal of _nng_ is to be portable to many kinds of different +A significant goal of _NNG_ is to be portable to many kinds of different kinds of systems, and embedded in systems that do not support POSIX or Win32 APIs. To that end we have a clear platform portability layer. We do require that platforms supply entry points for certain networking, synchronization, @@ -265,7 +260,7 @@ TIP: If you want to write a coroutine-based platform, let me know! == New Transports -The other, most critical, motivation behind _nng_ was to enable an easier +The other, most critical, motivation behind _NNG_ was to enable an easier creation of new transports. In particular, one client ( http://www.capitar.com[Capitar IT Group BV]) contracted the creation of a http://www.zerotier.com[ZeroTier] transport for @@ -276,7 +271,7 @@ myself if it would not be easier just to rewrite _nanomsg_ using the model I had created for _mangos_. In retrospect, I'm not sure that the answer was a clear and definite yes -in favor of _nng_, but for the other things I want to do, it has enabled a +in favor of _NNG_, but for the other things I want to do, it has enabled a lot of new work. The ZeroTier transport was created with a relatively modest amount of effort, in spite of being based upon a connectionless transport. I do not believe I could have done this easily in the existing @@ -284,38 +279,53 @@ _nanomsg_. I've since added a rich TLS transport, and have implemented a WebSocket transport that is far more capable than that in _nanomsg_, as it can -support TLS and sharing the TCP port across multiple _nng_ sockets (using +support TLS and sharing the TCP port across multiple _NNG_ sockets (using the path to discriminate) or even other HTTP services. There are already plans afoot for other kinds of transports using QUIC -or KCP or SSH, as well as a pure UDP transport. The new _nng_ transport +or KCP or SSH, as well as a pure UDP transport. The new _NNG_ transport layer makes implementation of these all fairly straight-forward. == HTTP and Other services As part of implementing a real WebSocket transport, it was necessary to implement at least some HTTP capabilities. Rather than just settle for a toy -implementation, _nng_ has a very capable HTTP server and client framework. +implementation, _NNG_ has a very capable HTTP server and client framework. The server can be used to build real web services, so it becomes possible -for example to serve static content, REST API, and _nng_ based services +for example to serve static content, REST API, and _NNG_ based services all from the same TCP port using the same program. We've also made the WebSocket services fairly generic, which may support a plethora of other kinds of transports and services. There is also a portability layer -- so some common services (threading, -timing, etc.) are provided in the _nng_ library to help make writing -portable _nng_ applications easier. +timing, etc.) are provided in the _NNG_ library to help make writing +portable _NNG_ applications easier. -It will not surprise me if developers start finding uses for _nng_ that +It will not surprise me if developers start finding uses for _NNG_ that have nothing to do with Scalability Protocols. +**** +Actually, now in 2020, this has come to pass. There exists, for +example, a project that provides a full REST API framework built on top +NNG. +**** + +== Streaming API + +Along with the ability to use Scalability Protocols, and a generic +HTTP framework, we've provided a nice abstraction for byte-stream +oriented protocols. This makes it possible to build portable applications +that can use WebSockets, TCP, IPC, and TLS all in much the same way. +In 2019, we refactored much of the core Scalability Protocols code to +be based on top of this work. + == Separate Contexts As part of working on a demo suite of applications, I realized that the requirement to use raw mode sockets for concurrent applications was rather onerous, forcing application developers to re-implement much of the -same logic that is already in _nng_. +same logic that is already in _NNG_. Thus was the born the idea of separating the context for protocols from the socket, allowing multiple contexts (each of which managing it's own @@ -331,14 +341,61 @@ still use the implicit context associated with the socket just like always, if it has no need for extra concurrency. One side benefit of this work was that we identified several places -to make _nng_ perform more efficiently, reducing the number of context +to make _NNG_ perform more efficiently, reducing the number of context switches and extra raw vs. cooked logic. == Towards _nanomsg_ 2.0 -It is my intention that _nng_ ultimately replace _nanomsg_. I do think of it -as "nanomsg 2.0". In fact "nng" stands for "nanomsg next generation" in -my mind. Some day soon I'm hoping that the various website -references to nanomsg my simply be updated to point at _nng_. It is not -clear to me whether at that time I will simply rename the existing -code to _nanomsg_, nanomsg2, or leave it as _nng_. +It is my intention that _NNG_ ultimately replace _nanomsg_. I do think of it +as "nanomsg 2.0". In fact "`NNG`" stands for "nanomsg next generation" in mind. +Some day soon I'm hoping that the various website +references to nanomsg my simply be updated to point at _NNG_. +It is not clear to me whether at that time I will simply rename the existing +code to _nanomsg_, nanomsg2, or leave it as _NNG_. + +**** +My thinking in 2020, is that the _nanomsg_ has significant value, but now +_NNG_ has carved an identity out of it's own, as has mangos. +The legacy _nanomsg_ project is likely to thus be re-branded as _libnanomsg_, +and we will re-purpose the _nanomsg_ brand to represent the entire family +of projects around these protocols. In fact _NNG_ has already been +relocated to it's own xref:https://nng.nanomsg.org[site] under +xref:https://nanomsg.org[nanomsg.org]. +**** + +== Future Directions + +In the few years since I started the _NNG_ project, it's become apparent +that a few enhancements are sorely needed. A few things we have planned +as of January 2020 are: + +* support for pluggable TLS libraries (including FIPS and liberal licensed options) +* better support for UDP (including multicast when appropriate) +* new wire protocols to negotiate capabilities between peers +* publisher side filtered PUB/SUB protocol +* a true mesh protocol, including directed unicast and peer discovery +* better support for QoS tuning in certain protocols +* richer statistics and observability +* support for HTTP/2 +* more platform ports to platforms like FreeRTOS + +Of course, we will also continue to make strides in improving the performance +and scalability of the library. In the past several months alone there have +been significant moves in this direction, and we've identified several other +major opportunities to improve performance and reduce latency. + +Also, we've been working to improve our test coverage, which has uncovered +a raft of bugs (and we expect will continue to help us find and squash them), +as well as identifying some areas to tighten the code leading to both +simpler, and more efficient implementation. + +The future of NNG is looking bright, indeed. + +== History + +|=== +| v0.4 | January 28, 2020 | Branding, updates for the new decade. +| v0.3 | April 12, 2018 | Minor fixes, added contexts. +| v0.2 | February 23, 2018 | Updates with new information, markup fixes. +| v0.1 | October 18, 2017 | First version. +|=== \ No newline at end of file -- cgit v1.2.3-70-g09d2