+ +

Asynchronous Operations

In order to obtain significant scalability, with low-latency, and minimal +overheads, NNG supports performing operations asynchronously.

One way that applications can perform work asynchronously and concurrently +is by using threads, but threads carry significant resource overheads +and frequently there are limits on the number that can easily be created.

Additionally, with most network applications, the flow of execution will spend +the bulk of its time waiting for traffic from a peer.

For these kinds of applications, it is far more efficient to use asynchronous operations +using the mechanisms described in this chapter.

+ + tip +

To get the highest performance with the least overhead, applications should use +asynchronous operations described in this chapter whenever possible.

Asynchronous I/O Handle

typedef struct nng_aio nng_aio;
+

An nng_aio is an opaque structure used in conjunction with +asynchronous I/O operations. +Every asynchronous operation uses one of these structures, each of which +can only be used with a single operation at a time.

Asynchronous operations are performed without blocking calling application threads. +Instead the application registers a callback function to be executed +when the operation is complete (whether successfully or not). +This callback will be executed exactly once.

The asynchronous I/O framework also supports cancellation of +operations that are already in progress as well setting a maximum +timeout for them to complete.

It is also possible to initiate an asynchronous operation, and wait for it to +complete, creating a synchronous flow from an asynchronous one.

Create Handle

int nng_aio_alloc(nng_aio **aiop, void (*callb)(void *), void *arg);
+

The nng_aio_alloc function creates an nng_aio object, with the +callback callb taking the argument arg, and returns it in aiop.

If this succeeds, the function returns zero. Otherwise it may return NNG_ENOMEM.

+ + tip +

The arg should normally be a structure that contains a pointer to the aiop, +or from which it can be located. This allows callb to check the handle for +success using nng_aio_result, as well access other properties of aiop.

+ + tip +

The callb may be executed on another thread, so it may be necessary to use +synchronization methods in callb to avoid data races.

Destroy Handle

void nng_aio_free(nng_aio *aio);
+void nng_aio_reap(nng_aio *aio);
+

The nng_aio_free handle destroys the handle aio, waiting for any operations +and associated callbacks to complete before doing so.

The nng_aio_reap handle destroys the handle aio asynchronously, using a reaper +thread to do so. It does not wait for the object to be destroyed. Thus this function +is safe to call from aio’s own callback.

+ + note +

The nng_aio_free function must never be called from an aio callback. +Use nng_aio_reap instead if an object must be destroyed from a callback.

Cancellation

void nng_aio_abort(nng_aio *aio, int err);
+void nng_aio_cancel(nng_aio *aio);
+void nng_aio_stop(nng_aio *aio);
+

These functions are used to stop a previously submitted asynchronous +I/O operation. The operation may be canceled, or may continue to +completion. If no operation is in progress (perhaps because it has +already completed), then these operations have no effect. +If the operation is successfully canceled or aborted, then the callback +will still be called.

The nng_aio_abort function aborts the operation associated with aio +and returns immediately without waiting. If cancellation was successful, +then nng_aio_result will return err.

The nng_aio_cancel function acts like nng_aio_abort, but uses the error code +NNG_ECANCELED.

The nng_aio_stop function aborts the aio operation with NNG_ECANCELED, +and then waits the operation and any associated callback to complete. +This function also marks aio itself permanently stopped, so that any +new operations scheduled by I/O providers using nng_aio_begin +return false. Thus this function should be used to teardown operations.

+ + tip +

When multiple asynchronous I/O handles are in use and need to be +deallocated, it is safest to stop all of them using nng_aio_stop, +before deallocating any of them with nng_aio_free, +particularly if the callbacks might attempt to reschedule further operations.

Set Timeout

void nng_aio_set_timeout(nng_aio *aio, nng_duration timeout);
+void nng_aio_set_expire(nng_aio *aio, nng_time expiration);
+

The nng_aio_set_timeout function sets a timeout +for the asynchronous operation associated with aio. +This causes a timer to be started when the operation is actually started. +If the timer expires before the operation is completed, then it is +aborted with an error of NNG_ETIMEDOUT. +The timeout duration is specified as a relative number of milliseconds.

If the timeout is NNG_DURATION_INFINITE, then no timeout is used. +If the timeout is NNG_DURATION_DEFAULT, then a “default” or socket-specific +timeout is used. +(This is frequently the same as NNG_DURATION_INFINITE.)

The nng_aio_set_expire function is similar to nng_aio_set_timeout, but sets +an expiration time based on the system clock. The expiration +time is a clock timestamp, such as would be returned by nng_clock.

Wait for Completion

void nng_aio_wait(nng_aio *aio);
+

The nng_aio_wait function waits for an asynchronous I/O operation to complete. +If the operation has not been started, or has already completed, then it returns immediately.

If a callback was set with aio when it was allocated, then this +function will not be called until the callback has completed.

+ + important +

The nng_aio_wait function should never be called from a function that itself +is a callback of an nng_aio, either this one or any other. +Doing so may result in a deadlock.

Test for Completion

bool nng_aio_busy(nng_aio *aio);
+

The nng_aio_busy function returns true if the aio is currently busy performing an +operation or is executing a completion callback. Otherwise it return false. +This is the same test used internally by nng_aio_wait.

+ + important +

The caller is responsible for coordinating any use of this with any reuse of the aio. +Because the aio can be reused use of this function can be racy.

Result of Operation

int nng_aio_result(nng_aio *aio);
+size_t nng_aio_count(nng_aio *aio);
+

The nng_aio_result function returns the result of the operation associated +with the handle aio. +If the operation was successful, then 0 is returned. +Otherwise a non-zero error code, such as NNG_ECANCELED or NNG_ETIMEDOUT, is returned.

For operations that transfer data, nng_aio_count returns the +number of bytes transferred by the operation associated with the handle aio. +Operations that do not transfer data, or do not keep a count, may return zero for this function.

+ + note +

The return value from these functions is undefined if the operation has not completed yet. +Either call these from the handle’s completion callback, or after waiting for the +operation to complete with nng_aio_wait.

Messages

nng_msg *nng_aio_get_msg(nng_aio *aio);
+void nng_aio_set_msg(nng_aio *aio, nng_msg *msg);
+

The nng_aio_get_msg and nng_aio_set_msg functions retrieve and store a message +in aio. +For example, if a function to receive data is called, that function can generally be expected +to store a message on the asssociated aio, for the application to retrieve with +nng_aio_get_msg. +Conversely an application desiring to send a message msg will store it in the aio using +nng_aio_set_msg. The function implementing the send operation will retrieve the message +and arrange for it to be sent.

Message Ownership

For send or transmit operations, the rule of thumb is that implementation of the operation +is responsible for taking ownership of the message (and releasing resources when it is complete), +if it will return success. If the operation will end in error, then the message will be +retained and it is the consuming application’s responsibility to dispose of the message. +This allows an application the opportunity to reuse the message to try again, if it so desires.

For receive operations, the implementation of the operation will set the message on the aio +on success, and the consuming application hasa responsibility to retrieve and dispose of the +message. Failure to do so will leak the message. If the operation does not complete successfully, +then no message is stored on the aio.

I/O Vector

typedef struct nng_iov {
+    void * iov_buf;
+    size_t iov_len;
+};
+
+int nng_aio_set_iov(nng_aio *aio, unsigned int niov, nng_iov *iov);
+

For some operations, the unit of data transferred is not a message, but +rather a stream of bytes.

For these operations, an array of niov nng_iov structures can be passed to +the nng_aio_set_iov function to provide a scatter/gather array of +elements describing the location (iov_buf) and length (iov_len) of data, +to transfer.

The iov vector is copied into storage in the aio itself, so that callers may use stack allocated nng_iov structures. +The values pointed to by the iov_buf members are not copied by this function though.

A maximum of four (4) nng_iov members may be supplied.

+ + tip +

Most functions using nng_iov do not guarantee to transfer all of the data that they +are requested to. To be sure that correct amount of data is transferred, as well as to +start an attempt to complete any partial transfer, check the amount of data transferred by +calling nng_aio_count.

Inputs and Outputs

void nng_aio_set_input(nng_aio *aio, unsigned int index, void *param);
+void *nng_aio_get_output(nng_aio *aio, unsigned int index);
+

Asynchronous operations can take additional input parameters, and +provide additional result outputs besides the result code.

The nng_aio_set_input function sets the input parameter at index +to param for the operation associated with aio.

The nng_aio_get_output function returns the output result at index +for the operation associated with aio.

The type and semantics of input parameters and output results are determined by specific +operations. The documentation for the operation should provide details.

The valid values of index range from zero (0) to three (3), as no operation +currently defined can accept more than four parameters or return more than four additional +results.

+ + note +

If the index does not correspond to a defined input for the operation, +then nng_aio_set_input will have no effect, and nng_aio_get_output will +return NULL.

+ + important +

It is an error to call this function while the aio is currently +in use by an active asynchronous operation.

Error	Value	Description
`NNG_EINTR`	1	Operation interrupted.
`NNG_ENOMEM`	2	Out of memory, or other resource exahusted.
`NNG_EINVAL`	3	Invalid argument. The arguments are invalid or malformed somehow.
`NNG_EBUSY`	4	Resource busy.
`NNG_ETIMEDOUT`	5	Timed out. The operation took longer than the allotted time.
`NNG_ECONNREFUSED`	6	Connection refused. Usually indicates the wrong address or a server is running.
`NNG_ECLOSED`	7	Object closed. Typically the socket is closed.
`NNG_EAGAIN`	8	Try again. Typcally for a non-blocking operation that might succeed later.
`NNG_ENOTSUP`	9	Not supported. Perhaps the protocol or transport is not supported, or the operation is not not supported with the transport or protocol.
`NNG_EADDRINUSE`	10	Address in use. The network address is already used by another process. Most often this is seen for listeners.
`NNG_ESTATE`	11	Incorrect state. The operation cannot be performed in the current state, such as trying to send a response when no request has yet been received.
`NNG_ENOENT`	12	Entry not found (no such object.) Can also indicate that a file does not exist.
`NNG_EPROTO`	13	Protocol error. Typically this indicates incorrect messages over a network.
`NNG_EUNREACHABLE`	14	Destination unreachable.
`NNG_EADDRINVAL`	15	Address invalid. Like `NNG_EINVAL`, but only for network addresses.
`NNG_EPERM`	16	Permission denied.
`NNG_EMSGSIZE`	17	Message too large.
`NNG_ECONNABORTED`	18	Connection aborted. A connection attempt was aborted locally.
`NNG_ECONNRESET`	19	Connection reset. The remote peer reset the connection unexpectedly.
`NNG_ECANCELED`	20	Operation canceled. Typically as a result of `nng_aio_cancel` or similar.
`NNG_ENOFILES`	21	Out of files. Either the destination file system cannot store files, or all available file handles are used.
`NNG_ENOSPC`	22	Out of space. Destination table or filesystem is full.
`NNG_EEXIST`	23	Resource already exists.
`NNG_EREADONLY`	24	Read only resource. An attempt to modify a read-only file or other object.
`NNG_EWRITEONLY`	25	Write only resource. A read operation failed because the object only supports writes.
`NNG_ECRYPTO`	26	Cryptographic error. Usually indicates an invalid key was used for TLS.
`NNG_EPEERAUTH`	27	Peer could not be authenticated.
`NNG_ENOARG`	28	Option requires argument. A command-line option was supplied without an argument. Only used with `nng_opts_parse`.
`NNG_EAMBIGUOUS`	29	Ambiguous option. The command line option could not be unambiguously resolved. Only used with `nng_opts_parse`.
`NNG_EBADTYPE`	30	Incorrect type. A type-specific function was used for an object of the wrong type.
`NNG_ECONNSHUT`	31	Connection shutdown. The connection was shut down and cannot be used.
`NNG_EINTERNAL`	1000	An unidentifier internal error occurred.
`NNG_ESYSERR`	0x10000000 - 0x1FFFFFFF	An unidentified system error occurred. These are errors reported by the operating system.
`NNG_ETRANERR`	0x20000000 - 0x2FFFFFFF	An unidentified transport error occurred.

Messages

Messages in Scalability Protocols are the fundamental unit of +transmission and reception, as these protocols are fundamentally message-oriented.

Messages have a [body][nng_msg_body], containing the application-supplied +payload, and a [header][nng_msg_header], containing protocol specific routing and similar +related information.

+ + tip +

Only applications using raw mode need to access the message header. +Very few NNG applications do this.

Message Structure

typedef struct nng_msg nng_msg;
+

The nng_msg structure represents a single message. It carries a body +and a header.

Create a Message

int nng_msg_alloc(nng_msg **msgp, size_t size);
+

The nng_msg_alloc function allocates a new message. +It takes a size argument, and returns a message +with a preallocated body of that size in the msgp parameter.

If it succeeds, it returns zero, otherwise this function may return NNG_ENOMEM, +indicating that insufficient memory is available to allocate a new message.

Destroy a Message

void nng_msg_free(nng_msg *msg);
+

The nng_msg_free function deallocates a message.

Duplicate a Message

int nng_msg_dup(nng_msg **dup, nng_msg *msg);
+

The nng_msg_dup function duplicates the message msg, storing a pointer +to the new duplicate in dup. This function also returns zero on succes, or NNG_ENOMEM +if memory is exhausted.

Message Size and Capacity

size_t nng_msg_capacity(nng_msg *msg);
+int nng_msg_realloc(nng_msg *msg, size_t size);
+int nng_msg_reserve(nng_msg *msg, size_t capacity);
+

Messages have a certain amount of pre-reserved space, which may exceed the total +size of the message. This allows for content to be added to the message later, +without necessarily performing a reallocation.

The nng_msg_capacity function returns the amount of prereserved space. +If a message size change is required, and the new size will fit within the capacity +reported by this function, then change will be done without a reallocation, and +likely without a data copy as well.

+ + tip +

The capacity reported by nng_msg_capacity may not include reserved headroom, which +is present to allow a very limited amount of content to be inserted in front of the +message without requiring the rest of the message to be copied.

The message size may be changed by use of the nng_msg_realloc function. This +function will reallocate the underlying memory for the message msg, +preserving contents while doing so. +If the new size is smaller than the original message, it will +truncate the message, but not perform any allocations. +If reallocation fails due to insufficient memory, then the original is left intact.

The nng_msg_reserve function ensures that the total message capacity +is at least capacity bytes. Use of this function to ensure the total anticipated +capacity is present in the message may help prevent many small allocations.

Both nng_msg_realloc and nng_msg_reserve return zero on success, or may return +NNG_ENOMEM if insufficient memory exists to preform allocation.

+ + important +

Any pointers to message content obtained before a call to nng_msg_realloc or +nng_msg_reserve (or any other function that changes the message size) should be +treated as invalid, as the locations pointed to may be deallocated by these functions.

Message Body

void *nng_msg_body(nng_msg *msg);
+size_t nng_msg_len(nng_msg *msg);
+

The body and body length of msg are returned by nng_msg_body and +nng_msg_len, respectively.

Clear the Body

void *nng_msg_clear(nng_msg *msg);
+

The nng_msg_clear simply resets the total message body length to zero, but does +not affect the capacity. It does not change the underlying bytes of the message.

Add to Body

int nng_msg_append(nng_msg *msg, const void *val, size_t size);
+int nng_msg_append_u16(nng_msg *msg, uint16_t val16);
+int nng_msg_append_u32(nng_msg *msg, uint32_t val32);
+int nng_msg_append_u64(nng_msg *msg, uint64_t val64);
+
+int nng_msg_insert(nng_msg *msg, const void *val, size_t size);
+int nng_msg_insert_u16(nng_msg *msg, uint16_t val16);
+int nng_msg_insert_u32(nng_msg *msg, uint32_t val32);
+int nng_msg_insert_u64(nng_msg *msg, uint64_t val64);
+

Appending data to a message body is done by using the nng_msg_append functions. +The base nng_msg_append function appends size bytes of untyped data to the end of the +message.

Use of the typed versions, ending in suffixes _u16, _u32, and _u64 allows +for unsigned integers to be appended directly. The integers are encoded in network byte order, with +the most significant byte appearing first. The message body will by two, four, or eight +bytes accordingly.

Data may inserted before the rest of the message body by using the nng_msg_insert functions. +This will attempt to use “headroom” in the message to avoid a data copy. +Otherwise they are like the nng_msg_append functions except that the put the data in front +of the messages instead of at the end.

+ + tip +

Message headroom is limited, so nng_msg_insert is best used sparingly. +It is much more efficient to build the message content from start to end +using nng_msg_append.

Consume From Body

int nng_msg_chop(nng_msg *msg, size_t size);
+int nng_msg_chop_u16(nng_msg *msg, uint16_t *val16);
+int nng_msg_chop_u32(nng_msg *msg, uint32_t *val32);
+int nng_msg_chop_u64(nng_msg *msg, uint64_t *val64);
+
+int nng_msg_trim(nng_msg *msg, size_t size);
+int nng_msg_trim_u16(nng_msg *msg, uint16_t *val16);
+int nng_msg_trim_u32(nng_msg *msg, uint32_t *val32);
+int nng_msg_trim_u64(nng_msg *msg, uint64_t *val64);
+

The nng_msg_chop functions remove data from the end of the body of message msg, +reducing the message length by either size, or the appropriate value size.

The nng_msg_trim functions remove data from the beginning of the message body of msg. +but are otherwise just like the nng_msg_chop functions.

If the message is not big enough to remove requisite amount of bytes, these functions +return NNG_EINVAL. Otherwise they return zero.

Additionally, functions with typed suffixes (_u16, _u32, _u64) decode the data and return it +through the appropriate val pointer.

The data is assumed to have been in network byte order in the message, but is returned in +the native machine byte order. The appropriate number of bytes is consumed for each of these types, +so two bytes for _u16, four bytes for _u32, and eight bytes for _u64.

Message Header

void *nng_msg_header(nng_msg *msg);
+size_t nng_msg_header_len(nng_msg *msg);
+

The header and header length of msg are returned by nng_msg_header and +nng_msg_header_len, respectively.

The message headers are generally intended for limited use, to store protocol headers.

+ + important +

The message headers are for protocol and transport headers, and not for general +application payloads. Misuse of the header may prevent the application from functioning +properly.

Clear the Header

void *nng_msg_header_clear(nng_msg *msg);
+

The nng_msg_header_clear simply resets the total message header length to zero.

Append or Insert Header

Appending data to a message header is done by using the nng_msg_header_append functions, +and inserting data in the header is done using the nng_msg_header_insert functions.

These functions act just like the nng_msg_append and nng_msg_insert functions, +except that they operate on the message header rather than the message body.

Consume from Header

The nng_msg_header_trim functions remove data from the beginning of the message header, +and the nng_msg_header_chop functions remove data from the end of the message header.

These functions act just like the nng_msg_trim and nng_msg_chop functions, +except that they operate the message header rather than the message body.

Message Pipe

nng_pipe nng_msg_get_pipe(nng_msg *msg);
+void nng_msg_get_pipe(nng_msg *msg, nng_pipe p);
+

The nng_msg_set_pipe function sets the pipe associated with msg to p. +This is most often useful when used with protocols that support directing +a message to a specific peer. +For example the PAIR version 1 protocol can do +this when NNG_OPT_PAIR1_POLY mode is set.

The nng_msg_get_pipe function returns the pipe that was previously set on the message m, +either directly by the application, or when the message was received by the protocol.

+ + note +

Not all protocols support overriding the destination pipe.

Examples

Example 1: Preparing a message for use

    #include <nng/nng.h>
+
+    nng_msg *m;
+    if (nng_msg_alloc(&m, strlen("content") + 1) != 0) {
+       // handle error
+    }
+    strcpy(nng_msg_body(m), "content");
+

Example 2: Preallocating message content

    if (nng_msg_alloc(&m, 1024) != 0) {
+        // handle error
+    }
+    while ((val64 = next_datum()) != 0) P
+        if (nng_msg_append_u64(m, val64) != 0) {
+            // handle error
+        }
+    }
+

+ + + + + + +

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)

NNG Reference Manual (DRAFT)