From 16b4c4019c7b7904de171c588ed8c72ca732d2cf Mon Sep 17 00:00:00 2001 From: Garrett D'Amore Date: Wed, 9 May 2018 17:21:27 -0700 Subject: fixes #352 aio lock is burning hot fixes #326 consider nni_taskq_exec_synch() fixes #410 kqueue implementation could be smarter fixes #411 epoll_implementation could be smarter fixes #426 synchronous completion can lead to panic fixes #421 pipe close race condition/duplicate destroy This is a major refactoring of two significant parts of the code base, which are closely interrelated. First the aio and taskq framework have undergone a number of simplifications, and improvements. We have ditched a few parts of the internal API (for example tasks no longer support cancellation) that weren't terribly useful but added a lot of complexity, and we've made aio_schedule something that now checks for cancellation or other "premature" completions. The aio framework now uses the tasks more tightly, so that aio wait can devolve into just nni_task_wait(). We did have to add a "task_prep()" step to prevent race conditions. Second, the entire POSIX poller framework has been simplified, and made more robust, and more scalable. There were some fairly inherent race conditions around the shutdown/close code, where we *thought* we were synchronizing against the other thread, but weren't doing so adequately. With a cleaner design, we've been able to tighten up the implementation to remove these race conditions, while substantially reducing the chance for lock contention, thereby improving scalability. The illumos poller also got a performance boost by polling for multiple events. In highly "busy" systems, we expect to see vast reductions in lock contention, and therefore greater scalability, in addition to overall improved reliability. One area where we currently can do better is that there is still only a single poller thread run. Scaling this out is a task that has to be done differently for each poller, and carefuly to ensure that close conditions are safe on all pollers, and that no chance for deadlock/livelock waiting for pfd finalizers can occur. --- src/platform/posix/posix_pollq.h | 38 ++++++++++++-------------------------- 1 file changed, 12 insertions(+), 26 deletions(-) (limited to 'src/platform/posix/posix_pollq.h') diff --git a/src/platform/posix/posix_pollq.h b/src/platform/posix/posix_pollq.h index 2c855da1..b9786330 100644 --- a/src/platform/posix/posix_pollq.h +++ b/src/platform/posix/posix_pollq.h @@ -22,32 +22,18 @@ #include "core/nng_impl.h" #include -typedef struct nni_posix_pollq_node nni_posix_pollq_node; -typedef struct nni_posix_pollq nni_posix_pollq; - -struct nni_posix_pollq_node { - nni_list_node node; // linkage into the pollq list - nni_posix_pollq *pq; // associated pollq - int index; // used by the poller impl - int armed; // used by the poller impl - int fd; // file descriptor to poll - int events; // events to watch for - int revents; // events received - void * data; // user data - nni_cb cb; // user callback on event - nni_mtx mx; - nni_cv cv; -}; - -extern nni_posix_pollq *nni_posix_pollq_get(int); -extern int nni_posix_pollq_sysinit(void); -extern void nni_posix_pollq_sysfini(void); - -extern int nni_posix_pollq_init(nni_posix_pollq_node *); -extern void nni_posix_pollq_fini(nni_posix_pollq_node *); -extern int nni_posix_pollq_add(nni_posix_pollq_node *); -extern void nni_posix_pollq_remove(nni_posix_pollq_node *); -extern void nni_posix_pollq_arm(nni_posix_pollq_node *, int); +typedef struct nni_posix_pfd nni_posix_pfd; +typedef void (*nni_posix_pfd_cb)(nni_posix_pfd *, int, void *); + +extern int nni_posix_pollq_sysinit(void); +extern void nni_posix_pollq_sysfini(void); + +extern int nni_posix_pfd_init(nni_posix_pfd **, int); +extern void nni_posix_pfd_fini(nni_posix_pfd *); +extern int nni_posix_pfd_arm(nni_posix_pfd *, int); +extern int nni_posix_pfd_fd(nni_posix_pfd *); +extern void nni_posix_pfd_close(nni_posix_pfd *); +extern void nni_posix_pfd_set_cb(nni_posix_pfd *, nni_posix_pfd_cb, void *); #endif // NNG_PLATFORM_POSIX -- cgit v1.2.3-70-g09d2