A dive into NodeJS I/O

Network Sockets, Files, DNS and threads

Node is a non-blocking JavaScript runtime. You can concurrently run an HTTP Server, read files from disk, send UDP datagrams, accept TCP connections from clients and still have room to execute JavaScript code operations without blocking. Most of these operations are known as I/O, you send an input to a device, file or a socket and it replies back with an output. Node achieves non-blocking I/O with mostly a single thread executed asynchronously using a library called lib_uv.

The exception being DNS queries, which use a thread pool. This means when you use fetch or axios to make an HTTP request to a domain, the DNS resolution for that domain will most probably go through the thread pool, while the actual request itself will be sent asynchronously on the main thread.

So when does Node block and when it doesn’t? I attempt to answer this question in this article by diving deep into the NodeJS I/O system.

libuv library in NodeJS

Socket I/O

When a backend application binds to an address and a port it creates a socket. Attempts to conenct to the socket will spawn connections that can be accessed through file descriptors. File descriptors are integer values representing TCP connections, UDP sockets or even literal files on disk. When a client sends data on the connection, the server operating system reads the data from the NIC. Using the from and to IP Address and port, the OS maps the data to a file descriptor and puts it in the kernel buffer. It is now up to the application logic to explicitly read the data on the connection so it can be moved from the kernel memory to the application’s dedicated memory. Calling read(2) from the application will do exactly that.

This is all good and well when there is data in the buffer for that connection, but what if there isn’t anything in the buffer? the client didn’t send anything yet. The read call from our backend will block, the OS will move the application process out of the CPU so the CPU can do better stuff, while our app remains blocked until someone sends data on that connection. This behavior is called synchronous I/O and it is mostly undesirable because guess what, not only the CPU has things to do, the app also has other things to do. Thus asynchronous I/O was born.

Asynchronous I/O was designed initially based on the idea of readiness and eventing to solve this exact problem, at least in Linux, (a knee-jerk reaction if you asked me). We split the I/O into two calls, a call select to check if the file descriptor is ready (has something to read) and the second call to perform the read. If select returns ready, we know that read will not block us, else if select returned not ready, our app can move on to do something else meanwhile. Variations of this model were developed to improve asynchronous I/O most popular being epoll. Details are irrelevant for this post, but select, poll and epoll are based on the idea of eventing. Here are a bunch of file descriptors tell me whenever any of them are ready for me to read.

We could have modified the read function to no-op when there isn’t anything in the file descriptor instead of blocking. My guess is this would have created bad patterns where the app makes a lot of unnecessary calls to the OS which will hinder performance eventually. I could argue the read calls blocks on purpose because of this reason.

File I/O

Reading and writing to files didn’t benefit from the asynchronous eventing model described above because there are no events when it comes to files. Unlike network sockets where the app can be notified when data arrives, the file data is always there. I suppose you can use the model to listen to changes on a file and notify the app but use cases for that are limited.

Calling read on files will still block but not because of absence of data but because it just takes time to read from HDD or SSD. The read transfers the requested blocks from the disk controller to the OS cache and then to the application. In the few milliseconds that the read is blocking, the app could do other things so an asynchronous model for reading files is still desirable.

If the requested portion of the file happened to be the OS cache, the read will be faster. Whether you consider the read to be blocking in that particular case is up for debate. It sounds like there is a specific definition to what blocking really is but to me if the app is waiting for something, it is blocked, whether it is 10 milliseconds or 10 nanoseconds. The app can do something while it waits. You may argue that the cost to switch to do something else will be slower than the time the app waits the 10 nanoseconds, however today I think it is just a technical limitation.

What do you do if you don’t want to wait in line? You send someone who does.

We may spin up a thread to perform the blocking read while keeping the main thread block free. The logical read operation becomes asynchronous non-blocking from a user perspective but technically speaking, it is still synchronous and blocking to the OS, it just the thread that is blocked instead of the main process.

DNS

DNS is a protocol that resolves domains and hostnames to network addresses. It is built on top of UDP and more recently on top of TCP/QUIC via DoT and DoH for encrypted DNS. We know how to do socket I/O asynchronously so you would think DNS would fit right in. This is true if you rolled out your own DNS implementation, however most libraries, framework and runtimes call the existing OS implementation of DNS which is getaddrinfo(3). This method is synchronous and blocking and it will require the thread pool trick we did with files to call it asynchronously. Why is it synchronous? I couldn’t find an answer on the web, but as always doesn’t hurt to guess. My guess that getaddrinfo function isn’t just doing DNS resolution via the network, but it might be reading the hosts file and getting any user defined resolution from there. I know that there is an asynchronous version of getaddrinfo function but it doesn’t seem to be used as often.

NodeJS Async Single Thread

NodeJS is a single threaded runtime and most of the work is done asynchronously with that single thread. NodeJS uses whatever tools available to it through the OS to perform the tasks asynchronously. In Linux it uses epoll, in MacOSx it uses kqueue, for SunOS it uses event ports and for Windows it uses the IOCP. Regardless of the method performed, after the asynchronous I/O is done, Node calls the user callbacks that are scheduled in the event main loop.

Take the TCP server in Node for example, as developers writing JavaScript code, we never read from connections. Instead we listen to an event with a callback function and whenever there is data, Node calls our function. Behind the scenes Node is always reading from the file descriptor asynchronously. In Linux, it uses epoll passing the file descriptor (connection) and when the connection has data the OS notifies Node which turns around and reads the socket. This will copy the data from kernel buffer to Node user space which finally calls our function with the data. This is identical to HTTP Server and in turn Express.

socket.on('data', chunk => console.log(`Got data from client: ${chunk.toString()`); );

NodeJS Thread pool

There are cases where asynchronous ready-based eventing is not possible and where the call must block. Examples are files read/write or DNS lookup as we explained earlier. If we put the blocking calls on the main thread event loop, Node will block starving other tasks in the process. Node uses a thread pool to address this problem. When a blocking call (e.g. DNS lookup) is scheduled in Node, a thread is assigned that blocking call and whenever the thread is done it creates a callback in Node main thread where the next iteration of the loop will pick it up and execute the callback with the result. DNS lookups, File I/O and user specific workload can use the thread pool. Some CPU intensive libraries in Node such as the crypto library uses the thread pool too. All this is implemented in the lib_uv library which Node uses.

Linux new shiny asynchronous I/O io_uring is based on completion instead of readiness and it supports all file descriptors. This is relatively new technology and lib_uv has an open pull request to implement it. (Merged since this article has been authored)

It is possible to not use the asynchronous version of readfile and opt in for synchronous version. Those calls ends in Sync, (e.g. fs.readFileSync()), calling this function will block the event main loop. Why you would do that? Maybe you want your code to look pretty, maybe you are using the threads for some other more important things.

DNS lookup bottleneck in Node

As discussed, DNS lookup happens in the background using the thread pool. When calling fetch the domain in the URL needs to be resolved to an IP address (IPv4 or IPv6) so fetch can establish a TCP connection with the server. It could be the case where all the threads in the thread pool are busy (blocked) looking up DNS which will block the thread pool from doing other DNS resolution or even asynchronous file I/O.

You can of course increase the thread pool size UV_THREADPOOL_SIZE but once that number exceeds the number of hardware threads on your CPU it can backfire. So Node provides DNS.resolve() to resolve DNS that always uses the network and as a result it is asynchronous. The problem is how do you know what DNS method your library is using. Leaky abstractions.

DNS lookup is slightly different from DNS resolving. Lookup indicates that you might have the entry cached locally or might need to read it in the hosts file. While resolving indicate that you actively do a network call to a DNS resolver to find the result.

I hope you enjoyed this article. If you are interested to learn more about NodeJS check out my NodeJS Internals udemy course, get a discount coupon and support my work link redirects to udemy with coupon applied. Thank you!