Mysteries of a write()
First of all, let’s agree that a call to write() on a “regular file” does not usually make data durable on disk. Instead, it copies data from an application’s buffer into the kernel page cache. There are, however, exceptions to this rule.
write() is user-space library wrapper around the write system call in the Linux kernel that writes data from a buffer to a file. It is one of those really minimal system call wrappers with an intuitive interface but it hides alot of complexity that can make things go wrong if not properly used.
In this article, we are going to explore how this simple library call hides the complexity involved in writinh data and how that complexity makes write operations in storage systems prone to errors.
A successful write call might not be fully successful after all
The Linux manual and POSIX pages state that write() returns the number of bytes written on success otherwise -1 is returned and errno set appropriately. However, this is easy to misunderstand. A successful call might not transfer all the bytes requested by the program.
1
2
// Requests kernel to copy 50 bytes from buf into the open file pointed to by the fd 10
write(10, buf, 50)
The assumption is that if this call succeeds, then the return value of the function will be 50. That is a wrong assumption because the kernel might only copy 20 out of the 50 bytes resulting into what is known as a partial write / short-write. This might happen if there is insufficient space in the underlying physical medium or if the call gets interrupted after at least one byte has been written.
Well-written I/O programs must therefore write data in a loop until either all requested bytes have been written (that is, the total number of bytes written equals the number originally requested) or the write call explicitly returns -1. As so;
1
2
3
4
5
6
7
8
9
10
11
int written = 0;
char *buffer = "Hello world";
int nbyte = sizeof(buffer);
while (written < nbyte){
ssize_t ret = write(fd, buffer+written, nbyte-written);
if (ret == -1)
return -1;
written += ret;
}
Another common mistake is treating a short write as a failure. For example;
1
2
3
4
5
6
char *buffer = "Hello world";
int nbyte = sizeof(buffer);
ssize_t ret = write(fd, buffer, nbyte);
if (ret != nbyte)
return -1;
The POSIX standard states;
upon succesful completion, the function shall return the number of bytes actually written to the file, which under some circumstances might be less than what the user requested to be written.
A successful write() call doesn’t always write data to disk
By default, write() only writes data from an application buffer into the kernel page cache. Writing that data to disk requires another system call(fsync or fdatasync), or requires the file to be opened with either O_SYNC or O_DSYNC flags which leaves durability guarantees to the kernel.
I might talk about this in a future article; how the kernel gets dirty pages from the page cache to disk. But just as a synopsis, when an application calls write(), the kernel copies data from your user space buffer into its page cache, and marks those pages dirty. A background flusher threads will run at configured intervals to write dirty pages based on several conditions, including dirty-page age, dirty-memory thresholds, memory pressure, filesystem behavior, and explicit sync operations
The danger of relying solely on the page cache is that data loss might occur incase the system crashes or there is power loss before the flusher thread flushes dirty pages to disk.
Example 1; writing data to page cache, and leaving durability to the kernel flusher thread(prone to data loss on power loss or system crash)
1
2
3
4
5
6
int fd = open("/path-to-file", O_CREAT | O_RDWR, 0644);
char *buf = "Hello!";
size_t nbyte = strlen(buf);
ssize_t ret = write(fd, buf, nbyte);
Example 2; the call only returns after data has been made durable on disk(ideally once the storage stack has acknowledged the required durability guarantees.)
1
2
3
4
5
6
int fd = open("/path-to-file", O_CREAT | O_RDWR | O_DSYNC, 0644);
char *buf = "Hello!"
size_t nbyte = strlen(buf);
ssize_t ret = write(fd, buf, nbyte);
Example 3; the write call returns once the data is in the page cache, the application then issues a subsequent sync call to force data from page cache to disk.
1
2
3
4
5
6
7
8
9
int fd = open("/path-to-file", O_CREAT | O_RDWR, 0644);
char *buf = "Hello!"
size_t nbyte = strlen(buf);
ssize_t ret = write(fd, buf, nbyte);
// fsync ensures that only pages belonging to the file pointed to by the open file descriptor fd are made durable. Otherwise, `sync()` flushes any dirty page regardless of the file descriptor.
fsync(fd);
Write() has a data transfer limit
The implementation for this rule might vary accross different kernels, so we wil focus on the Linux kernel.
For Linux, the maximum amount of data the write() system call can transfer in a single operation is bounded by 0x7ffff000 (2,147,479,552) bytes.
This means that if your write call tries transferring more than this limit, the kernel will only copy up to the limit and return the number of bytes actually written.
A common approach for writing really large data is doing it in a loop, so that all data within the user space buffer is transferred to the page cache.
Multi-threaded writes are even more subtle
The retry loop shown above works correctly when a single thread owns the file descriptor. However, things become much more complicated once multiple threads share the same file descriptor.
The problem is that write() uses the file’s current file offset, and that offset is shared by every thread using the same open file description. If one thread experiences a short write, another thread can successfully complete its own write, advancing the shared file offset before the first thread retries.
For example, suppose Thread A attempts to write an 11-byte record but only 5 bytes are written:
1
2
3
4
Thread A
write(fd, "Hello world", 11)
returns 5
At this point, the shared file offset has already advanced by 5 bytes.
Before Thread A retries, Thread B writes its own 11-byte record:
1
2
3
4
Thread B
write(fd, "ABCDEFGHIJK", 11)
returns 11
The shared file offset now points 16 bytes into the file. When Thread A retries its remaining 6 bytes, the kernel writes them at the current file offset (16), not immediately after the first 5 bytes written earlier.
Linux atomically updates the shared file offset for each individual write() call. The problem is not that concurrent writes overlap; the problem is that a retry after a short write is a new write() call, and another thread may have legitimately advanced the shared file offset before that retry occurs.
The resulting file looks something like this:
1
2
3
Offset 0: Hello
Offset 5: ABCDEFGHIJK
Offset 16: world
The original "Hello world" record has now been split apart by another thread’s write. This is known as record interleaving, and it occurs because each retry uses the shared file offset rather than the offset where the previous attempt stopped.
For this reason, retry loops are only safe when a thread has exclusive ownership of the file descriptor. In concurrent storage systems, it is generally preferable to use pwrite() (which specifies an explicit file offset) or to serialize writes so that multiple threads do not compete for the same file offset.
Write() is not always the correct write method to use
Depending on what you want to achieve, Linux and any other kernel provide various interfaces to write data to a file.
Among them include pwrite(), pwritev(), writev(), io_uring.
pwritev() and writev() provide support for vectored I/O, meaning you can write data from multiple buffers(referred to as iovecs) with this single system call instead of issuing multiple syscall requests for each buffer.
Conclusion
The write system call lies at the heart of modern computing which essentially involves reading and writing data. Understanding its semantics allows system programmers and storage engine developers to reason about failure modes and write software that remains correct under adverse conditions.