Zero-Copy Networking

Zero-copy networking eliminates the need to copy data between user space and kernel space, dramatically reducing CPU overhead and improving performance. In traditional networking, data is copied multiple times as it moves through the system.

The Problem: Multiple Data Copies

Traditional networking involves several data copies:

// Traditional file transfer with multiple copies
int fd = open("data.txt", O_RDONLY);
char buffer[4096];
read(fd, buffer, 4096);        // Copy 1: Disk → Kernel buffer
send(socket, buffer, 4096, 0); // Copy 2: Kernel buffer → Socket buffer
// Copy 3: Socket buffer → Network card buffer

Data Flow in Traditional Networking:

Disk → Kernel Buffer → User Buffer → Socket Buffer → Network Card
     ↑ Copy 1        ↑ Copy 2      ↑ Copy 3

Each copy consumes CPU cycles and memory bandwidth, limiting performance.

Zero-Copy System Calls

1. sendfile() System Call

sendfile() allows sending a file directly from disk to network socket without copying data to user space.

// Zero-copy file transfer with sendfile()
int fd = open("data.txt", O_RDONLY);
int sock = socket(AF_INET, SOCK_STREAM, 0);
// ... setup socket ...

// Direct transfer: Disk → Socket → Network
ssize_t sent = sendfile(sock, fd, NULL, file_size);
// No data copying to user space!

Data Flow with sendfile():

Disk → Socket Buffer → Network Card
     ↑ Direct transfer (no user space copy)

2. splice() System Call

splice() moves data between file descriptors without copying it to user space.

// Zero-copy data movement with splice()
int pipe_fds[2];
pipe(pipe_fds);

// Move data from file to pipe
splice(file_fd, NULL, pipe_fds[1], NULL, size, SPLICE_F_MOVE);

// Move data from pipe to socket
splice(pipe_fds[0], NULL, socket_fd, NULL, size, SPLICE_F_MOVE);
// Data never touches user space

3. tee() System Call

tee() copies data between pipes without copying to user space.

// Copy data between pipes
int pipe1[2], pipe2[2];
pipe(pipe1);
pipe(pipe2);

// Copy from pipe1 to pipe2 without user space copy
tee(pipe1[0], pipe2[1], size, 0);

4. vmsplice() System Call

vmsplice() efficiently transfers user memory to kernel space.

// Efficient user-to-kernel transfer
struct iovec iov = {
    .iov_base = user_buffer,
    .iov_len = buffer_size
};
vmsplice(pipe_fd, &iov, 1, SPLICE_F_GIFT);
// Memory is "gifted" to kernel, no copying

Memory Mapping for Zero-Copy

Memory mapping allows direct access to files without copying.

// Memory-mapped file access
int fd = open("data.txt", O_RDONLY);
char* mapped_data = (char*)mmap(NULL, file_size, PROT_READ, MAP_PRIVATE, fd, 0);

// Direct access to file data in memory
send(socket, mapped_data, file_size, 0);
// No copying from disk to user space

munmap(mapped_data, file_size);
close(fd);

Network Card Zero-Copy

Modern network cards support zero-copy operations directly.

Scatter-Gather I/O

// Scatter-gather with multiple buffers
struct iovec iov[3];
iov[0].iov_base = header;
iov[0].iov_len = header_size;
iov[1].iov_base = payload;
iov[1].iov_len = payload_size;
iov[2].iov_base = footer;
iov[2].iov_len = footer_size;

// Send multiple buffers as single message
writev(socket, iov, 3);
// Network card handles gathering without copying

Direct Memory Access (DMA)

Network cards can directly access system memory without CPU involvement.

// DMA-enabled network card
struct dma_buffer {
    void* virtual_addr;
    uint64_t physical_addr;
    size_t size;
};

// Register memory region for DMA
dma_buffer buffer = register_dma_memory(size);

// Network card directly reads/writes this memory
// No CPU involvement, no copying

Use Cases

High-Frequency Trading Example

// Traditional approach with copying
while (true) {
    // Receive market data
    char buffer[1024];
    recv(socket, buffer, 1024, 0);  // Copy to user space

    // Process data
    process_market_data(buffer);

    // Send response
    char response[512];
    prepare_response(response);
    send(socket, response, 512, 0);  // Copy from user space
}

// Zero-copy approach
while (true) {
    // Direct memory access
    market_data* data = get_mapped_market_data();
    process_market_data(data);

    // Direct send without copying
    send_direct(socket, response_buffer, response_size);
}

Web Server Example

// Traditional file serving
int fd = open("file.html", O_RDONLY);
char buffer[4096];
while ((bytes = read(fd, buffer, 4096)) > 0) {
    send(client_socket, buffer, bytes, 0);  // Copy for each chunk
}

// Zero-copy file serving
int fd = open("file.html", O_RDONLY);
struct stat st;
fstat(fd, &st);
sendfile(client_socket, fd, NULL, st.st_size);  // Single operation

Performance Benefits

Latency Reduction

Operation	Traditional	Zero-Copy	Improvement
File Transfer	1000+ cycles	100-200 cycles	5-10x
Network Send	500+ cycles	50-100 cycles	5-10x
Memory Copy	200+ cycles	0 cycles	∞

Throughput Improvement

// Performance comparison
// Traditional: Multiple copies limit throughput
// Zero-copy: Direct transfer maximizes throughput

// Example: 10GB file transfer
// Traditional: 2-5 Gbps (limited by copying)
// Zero-copy: 10-40 Gbps (limited by network)

Requirements and Limitations

Hardware Requirements

• Network Cards: Must support scatter-gather and DMA
• CPU: Modern CPUs with efficient memory controllers
• Memory: Sufficient bandwidth for direct access

Software Requirements

• Kernel Support: Linux 2.6+ for sendfile(), splice()
• File Systems: Support for direct I/O
• Applications: Must be designed for zero-copy

Limitations

1. sendfile() Limitations

// sendfile() only works with files
// Doesn't work with user buffers
sendfile(socket, file_fd, NULL, size);  // ✓ Works
sendfile(socket, user_buffer, NULL, size);  // ✗ Doesn't work

2. Memory Alignment

// DMA requires proper memory alignment
void* buffer = aligned_alloc(4096, size);  // 4KB alignment
// Required for efficient DMA operations

3. Buffer Management

// Zero-copy requires careful buffer management
// Buffers must remain valid during transfer
// No premature deallocation

Advanced Zero-Copy

Kernel Bypass Zero-Copy

DPDK and other kernel bypass techniques provide additional zero-copy capabilities.

// DPDK zero-copy packet processing
struct rte_mbuf *pkts[32];
uint16_t nb_rx = rte_eth_rx_burst(port_id, 0, pkts, 32);

for (int i = 0; i < nb_rx; i++) {
    // Direct access to packet data
    char* data = rte_pktmbuf_mtod(pkts[i], char*);
    process_packet(data, pkts[i]->data_len);

    // Reuse buffer for transmission
    rte_eth_tx_burst(port_id, 0, &pkts[i], 1);
}

RDMA Zero-Copy

RDMA provides the ultimate zero-copy experience.

// RDMA zero-copy
// Sender: Direct memory access
ibv_post_send(qp, &wr, &bad_wr);

// Receiver: Data appears directly in memory
// No syscalls, no copying, no CPU involvement

Best Practices

1. Choose the Right Technique

// For file transfers: use sendfile()
sendfile(socket, file_fd, NULL, file_size);

// For user buffers: use splice() or vmsplice()
splice(pipe_fd, NULL, socket_fd, NULL, size, SPLICE_F_MOVE);

// For maximum performance: use kernel bypass
// DPDK, RDMA, or custom drivers

2. Buffer Management

// Pre-allocate and reuse buffers
struct buffer_pool {
    void* buffers[MAX_BUFFERS];
    size_t buffer_size;
    int available;
};

// Avoid dynamic allocation in hot paths
// Use memory pools for predictable performance

3. Memory Alignment

// Ensure proper alignment for DMA
void* buffer = aligned_alloc(4096, size);
// 4KB alignment for most efficient DMA operations

The Bottom Line

Zero-copy networking eliminates unnecessary data copying, dramatically improving performance for high-throughput applications. The choice of technique depends on your specific use case, from simple sendfile() for file transfers to complex kernel bypass solutions for maximum performance.

Key Takeaways:

• Zero-copy eliminates data copying between user and kernel space
• sendfile() provides zero-copy file transfers
• splice() enables zero-copy data movement
• Memory mapping provides direct file access
• Hardware DMA enables true zero-copy networking
• Performance improvements can be 5-10x or more
• Careful buffer management is essential

Questions

Q: What is zero-copy networking?

Zero-copy networking eliminates the need to copy data between user space and kernel space, reducing CPU overhead and improving performance.

Q: What is the sendfile() system call used for?

sendfile() allows sending a file directly from disk to network socket without copying data to user space, implementing zero-copy file transfer.

Q: What is the main benefit of zero-copy techniques?

Zero-copy techniques reduce CPU overhead by eliminating data copying, which improves throughput and reduces latency in high-performance networking.

Q: What is splice() used for?

splice() moves data between file descriptors without copying it to user space, enabling zero-copy data transfer between different sources.

Q: When is zero-copy most beneficial?

Zero-copy is most beneficial for large data transfers and high-throughput applications where the overhead of data copying would significantly impact performance.