Supercharging Devin + Supabase: Fixing Docker Performance on EC2 with overlay2

James Phoenix

The Problem

While setting up Devin (a coding assistant) with Supabase CLI on an EC2 instance, I encountered significant performance issues. After investigation, I discovered that Docker was using the VFS storage driver, which is known for being significantly slower than other storage drivers like overlay2.

The root cause was interesting: the EC2 instance was already using overlayfs for its root filesystem, which prevented Docker from using the overlay2 storage driver (you can’t nest overlay filesystems). Here’s what I initially saw when checking the system:

$ docker info | grep "Storage Driver"
Storage Driver: vfs

Using vfs as a storage driver can cause significant performance degradation, especially when:

Pulling Docker images
Building containers
Running containers with heavy I/O operations
Using tools like Supabase CLI that rely heavily on Docker

Initial System Analysis

When investigating the issue, I found these key characteristics of the system:

$ mount | grep overlay
overlayfs:/overlay/root on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/root,workdir=/overlay/work)

This showed that the root filesystem was using overlay, which explained why Docker defaulted to VFS. The system specifications were:

CPU: AMD EPYC with 8 cores
Memory: 31GB RAM
Root filesystem: Using overlayfs
Storage: 125GB available

The Solution

I developed a solution that creates a separate filesystem for Docker to use, bypassing the overlay filesystem limitations. Here’s the step-by-step process:

1. Stop Docker Services

First, you need to stop all Docker services:

sudo systemctl stop docker.socket
sudo systemctl stop docker.service

2. Create a Virtual Disk

Then you need create a large file to serve as a virtual disk:

sudo dd if=/dev/zero of=/docker.img bs=1G count=20

This command creates a 20GB file filled with zeros. The parameters are:

if=/dev/zero: Input file (source of zeros)
of=/docker.img: Output file (our virtual disk)
bs=1G: Block size of 1GB
count=20: Write 20 blocks

3. Format the Virtual Disk

Create an ext4 filesystem on our virtual disk:

sudo mkfs.ext4 /docker.img

This formats the file as an ext4 filesystem, which Docker can use efficiently with the overlay2 driver.

4. Create and Mount the Filesystem

# Create mount point
sudo mkdir -p /docker-data

# Mount the virtual disk
sudo mount -o loop /docker.img /docker-data

# Make mount persistent across reboots
echo "/docker.img /docker-data ext4 loop 0 0" | sudo tee -a /etc/fstab

The loop mount option allows us to mount a file as if it were a block device.

5. Configure Docker

Create a new Docker daemon configuration to use our new filesystem:

sudo bash -c 'cat > /etc/docker/daemon.json << EOF
{
  "data-root": "/docker-data",
  "storage-driver": "overlay2"
}
EOF'

This configuration:

Sets Docker’s root directory to our new filesystem
Explicitly sets the storage driver to overlay2

6. Set Permissions and Restart Docker

# Set correct ownership
sudo chown -R root:root /docker-data

# Start Docker service
sudo systemctl start docker.service

Verification

After implementing the solution, you can verify the changes:

$ docker info
Storage Driver: overlay2
Backing Filesystem: extfs
Docker Root Dir: /docker-data

The key improvements:

Storage driver changed from vfs to overlay2
Backing filesystem is now extfs (ext4)
Docker root directory is now on our separate filesystem

Why This Works

This solution works by providing Docker with a clean ext4 filesystem that isn’t nested within the system’s overlay filesystem. By mounting a file-based ext4 filesystem, you give Docker a suitable environment to use the overlay2 storage driver, which offers much better performance than vfs.

Performance Implications

The overlay2 storage driver provides several advantages over vfs:

Better layering and caching of image layers
More efficient use of storage space
Significantly faster container operations
Improved build and deployment times

Considerations

While this solution works well, there are a few things to consider:

The virtual disk file size is fixed (20GB in our example)
The solution requires root access to implement
You need enough free space on the root filesystem for the virtual disk file

Conclusion

For teams using Devin or Supabase CLI on EC2 instances where Docker defaults to the vfs storage driver, this solution provides a significant performance improvement. By creating a separate filesystem for Docker, you enable the use of the more efficient overlay2 driver, resulting in better overall performance for container operations.

Remember to adjust the virtual disk size based on your specific needs – the 20GB example can be modified as needed for your use case.

Last updated: December 13, 2024