DEV Community

Remdore
Remdore

Posted on

A DigitalOcean volume next to the Droplet's own disk

Every Droplet comes with its own SSD, and DigitalOcean also sells Volumes, network block storage you attach to a Droplet and can move to another one. I'd only ever used volumes for "more space", and never checked what they cost in speed or what moving one actually looks like. So I put a volume and a Droplet's own disk through the same tests, then moved the volume between two Droplets, once properly and once by detaching it while Postgres was writing to it.

The setup

Two s-2vcpu-4gb Droplets in fra1, which is the shared-CPU plan at $24 a month, running Ubuntu 24.04, and a 100 GiB volume at $10 a month. I let DigitalOcean format the volume as ext4 on creation. It doesn't add an fstab line; it writes a systemd mount unit, mnt-vt.mount, with defaults,nofail,discard,noatime.

DigitalOcean documents the limits for volumes, and they depend on the Droplet. On a shared-CPU Droplet a volume gets 7,500 IOPS and 300 MB/s, and can burst to 10,000 IOPS and 450 MB/s "for up to 60 seconds". The docs say nothing about latency.

I ran fio 3.36 with direct I/O on an 8 GiB file on each disk, local first, then the volume, with a 120 second pause before every volume run so its burst could reset. Then pgbench against two Postgres 16 instances with identical settings, one with its data directory on each disk.

The burst is exactly as documented

The volume's throughput over time: sequential reads at 450 MiB/s for about a minute, then 300; random writes at 10,000 IOPS for about 80 seconds, then 7,500

Sequential reads in 1 MiB blocks ran at 450 MiB/s for 62 seconds and then dropped to a flat 300. Sequential writes did the same, within a second. Random 4 KiB writes with 32 in flight held 10,000 IOPS for about 80 seconds and then sat on 7,500. These are hard limits, not rough ones. The line is almost perfectly straight once it drops.

The Droplet's own disk did 4,102 MB/s sequential reads and 1,206 MB/s writes over the same runs, and 39,703 random write IOPS. It wasn't close on throughput, and nobody should expect it to be.

Test Droplet's disk Volume
Sequential read, 1 MiB, 150 s 4,102 MB/s 378 MB/s
Sequential write, 1 MiB, 150 s 1,206 MB/s 378 MB/s
Random read, 4 KiB, 32 in flight 75,274 IOPS 7,162 IOPS
Random write, 4 KiB, 32 in flight 39,703 IOPS 8,833 IOPS
Random read, 4 KiB, one at a time 4,391 IOPS 1,006 IOPS
Random write, 4 KiB, one at a time 2,575 IOPS 3,432 IOPS
4 KiB write + fdatasync, one at a time 2,133/s 838/s

The volume's 378 MB/s averages are the 450 burst and the 300 after it, blended over 150 seconds.

Latency is where it shows up

Median latency with one operation in flight: random read 146 µs local vs 700 µs on the volume; random write 272 vs 169; fdatasync 358 vs 922

With one request at a time, which is how a lot of code reads, the limits don't matter. Latency does. A random 4 KiB read took a median of 146 µs on the local disk and 700 µs on the volume, and at the 99th percentile, 0.95 ms against 5.7 ms. Every read is a trip over the network.

Writes went the other way. A 4 KiB write took 169 µs on the volume and 272 µs locally. The kernel reports a write cache on the volume's disk, so the write may be acknowledged before it's stored anywhere permanent. The flush shows the real cost: fdatasync after a write took a median of 922 µs on the volume and 358 µs locally.

Random reads with 32 in flight never settled on the volume. Second by second they swung between 2,552 and 10,809 IOPS, averaging 7,162, under the limit rather than pinned to it.

Postgres

pgbench at scale 100, about 1.5 GB, with 8 clients for 120 seconds, three runs on each disk, alternating:

Run Droplet's disk Volume
1 1,324 tps 908 tps
2 1,585 tps 866 tps
3 1,671 tps 1,070 tps

The volume was about a third slower, with average latency of 7.5 to 9.2 ms against 4.8 to 6.0 ms. The database fits in memory here, so I think this is mostly the cost of flushing the WAL on every commit, the same fdatasync from the table above. Loading the data and building the indexes took 37.0 seconds on the volume and 22.9 locally.

Moving the volume properly

This is the part a local disk can't do. I ran two writers on the volume on the first Droplet: one appending a numbered line to a file and fsyncing after each, one inserting rows into Postgres. Each logged what it had been told was saved, on the Droplet's own disk. Then I stopped them, stopped Postgres, unmounted, and moved the volume.

The detach took 8.2 seconds from API request to completed, and the attach 7.2. The disk appeared on the second Droplet 3.4 seconds after the attach request, almost four seconds before the API said it was done. Mounting took 0.6 seconds, and Postgres started with "database system was shut down", no recovery needed. From the detach request to Postgres accepting connections on the other Droplet took 16.8 seconds.

Nothing was missing. The second Droplet had 8,424 rows and 5,790 lines, one more of each than the writers had logged as saved. That last write landed, but the writer was killed before it could log it.

Detaching it while Postgres was writing

Next I skipped the unmount. With both writers running and Postgres up, I asked the API to detach the volume. I expected it to refuse, since the volume was mounted and busy. It accepted.

About three seconds after the request (2.7 and 3.3 seconds in two runs), the writes started failing with Input/output error. The kernel logged "device offline error", ext4 shut the filesystem down, and Postgres stopped with:

PANIC:  could not fdatasync file "000000010000000100000008": Input/output error
Enter fullscreen mode Exit fullscreen mode

The filesystem still showed as mounted. The API reported the detach complete four to five seconds after the errors began.

Then I attached the volume to the other Droplet. ext4 replayed its journal on mount ("recovery complete"), and Postgres ran crash recovery, which took 0.08 seconds of redo. Postgres had confirmed 16,800 inserts before the error, and all 16,800 rows were there. The file writer had fsynced 11,686 lines, and all 11,686 were there.

In the first forced run, the volume had one line more than the writer had confirmed. The last write was on disk even though its fsync had returned an error, which is the uncomfortable direction for an application that retries on errors.

What I got wrong

I expected the volume to lose every test. It was faster for single 4 KiB writes, and I can only point at the write cache as the likely reason. I didn't prove it.

I expected the API to refuse to detach a mounted volume that was being written to. It doesn't check, and from the Droplet's side the disk simply disappears.

I expected the 32-in-flight random reads to sit on the documented 10,000 IOPS burst like the writes did. They never got there for long.

My Postgres writer broke on the first forced run. The table already existed, CREATE TABLE IF NOT EXISTS printed a notice, and my script read the notice as the answer to the next query and stopped after one row. I fixed it and ran the forced test again in the other direction, which is where the 16,800 rows come from.

And I stopped the writers with pkill -f writer.py, which matched the SSH command that ran it and killed my own session. It's an old trap, and I still walked into it.

Caveats

One Droplet size with a shared CPU, one region, one volume. Each fio test ran once, pgbench three times. The local disk's 4.1 GB/s reads may come partly from a cache on the host, which I can't see from inside the Droplet. The volume limits are higher on dedicated-CPU Droplets, 10,000 IOPS and 450 MB/s before any burst, and I didn't test those. The forced detach ran twice. The whole experiment ran for 70 minutes and cost about 7 cents.

What I'd take from it

  • Put data on a volume because it has to outlive the Droplet or move between Droplets, not for speed. On a shared-CPU Droplet the volume is capped at 300 MB/s and 7,500 IOPS, and a burst lasts about a minute.
  • Expect single reads around 0.7 ms instead of 0.15, and flushes around 0.9 ms. A database that commits often will feel the flush.
  • Moving a volume properly took 17 seconds end to end, most of it the API's own detach and attach.
  • Unmount before you detach. DigitalOcean won't stop you, and the Droplet gets I/O errors three seconds later. In my runs nothing that had been confirmed was lost, but the application on that Droplet crashes.

Top comments (1)

Collapse
 
mariosfong profile image
Mario Fang •

These are the numbers I wanted when someone told me to "just put it on a volume". The fdatasync line is the one that would have saved me a week: 922 µs against 358 µs is a 2.5x hit on every commit, and no throughput test shows it, so the first symptom is a write-heavy job that suddenly takes three times as long and looks like a query problem.

I hit exactly this once, moving PGDATA onto network block storage for capacity reasons. It presented as the backup window going from 40 minutes to two and a half hours, because the dump was bound on fsync rather than on MB/s. PGDATA went back to the local disk and the volume kept only the WAL archive and the nightly dumps, since archiving does not care about per-commit latency.

One question, since you had the burst behaviour so cleanly: did the 300 MB/s post-burst figure hold on a long sequential write, like a 100 GB restore? Sixty seconds of burst is fine for a benchmark, but an hour-long restore is exactly the workload where the drop matters, and I have never seen that measured honestly.