So, i got persuaded to switch from a “server that is going to do everything” to “compute server + storage server”

The two are connected via a DAC on an intel x520 network card.

Compute is 10.0.0.1, Storage is 10.255.255.254 and i left the usable hosts in the middle for future expansion.

Before I start to use it, I’m wondering if i chose the right protocols to share data between them.

I set NFS and iSCSI.

With iSCSI i create an image, share that image on the compute server, format it as btrfs, use it as a native drive. Files are not accessible anywhere else.

With NFS i just mount the share and files can be accessed from another computer.

Speed:

I tried to time how long it takes to fill a dummy file with zeroes.

/iscsi# time sh -c "dd if=/dev/zero of=ddfile bs=8k count=250000 && sync"
250000+0 records in
250000+0 records out
2048000000 bytes (2.0 GB, 1.9 GiB) copied, 0.88393 s, 2.3 GB/s

real    0m2.796s
user    0m0.051s
sys     0m0.915s

/nfs# time sh -c "dd if=/dev/zero of=ddfile bs=8k count=250000 && sync"
250000+0 records in
250000+0 records out
2048000000 bytes (2.0 GB, 1.9 GiB) copied, 2.41414 s, 848 MB/s

real    0m3.539s
user    0m0.038s
sys     0m1.453s

/sata-smr-wd-green-drive-for-fun# time sh -c "dd if=/dev/zero of=ddfile bs=8k count=250000 && sync"
250000+0 records in
250000+0 records out
2048000000 bytes (2.0 GB, 1.9 GiB) copied, 10.1339 s, 202 MB/s

real    0m46.885s
user    0m0.132s
sys     0m2.423s

what i see from this results:

the sata slow drive goes at 1.6 gigabit/s but then for some reason the computer needs so much time to acknowledge the operation.

nfs transferred it at 6.8 gigabit/s which is what i expected from a nvme array. Same command on the storage server gives similar speed.

iscsi transfers at 18.4 gigabit/s which is not possible with my drives and the fiber connection. Probably is using some native file system trickery to detect “it’s just a file full of zeroes, just tell the user it’s done”

The biggest advantage of NFS is that I can share a whole directory and get direct access. Also sharing another disk image via iscsi requires a service restart which means i have to take down the compute server.

But with iscsi i am the owner of the disk so i can do whatever i want, don’t need to worry about permissions, i am root, chown all the stuff

So… after this long introduction and explanation, what protocol would you use for…:

  • /var/lib/mysql - a database. Inside a disk image shared via iscsi or via nfs?

  • virtual machine images. Copy them inside another image that’s then shared via iscsi? Maybe nfs is much better for this case. Otherwise with iscsi i would have a single giant disk image that contains other disk images…

  • lots of small files like WordPress. Maybe nfs would add too much overhead? But it would be much easier to backup if it was an NFS share instead of a disk image

  • TechAdmin@lemmy.worldEnglish
    1·
    1 year ago

    For the sata drive behavior it’s probably finishing the writes from buffer. I like to use the iotop utility to watch storage IO activity on my systems. Could try running it on both systems to get a better picture of what’s going on.

    I currently use NFS and CIFS but have used iSCSI in the past. I like the simplicity of NFS & CIFS and they meet my uses. iSCSI has it’s strengths as others have stated.

    • /var/lib/mysql - I would say iSCSI in it’s own image+lun. Should get lower latency as well as higher transfer rates compared to NFS for DB but it depends on the kinds & how much usage.
    • virtual machine images - I prefer NFS mounts for same reason, easier to work with the files directly. If you do go with iSCSI you can have different disk images for different kinds of VMs. Should be able to use both at same time on most hypervisors if you want to play with them too.
    • lots of small files - NFS should work without issue