• 0 Posts
  • 12 Comments
Joined 1 year ago
cake
Cake day: June 20th, 2023

help-circle



  • Myself over NFS can have serious latency issues. Some software can’t correctly file lock over NFS too which will cause write latency or just full blown errors.

    iSCSI drops however can be really really bad and cause full filesystem corruption. Also backing up iSCSI volumes can be tricky. Software will likely work better and feel happy however and underlying issues may be masked until they’re unfixable. (I had an iSCSI volume attached to vmware silently corrupt for months before it failed and lost the data even though all scrubs/checksums were good until the very least moment).

    You can make your situations with with either technology, both are just as correct. Would get a touch more throughput on iSCSI simply down to the write confirmation being drive based and not filesystem locks / os based.

    YMMV









  • Ceph works best if you have identical osd, quantity, type and capacity across the cluster, also works best on a 3+ node cluster.

    I ran a mixed sata SSD/HDD 256gb/4tb cluster and it was always a bit pants. Now I have 7x1tb SSD per node (4nodes) and it works fantastic now.

    Proxmox uses replica 3/2 failure at host level but you may find that EC works better for your mixed infra as you noticed you can’t meed the 3 host failure and so setting to osd failure level means data may be kept on a single host so would need to traverse the network to the other machine.

    You may also need more than a single 10Gb nic too as you might start hitting bandwidth issues.