The follow-up from part 1 is here for you to digest. Soon ipHouse will have new storage online!
This report is where I describe the hardware and guest operating systems in a different format and include the output from bonnie++ and Postmark. I opted to skip the Iometer benchmarks because then this post would just be multiple screenshots or I would need to parse it all into graphs.
Tintri T540 vs Tegile Zebi HA2200EP vs Nexsan E5000 series, and an old OpenSolaris 132 system. Please check the hardware link provided to read up on the specifications of each appliance.
Test system configuration
Physical host (Dell 2900) running vSphere 5.0
- 32 GiB RAM
- 1 4-core Intel E5420 2.5Ghz, no hyperthreading
- 4 146 GiB 15K RPM disks in RAID5 for physical boot and base VM storage
- 6 1Gbps ethernet interfaces (1 used for the NFS storage network)
Ubuntu 10.04.4 LTS 64-bit
- 2 GiB RAM
- 1 vCPU
- 32 GiB base system disk, VMDK on local storage, VMware paravirtual SCSI controller
- 1 E1000 ethernet for remote SSH connections
FreeBSD 8.2-STABLE 64bit
- 2 GiB RAM
- 1 vCPU
- 32 GiB base system disk, VMDK on local storage, LSI SAS controller
- 1 E1000 ethernet for remote SSH connections
Tests run software and settings
bonnie++
bonnie++ -u nobody -d <directory> -m bm--(f|l)-(os|tintri|tegile|nexsan) -n 10:102400:1024:1024Postmark
set size 10000 10000000set number 2000set transactions 2500I ran each benchmark multiple times and included the best (complete) results. I ran these tests at least 10 times per storage appliance.
Bonnie++ output
Flat text files with output from bonnie++ and Postmark
An interesting tidbit — creation of 2 TiB filesystems was surprising: FreeBSD took 21-22 seconds while Ubuntu took 5 minutes 40 seconds to do the same. I could recreate this on each system I tested against. There was variability of 1-2 seconds per run or per storage appliance, though not predictable. Another (surprisingly) big difference besides time to execute was that FreeBSD generated a very high spike in IOPS for about 3 seconds with only 50-60 MB/sec in throughput, while Ubuntu’s mkfs.ext4 command generated high sustained IOPS saturating the 1Gbps ethernet (over 100 MB/sec) for the duration.
I also found that there wasn’t a measurable difference between giving the test VMDK its own virtual SCSI controller vs using a single one. Running the benchmarks in parallel on 2-4 VMDKs only added latency once the 1Gbps pipe was full, which I did expect. Once I aggregated the data, throughput and iteration runs matched up with only latency really affected. I walked away with: VMware does a good job on their virtual SCSI adapters.