iSCSI traffic limited to 100-150 MB/s?

Hello all!
I am having trouble with the speed of my iSCSI traffic and I'm wondering if anybody can point me in the right direction? OK, so this is my lab:
Left-vSphere Host - vSphere 5.5
Asus Z87-PRO
Intel i7-4770T
32 GB DDR-3
500 GB SSD (Win 7) & 500 GB SSD (vSphere 5.5)
1 Onboard NIC & 2 HP NC364T Quad Port NICs - 9 NICs in total.
1 Distributed Switch (with Management, vMotion, iSCSI Port Binding & FT Logging)
Right-vSphere Host - vSphere 5.5
Same as above.
Server 1 - Domain Controller/iSCSI Target
Asus M4A785-M
AMD Phenom II x6 1075T
6 GB DDR-2
1 TB HDD (Server 2012) & 500 GB SSD (iSCSI Target)
5 NICs in total (1 onboard, 1 Quad Port - individual IPs for multiple iSCSI paths, no NIC teaming here)
Server 2 - vCenter Server 5.5/iSCSI Target
Same as Server 1.
The network switch is a HP v1910-48G - No LACP running.
Now to my problem...
I've got an SSD in the Left-vSphere host and when I create a VM using FreeNAS/Openfiler/Server 2012 - when I test this VM with CrystalDiskMark I get 450-500 MB/s running straight off the SSD. Happy days... no problem here.
But, when I create an iSCSI target on that VM and place another VM inside that newly created iSCSI Target, I only get about 100-150 MB/s going over the network. This is the same drive & datastore which runs at 450-500 MB/s.
I have set up iSCSI multipathing correctly (as shown in this YouTube link - using Round Robin) - and the performance tab shows nicely distributed data across the 9 NICs but just not running close to the top speed of the drive.
And, if I create an iSCSI Target on either the Domain Controller or on the vCenter Server the same thing happens. Why does my performance suck over the network so much? I am running Jumbo frames throughout, multiple GigE NICs and have multiple iSCSI paths. The switch says everything is running under 5% utilisation for everything connected while doing a Storage vMotion.
Any ideas on how to improve performance? What am I doing wrong that these guys are getting 380 MB/s?
Thanks in advance!
I am having trouble with the speed of my iSCSI traffic and I'm wondering if anybody can point me in the right direction? OK, so this is my lab:
Left-vSphere Host - vSphere 5.5
Asus Z87-PRO
Intel i7-4770T
32 GB DDR-3
500 GB SSD (Win 7) & 500 GB SSD (vSphere 5.5)
1 Onboard NIC & 2 HP NC364T Quad Port NICs - 9 NICs in total.
1 Distributed Switch (with Management, vMotion, iSCSI Port Binding & FT Logging)
Right-vSphere Host - vSphere 5.5
Same as above.
Server 1 - Domain Controller/iSCSI Target
Asus M4A785-M
AMD Phenom II x6 1075T
6 GB DDR-2
1 TB HDD (Server 2012) & 500 GB SSD (iSCSI Target)
5 NICs in total (1 onboard, 1 Quad Port - individual IPs for multiple iSCSI paths, no NIC teaming here)
Server 2 - vCenter Server 5.5/iSCSI Target
Same as Server 1.
The network switch is a HP v1910-48G - No LACP running.
Now to my problem...
I've got an SSD in the Left-vSphere host and when I create a VM using FreeNAS/Openfiler/Server 2012 - when I test this VM with CrystalDiskMark I get 450-500 MB/s running straight off the SSD. Happy days... no problem here.
But, when I create an iSCSI target on that VM and place another VM inside that newly created iSCSI Target, I only get about 100-150 MB/s going over the network. This is the same drive & datastore which runs at 450-500 MB/s.
I have set up iSCSI multipathing correctly (as shown in this YouTube link - using Round Robin) - and the performance tab shows nicely distributed data across the 9 NICs but just not running close to the top speed of the drive.
And, if I create an iSCSI Target on either the Domain Controller or on the vCenter Server the same thing happens. Why does my performance suck over the network so much? I am running Jumbo frames throughout, multiple GigE NICs and have multiple iSCSI paths. The switch says everything is running under 5% utilisation for everything connected while doing a Storage vMotion.
Any ideas on how to improve performance? What am I doing wrong that these guys are getting 380 MB/s?
Thanks in advance!
Comments
How many Devices, Path and Connected Targets do you see there ?http://blogs.vmware.com/tp/.a/6a00d8341c328153ef014e8ae08b19970d-pi
Then you say 'RR'. I can't check the YouTube video right now but how many vmkernel / nics are you using? For best performance you should dedicate a nic per vmkernel used solely by iscsi and use MPIO between those NICs.
So I did a benchmark off the "Server 2 iSCSI SSD" datastore and then another after I storage vMotioned to the Left-vSphere's Local SSD drive this is what I got:
First - ESXTOP
Second - Server-2's resources being used
Third - Completed storage benchmark on "Server 2 iSCSI SSD" datastore
Fourth - vSphere graph after benchmarks & storage vMotion over to local storage
Fifth - Completed storage benchmark on "Local Storage - Left" datastore
I played around with queue depth in Openfiler and still seemed to have the same problem as I got on Server 2012. This seems OS independent.
There is no Jumbo Frames setting on the HP V1910 because it does it out of the box. I wiped the lab and tested using regular and jumbo frames and got pretty similar results. But I have changed the vDS and the vmkernels to jumbo frames.
There is no flow control on the switch but it does support it. I am using 9 vmkernels & 9 NICs for iSCSI, vMotion and management traffic with one also doing FT logging. I also tried dedicating the vmkernels to iSCSI trafic only but it made no difference so I have left them doing everything now - there should be enough bandwidth in a lab situation.
I'm not sure why it doesn't saturate the links. Storage I/O is turned off on all the datastores by default. If I can't get to the bottom of it I will have to live with it! It only takes a few minutes to do a storage vMotion I was just wondering why there is a huge difference in results off the same make and model SSD drives.
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
Sixth - There are some "Active" paths instead of "Active I/O" that I can't get rid of
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
I only top out at 275 MB/s because Server 1 & Server 2 have SATA 3 Gb ports so I am going to get new SATA 6 Gb controllers for those servers to speed that up and hopefully hit 500 MB/s. I now get a datastore usage alarm which I didn't get before.
I still get poor results when doing a storage benchmark using CrystalDiskMark but if anybody has any ideas I would love to hear it! Storage vMotion performance matters to me more than CrystalDiskMark.
Thanks all!
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
Err good point
Gigabit is theoretical approx. 125MB/s ... Now the question is, what does the switch even support on the backend, because 9 links aren't equal 9x125MB/s ....
I re-read your first post, and have some questions:
You have 9 nics on each of vsphere hosts -how many dedicated to iSCSI network/ Vlan/subnet?
6 Nics on the DC/iSCSI targets, how many of these serve as target ports for the iSCSI ?
On the MS iSCSI Target side: did you allow all iSCSI IPs from your vSphere hosts to connect to the targets on Win 2012? it should look similar to http://blogs.technet.com/blogfiles/migreene/WindowsLiveWriter/ConfiguringiSCSIMPIOonWindowsServer2008R_9CF9/image_2.png
How many iSCSI vmkernels did you set up ? Did you set all the other vmnics to unused for each of the iSCSI uplinks on the vmware hosts?
See the Storage Traffic piece here:
Efficient Virtual Networking Designs for vSphere Home Lab Servers
Btw, there something definitely wrong with your MPIO, you have 4 Luns, and 126 connections, which makes it 31.5 path per LUN. How many iSCSI ports dedicated on the Vmware side and how many iSCSI target ports do you have on the WIn 2012?
see block post at Vmware: vSphere 5.0 Storage Features Part 12 – iSCSI Multipathing Enhancements | VMware vSphere Blog - VMware Blogs
Last thing: are you using e1000 or VMXNET3 driver inside that iSCSI Target VM?
Throughput - up to 77.4 Mpps (64-byte packets)
Routing/Switching capacity - 104 Gb/s
I am using HP NC364T NICs which are a re-branded Intel PRO/1000 PT NIC so VMware picks them up straight away. I had to use a different driver for the onboard motherboard NIC from this thread on the VMware forum. And used ESXi-Customiser to add it in to a custom ISO.
All 9 NICs are dedicated to iSCSI/Management/vMotion/FT Logging. There are 9 vmkernels, with each vmkernel linked to a single vDS portgroup which has only 1 active adapter, the rest are unused. Each iSCSI portgroup is added to the software iSCSI initiator for multipathing. VMs have their own portgroup with NIC teaming enabled.
I have 4 luns - ReadyNAS (2 NICS), Server 1 (5 NICs), Server 2 (5 NICs) and the Server 2012 Test iSCSI Target VM which has 3 VMXNET3 NICs (but I also tested it with only 1 with no difference in the result). Each iSCSI target uses all available NICs.
Let me know if you want any more screenshots.
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
Before going back to basics, It's given that you will not get the same MB/s over the wire as you get locally, specially given that you triple nesting, if my count is correct.
I am assuming that its all running in the same subnet?
Now the basics, pick an iSCSI target, lets say Server 1 with 5 nics:
ping each of those nics from your left-vsphere host. are you getting all the replies?
Did you add the IPs or IQNs to the "iSCSI Initiators", can you add the iSCSI IPs of your right and left vsphere hosts there?
Edit - I forgot to mention that it is on a single subnet and there are no VLANs setup... yet. 18 IP addresses added - doing a storage vMotion at present with all firewalls turned off also gives the same result as the attachments in post #8. 130MB/s then goes up to 230MB/s about 5 minutes after starting.
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
You might have already mentioned it, but, how your dvPortgroup set up, do you have seperate dvPG per iSCSI Vmkernel or did you **** everything into one single dvPG ? Could you create 4 PGs and assign one vmkernel per PG, and then ensure that only 1 uplink is active per PG, then go back to Software iSCSI and add these additional vmks.
could you post a video of the setup, from point a to b, or at least a screen of software iscsi bindings, networking, dvswitch/pg and relevant pieces.
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
Let me know if you need any other screenshots - I think everything you need is listed in these posts...
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
1) check if Jumbo frames working as expected
VMware KB: Testing VMkernel network connectivity with the vmkping command
2) Please ping from your esxi hosts each of these IPs and see if its available.
issue vmkping -I to 3 iSCSI target IPs you have in your static Discovery tab from 2 iscsi vmkernels (vmkX) (i know it shows 45 path in place, but just confirm it)
vmkping -I vmk0 192.168.0.1
vmkping -I vmk0 192.168.0.2
vmkping -I vmk0 192.168.0.6
vmkping -I vmk0 192.168.0.7
etc..
vmkping -I vmk1 192.168.0.1
vmkping -I vmk1 192.168.0.2
vmkping -I vmk1 192.168.0.6
vmkping -I vmk1 192.168.0.7
etc...
head over to page 13 and verify multipath configuration
http://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdf
3) in case if the above does not work, move 2 vmks to regular vswitches/Vlans; confirm multipathing again.
4) just accept it
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
Blog: https://readysetvirtual.wordpress.com
I took 4 cables out and it kept up pretty much, it's been up and down all evening, below 100MB/s then over 150MB/s for no real reason at all and here it hits 175MB/s. I need to change the SATA ports and see what happens then - the SSD is running at half it's potential. This link shows somebody getting 800MB/s from a FreeNAS install so I don't believe iSCSI is limited to 125MB/s otherwise why would you multipath?
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
I just did some tests in my lab -had a couple of hours of free time on my hand.
Your post #7 indicates Active I/O , did you change the Path Selection to Rounds Robin on both hosts ? Can you verify it by looking at the Datastore details on for each datastore on each of the hosts?
One more detail,check the traffic flow on iSCSI target (windows server side, on both source and destination) NICs traffic, while performing sVMotion. That will confirm if Multipathing works or not.
I only ordered the 6 Gb SATA controllers on Friday - I should have them by Tuesday/Wednesday though (I order from the UK so it takes a few days).
The second screenshot in post #6 shows the resources being used by Server 2's NICs during a storage vMotion so it looks like that is working OK - I have run CrystalDiskMark off Server 2's SSD and I am only getting 220MB/s there so only half the performance I get on either Left or Right vSphere hosts with 6 Gb SATA controllers.
I was thinking of striping the SSD's on Server 2 when I get the new SATA controllers to test out the full performance (5x125=625MB/s theoretically). All datastores are round robin and I have changed the IOPS to 1 as sugggested by a few places on the net... some other places suggest changing IOPS to 8800 bytes - I have also tried that but I can't say for sure that it's doing anything to increase the speed until I have the SATA controllers in my hands.
The commands I ran on each host are listed here:
ls /vmfs/devices/disks | grep naa.600
esxcli storage nmp device list | grep "Policy Device Config"
esxcli storage nmp psp roundrobin deviceconfig set -d "naa.60003ff44dc75adcae7329479b8b1a60" -I "1" -t "bytes" -B "8800"
esxcli storage nmp psp roundrobin deviceconfig set -d "naa.60003ff44dc75adcaf77c72a4dab8e8f" -I "1" -t "bytes" -B "8800"
esxcli storage nmp psp roundrobin deviceconfig set -d "naa.60003ff44dc75adcae9f19e280331a0e" -I "1" -t "bytes" -B "8800"
esxcli storage nmp psp roundrobin deviceconfig set -d "naa.60014052e41c0a0010ed001000000000" -I "1" -t "bytes" -B "8800"
esxcli storage nmp psp roundrobin deviceconfig get -d "naa.60003ff44dc75adcae7329479b8b1a60"
esxcli storage nmp psp roundrobin deviceconfig get -d "naa.60003ff44dc75adcaf77c72a4dab8e8f"
esxcli storage nmp psp roundrobin deviceconfig get -d "naa.60003ff44dc75adcae9f19e280331a0e"
esxcli storage nmp psp roundrobin deviceconfig get -d "naa.60014052e41c0a0010ed001000000000"
Thanks for your PM,
Asif
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger
Hey all, just an update..
I have updated the SATA controllers to 6 Gb/s but there is no real difference in transfer speed within vSphere (tested locally on Server 2 it is faster - as shown in the picture above). There is still a huge difference in the transfer speeds when I transfer from the Left Local SSD to Server 2 iSCSI SSD - and then from Server 2 iSCSI SSD to Server 1 iSCSI SSD as shown in the graph above.
You would think a transfer from a Local SSD to an iSCSI datastore would be faster but seemingly not here. And the strange thing is even though the transfer from iSCSI target to iSCSI target is faster, it somehow takes longer to transfer than from Local SSD to iSCSI Target which is slower? It makes no sense at all. I've had a look through the logs and I haven't found anything.
Many thanks JBrown for your help - I think I have learned a lot about iSCSI and how it works with vSphere but there is a lot for me to learn still I guess... I think I will have to live with the whole thing as it is until I get to know vSphere better.
The mind is the limit. As long as the mind can envision the fact that you can do something, you can do it, as long as you really believe 100 percent. - Arnold Schwarzenegger