iSCSI traffic limited to 100-150 MB/s?

Asif Dasl

Hello all!

I am having trouble with the speed of my iSCSI traffic and I'm wondering if anybody can point me in the right direction? OK, so this is my lab:

Left-vSphere Host - vSphere 5.5
Asus Z87-PRO
Intel i7-4770T
32 GB DDR-3
500 GB SSD (Win 7) & 500 GB SSD (vSphere 5.5)
1 Onboard NIC & 2 HP NC364T Quad Port NICs - 9 NICs in total.
1 Distributed Switch (with Management, vMotion, iSCSI Port Binding & FT Logging)

Right-vSphere Host - vSphere 5.5
Same as above.

Server 1 - Domain Controller/iSCSI Target
Asus M4A785-M
AMD Phenom II x6 1075T
6 GB DDR-2
1 TB HDD (Server 2012) & 500 GB SSD (iSCSI Target)
5 NICs in total (1 onboard, 1 Quad Port - individual IPs for multiple iSCSI paths, no NIC teaming here)

Server 2 - vCenter Server 5.5/iSCSI Target
Same as Server 1.

The network switch is a HP v1910-48G - No LACP running.


Now to my problem...

I've got an SSD in the Left-vSphere host and when I create a VM using FreeNAS/Openfiler/Server 2012 - when I test this VM with CrystalDiskMark I get 450-500 MB/s running straight off the SSD. Happy days... no problem here.

But, when I create an iSCSI target on that VM and place another VM inside that newly created iSCSI Target, I only get about 100-150 MB/s going over the network. This is the same drive & datastore which runs at 450-500 MB/s.

I have set up iSCSI multipathing correctly (as shown in this YouTube link - using Round Robin) - and the performance tab shows nicely distributed data across the 9 NICs but just not running close to the top speed of the drive.

And, if I create an iSCSI Target on either the Domain Controller or on the vCenter Server the same thing happens. Why does my performance suck over the network so much? I am running Jumbo frames throughout, multiple GigE NICs and have multiple iSCSI paths. The switch says everything is running under 5% utilisation for everything connected while doing a Storage vMotion.


Any ideas on how to improve performance? What am I doing wrong that these guys are getting 380 MB/s?

Thanks in advance!

tstrip007

Nice lab. At first I was thinking round-robin and jumo frames but you've done that. I wonder if adjusting the LUN queue depth will help. Just thinking out loud...

QHalo

Doesn't sound like it's going down all the paths. What does ESXTOP show on the iSCSI vmkernels?

JBrown

Do you see any traffic flow on the iSCSI NICs on Server 1 - Domain Controller/iSCSI Target ?

How many Devices, Path and Connected Targets do you see there ?http://blogs.vmware.com/tp/.a/6a00d8341c328153ef014e8ae08b19970d-pi

jibbajabba

Is your network kit using jumbo frames too? You mention 'throughout' but I'd check again if all network kit uses JFs, all vmkernel and so on. Also does the switch support flow control ?

Then you say 'RR'. I can't check the YouTube video right now but how many vmkernel / nics are you using? For best performance you should dedicate a nic per vmkernel used solely by iscsi and use MPIO between those NICs.

Asif Dasl

Thanks for the replies everyone, I probably should have attached screenshots the first time so I have loads of them now! I know that this is only in a lab but in the real world I'd be fighting this out with a storage vendor but I am intrigued as to what is causing it - maybe misconfiguration on my part. I don't work with vSphere yet.

So I did a benchmark off the "Server 2 iSCSI SSD" datastore and then another after I storage vMotioned to the Left-vSphere's Local SSD drive this is what I got:

First - ESXTOP


Second - Server-2's resources being used


Third - Completed storage benchmark on "Server 2 iSCSI SSD" datastore


Fourth - vSphere graph after benchmarks & storage vMotion over to local storage


Fifth - Completed storage benchmark on "Local Storage - Left" datastore


I played around with queue depth in Openfiler and still seemed to have the same problem as I got on Server 2012. This seems OS independent.

There is no Jumbo Frames setting on the HP V1910 because it does it out of the box. I wiped the lab and tested using regular and jumbo frames and got pretty similar results. But I have changed the vDS and the vmkernels to jumbo frames.

There is no flow control on the switch but it does support it. I am using 9 vmkernels & 9 NICs for iSCSI, vMotion and management traffic with one also doing FT logging. I also tried dedicating the vmkernels to iSCSI trafic only but it made no difference so I have left them doing everything now - there should be enough bandwidth in a lab situation.

I'm not sure why it doesn't saturate the links. Storage I/O is turned off on all the datastores by default. If I can't get to the bottom of it I will have to live with it! It only takes a few minutes to do a storage vMotion I was just wondering why there is a huge difference in results off the same make and model SSD drives.

Asif Dasl

Just another screenshot as I was limited to 5 in the previous post...

Sixth - There are some "Active" paths instead of "Active I/O" that I can't get rid of

Asif Dasl

OK folks, looks like this one is resolved somewhat. I turned on Storage I/O Control on to all of the datastores (30ms each) and now I'm getting 125-275 MB/s when doing a Storage vMotion. It seems to start off slow then goes really fast which looks strange. Anybody know why it might do that?



I only top out at 275 MB/s because Server 1 & Server 2 have SATA 3 Gb ports so I am going to get new SATA 6 Gb controllers for those servers to speed that up and hopefully hit 500 MB/s. I now get a datastore usage alarm which I didn't get before.


I still get poor results when doing a storage benchmark using CrystalDiskMark but if anybody has any ideas I would love to hear it! Storage vMotion performance matters to me more than CrystalDiskMark.


Thanks all!

deth1k

So hang on,the switch is gigabit and what do you expect from your network throughput? :O

jibbajabba

deth1k wrote: »

So hang on,the switch is gigabit and what do you expect from your network throughput? :O

Err good point

Gigabit is theoretical approx. 125MB/s ... Now the question is, what does the switch even support on the backend, because 9 links aren't equal 9x125MB/s ....

deth1k

have a look at what the backplane of that switch is and see if it's oversubscribed however if you are getting 125MB/s that's what you going to get on a Gig anyways

jdancer

Gonna ask a Captain Obvious question here: You using Intel NICs? VMWare really *LOVES* Intel NICs. Got a client fighting such an issue here using a non-Intel NICs. Once they brought Intel NICs, their problems went away.

deth1k

it doesn't mind Broadcom either

JBrown

Asif,
I re-read your first post, and have some questions:

You have 9 nics on each of vsphere hosts -how many dedicated to iSCSI network/ Vlan/subnet?
6 Nics on the DC/iSCSI targets, how many of these serve as target ports for the iSCSI ?
On the MS iSCSI Target side: did you allow all iSCSI IPs from your vSphere hosts to connect to the targets on Win 2012? it should look similar to http://blogs.technet.com/blogfiles/migreene/WindowsLiveWriter/ConfiguringiSCSIMPIOonWindowsServer2008R_9CF9/image_2.png


How many iSCSI vmkernels did you set up ? Did you set all the other vmnics to unused for each of the iSCSI uplinks on the vmware hosts?

See the Storage Traffic piece here:
Efficient Virtual Networking Designs for vSphere Home Lab Servers

Btw, there something definitely wrong with your MPIO, you have 4 Luns, and 126 connections, which makes it 31.5 path per LUN. How many iSCSI ports dedicated on the Vmware side and how many iSCSI target ports do you have on the WIn 2012?

see block post at Vmware: vSphere 5.0 Storage Features Part 12 – iSCSI Multipathing Enhancements | VMware vSphere Blog - VMware Blogs

Last thing: are you using e1000 or VMXNET3 driver inside that iSCSI Target VM?

Asif Dasl wrote: »

Just another screenshot as I was limited to 5 in the previous post...

Sixth - There are some "Active" paths instead of "Active I/O" that I can't get rid of

Asif Dasl

Hi all, on the switch I am using 31 ports (9+9+5+5+2+1), the backplane of the switch does:

Throughput - up to 77.4 Mpps (64-byte packets)
Routing/Switching capacity - 104 Gb/s

I am using HP NC364T NICs which are a re-branded Intel PRO/1000 PT NIC so VMware picks them up straight away. I had to use a different driver for the onboard motherboard NIC from this thread on the VMware forum. And used ESXi-Customiser to add it in to a custom ISO.

All 9 NICs are dedicated to iSCSI/Management/vMotion/FT Logging. There are 9 vmkernels, with each vmkernel linked to a single vDS portgroup which has only 1 active adapter, the rest are unused. Each iSCSI portgroup is added to the software iSCSI initiator for multipathing. VMs have their own portgroup with NIC teaming enabled.

I have 4 luns - ReadyNAS (2 NICS), Server 1 (5 NICs), Server 2 (5 NICs) and the Server 2012 Test iSCSI Target VM which has 3 VMXNET3 NICs (but I also tested it with only 1 with no difference in the result). Each iSCSI target uses all available NICs.

Let me know if you want any more screenshots.

JBrown

Asif,

Before going back to basics, It's given that you will not get the same MB/s over the wire as you get locally, specially given that you triple nesting, if my count is correct.

I am assuming that its all running in the same subnet?
Now the basics, pick an iSCSI target, lets say Server 1 with 5 nics:
ping each of those nics from your left-vsphere host. are you getting all the replies?

Did you add the IPs or IQNs to the "iSCSI Initiators", can you add the iSCSI IPs of your right and left vsphere hosts there?

Asif Dasl

Oh I understand that I would not get 1:1 performance nesting inside another VM but not 20% of the total performance of the drive. I entered the IQN on Server 2012. You first have to set it up in the vSphere initiator, rescan the host and then it shows up automatically in Server 2012. But I will add all 18 IP's to see where you are going. I can ping Server 1 & 2 from the left & right vSphere hosts.

Edit - I forgot to mention that it is on a single subnet and there are no VLANs setup... yet. 18 IP addresses added - doing a storage vMotion at present with all firewalls turned off also gives the same result as the attachments in post #8. 130MB/s then goes up to 230MB/s about 5 minutes after starting.

JBrown

it sounds like traffic goes additional port only after first and/or 2nd port is saturated. yet esxtop in post 6 clearly shows traffic transmitting almost evenly over every port (vmnic0-

You might have already mentioned it, but, how your dvPortgroup set up, do you have seperate dvPG per iSCSI Vmkernel or did you **** everything into one single dvPG ? Could you create 4 PGs and assign one vmkernel per PG, and then ensure that only 1 uplink is active per PG, then go back to Software iSCSI and add these additional vmks.

could you post a video of the setup, from point a to b, or at least a screen of software iscsi bindings, networking, dvswitch/pg and relevant pieces.

Asif Dasl wrote: »

snip snip

Edit - I forgot to mention that it is on a single subnet and there are no VLANs setup... yet. 18 IP addresses added - doing a storage vMotion at present with all firewalls turned off also gives the same result as the attachments in post #8. 130MB/s then goes up to 230MB/s about 5 minutes after starting.

Asif Dasl

Asif Dasl

Let me know if you need any other screenshots - I think everything you need is listed in these posts...

Asif Dasl

I just noticed a bug in the vSphere Web Client for those testing on 5.5 - the web client graph is wrong...

JBrown

last couple of attempts and i will leave you alone


1) check if Jumbo frames working as expected
VMware KB: Testing VMkernel network connectivity with the vmkping command

2) Please ping from your esxi hosts each of these IPs and see if its available.
issue vmkping -I to 3 iSCSI target IPs you have in your static Discovery tab from 2 iscsi vmkernels (vmkX) (i know it shows 45 path in place, but just confirm it)

vmkping -I vmk0 192.168.0.1
vmkping -I vmk0 192.168.0.2
vmkping -I vmk0 192.168.0.6
vmkping -I vmk0 192.168.0.7
etc..

vmkping -I vmk1 192.168.0.1
vmkping -I vmk1 192.168.0.2
vmkping -I vmk1 192.168.0.6
vmkping -I vmk1 192.168.0.7
etc...

head over to page 13 and verify multipath configuration
http://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdf

3) in case if the above does not work, move 2 vmks to regular vswitches/Vlans; confirm multipathing again.

4) just accept it

Asif Dasl

Much appreciated JBrown... I was able to ping everything and all paths are active which leaves me with point #4 - just accept it... NEVER! well, maybe yes! haha.. I will upgrade the SATA ports to 6 Gb and hope to see some difference after that and will report back either way... Thanks again!

kj0

What happens if you take out a couple of Nics, is there a decrease? IF 1Gb is limited to 125MB/s then you shouldn't see a decrease using only 3 Nics, If the speed stays the same, then I would suspect that it is either going down 1 path, or even if it is using all, then something else is failing.

Asif Dasl

I took 4 cables out and it kept up pretty much, it's been up and down all evening, below 100MB/s then over 150MB/s for no real reason at all and here it hits 175MB/s. I need to change the SATA ports and see what happens then - the SSD is running at half it's potential. This link shows somebody getting 800MB/s from a FreeNAS install so I don't believe iSCSI is limited to 125MB/s otherwise why would you multipath?

JBrown

Have you resolved this yet?
I just did some tests in my lab -had a couple of hours of free time on my hand.
Your post #7 indicates Active I/O , did you change the Path Selection to Rounds Robin on both hosts ? Can you verify it by looking at the Datastore details on for each datastore on each of the hosts?
One more detail,check the traffic flow on iSCSI target (windows server side, on both source and destination) NICs traffic, while performing sVMotion. That will confirm if Multipathing works or not.

Asif Dasl

Hey JBrown,

I only ordered the 6 Gb SATA controllers on Friday - I should have them by Tuesday/Wednesday though (I order from the UK so it takes a few days).

The second screenshot in post #6 shows the resources being used by Server 2's NICs during a storage vMotion so it looks like that is working OK - I have run CrystalDiskMark off Server 2's SSD and I am only getting 220MB/s there so only half the performance I get on either Left or Right vSphere hosts with 6 Gb SATA controllers.

I was thinking of striping the SSD's on Server 2 when I get the new SATA controllers to test out the full performance (5x125=625MB/s theoretically). All datastores are round robin and I have changed the IOPS to 1 as sugggested by a few places on the net... some other places suggest changing IOPS to 8800 bytes - I have also tried that but I can't say for sure that it's doing anything to increase the speed until I have the SATA controllers in my hands.

The commands I ran on each host are listed here:

ls /vmfs/devices/disks | grep naa.600
esxcli storage nmp device list | grep "Policy Device Config"

esxcli storage nmp psp roundrobin deviceconfig set -d "naa.60003ff44dc75adcae7329479b8b1a60" -I "1" -t "bytes" -B "8800"
esxcli storage nmp psp roundrobin deviceconfig set -d "naa.60003ff44dc75adcaf77c72a4dab8e8f" -I "1" -t "bytes" -B "8800"
esxcli storage nmp psp roundrobin deviceconfig set -d "naa.60003ff44dc75adcae9f19e280331a0e" -I "1" -t "bytes" -B "8800"
esxcli storage nmp psp roundrobin deviceconfig set -d "naa.60014052e41c0a0010ed001000000000" -I "1" -t "bytes" -B "8800"

esxcli storage nmp psp roundrobin deviceconfig get -d "naa.60003ff44dc75adcae7329479b8b1a60"
esxcli storage nmp psp roundrobin deviceconfig get -d "naa.60003ff44dc75adcaf77c72a4dab8e8f"
esxcli storage nmp psp roundrobin deviceconfig get -d "naa.60003ff44dc75adcae9f19e280331a0e"
esxcli storage nmp psp roundrobin deviceconfig get -d "naa.60014052e41c0a0010ed001000000000"

Thanks for your PM,
Asif

Asif Dasl

Hey all, just an update..

I have updated the SATA controllers to 6 Gb/s but there is no real difference in transfer speed within vSphere (tested locally on Server 2 it is faster - as shown in the picture above). There is still a huge difference in the transfer speeds when I transfer from the Left Local SSD to Server 2 iSCSI SSD - and then from Server 2 iSCSI SSD to Server 1 iSCSI SSD as shown in the graph above.

You would think a transfer from a Local SSD to an iSCSI datastore would be faster but seemingly not here. And the strange thing is even though the transfer from iSCSI target to iSCSI target is faster, it somehow takes longer to transfer than from Local SSD to iSCSI Target which is slower? It makes no sense at all. I've had a look through the logs and I haven't found anything.

Many thanks JBrown for your help - I think I have learned a lot about iSCSI and how it works with vSphere but there is a lot for me to learn still I guess... I think I will have to live with the whole thing as it is until I get to know vSphere better.