new ESXi/Netapp rollout. What am I missing?

bermovick

Hey guys. We're rolling out a new stack of ESXi servers & Netapps here and our server admin doesn't know how to perform the initial setups of either, so has asked me to help (read: do it for) them.

I've gotten them to the point of connecting them over iSCSI, but the ESXi boxes won't 'see' the Netapps unless I specifically tell each esxi box the IP address of each netapp. From what I understand I shouldn't have to do that but I don't know what step(s) I am missing, and am hoping the gurus in this section of the forum can tell.

The ESXi side has 2 VMNICs added to a SAN Vswitch, along with 2 VMKernel ports. Each VMKernel port has an address in the 10.0.0.0/24 subnet and only has 1 VMNIC active. A storage adapter has been created (vmhba34), with the 2 VMkernel ports added and its iqn noted.

The netapp side has 3 intefaces trunked into a VIF, and the VIF has an ip address in the 10.0.0.0/24 subnet (as an aside, the 10.0.0.0/24 SAN subnet has no default-gateway and is air-gapped from the rest of the network). iSCSI has been enabled, and an aggregate, volume and lun has been created. An Initiator Group has been created with each of the esxi iqns listed as initiators and the lun has been told to use that initiator group.

Am I missing something? I haven't found any other setting that seems to make sense, but if I rescan on the esxi, it completes within a second or two, which makes me wonder if some protocol for automatic discovery isn't running or something, but I'm at a loss.

dave330i

Right click vmhba34 > properties > Dynamic discovery > enter the Netapp VIF > rescan.

You'll need to do it for each ESXi host.

bermovick

So you DO have to manually list each address on the ESXi side then? That'll teach me to listen to the server admin who said it wasn't required!

dave330i

bermovick wrote: »

So you DO have to manually list each address on the ESXi side then? That'll teach me to listen to the server admin who said it wasn't required!

Unless you set up auto deploy or host profile.

bermovick

Curiously, after correcting a mismatch in trunking settings between the NETAPPs, brocades, and ESXi servers I find the ESXi side does automatically detect the NETapps!

Now to figure out why jumbo frames breaks connectivity. This project is a large box of headaches.

Architect192

bermovick wrote: »

Curiously, after correcting a mismatch in trunking settings between the NETAPPs, brocades, and ESXi servers I find the ESXi side does automatically detect the NETapps!

Now to figure out why jumbo frames breaks connectivity. This project is a large box of headaches.

Explain please, because I've never seen (and I've been VCP since 2.x) an ESXi host autodetect a storage out of the blue. You have to specify the target IP... Once you do that, it WILL detect the volumes presented to the host but finding storage by itself??

jibbajabba

Unless you got HBAs preconfigured with an IP range or DHCP and the whole subnet is allowed. No offence, but I hope this is not production yet. It sounds thrown together so make sure you got your storage setup as per NetApp and VMware best practises.

Architect192

jibbajabba wrote: »

Unless you got HBAs preconfigured with an IP range or DHCP and the whole subnet is allowed. No offence, but I hope this is not production yet. It sounds thrown together so make sure you got your storage setup as per NetApp and VMware best practises.

Then again, it's not autodetecting, it has been specified I agree with you that this sounds like an awful deployment, duct tape and all...

Deathmage

I concur with the OP post it appears on the right track, as for the jumbo frames make sure it's enable on all aspects of the fabric; vSwitch, vNic, switching fabric and lastly if it's windows based the windows based njc.

As for the IP address, I do concur it's needed to an extent, you can do a scan on a HBA and if it's connected to the svMotion vlan it should auto discover the luns.

Let me think about this for a bit. I myself just did a NetApp deployment for work but you seem your on the right track just don't go live until you know it's working correctly.

bermovick

Yeah, I missed 2 things regarding jumbo frames: 1) Enabling jumbo frames on the switches connecting the devices together, and 2) apparently you have to configure the jumbo MTU size on the vswitch itself, not only the vmkernel ports. Once those were done pings started working and all the latency problems we were seeing went away.

I'm grabbing screenshots regarding the ESXi servers automatically seeing the netapp luns when scanning without having the addresses added under the dynamic discovery tab. I dunno what to tell you though - perhaps it remembers from having them added before? If it helps, they're in the same layer 2 broadcast domain so perhaps that helps?

joelsfood

If you have the Netapp plugin installed in your Vcenter, I believe it will be able to find the SAN and storage without adding the discovery IP on the iscsi adapter (I'm not sure, as we run VSC, but we only present fibre channel LUNs in our Flexpod setup). That would be my guess though. The Vsphere hosts know about the arrays through Netapp VSC, so they don't require the iSCSI standard discovery address to be input.

bermovick

Well, I take it back. Today scans detects nothing, despite several test luns being created on the netapps.

I really don't get it. Yesterday it was seeing luns on netapp00 even though netapp00's address wasn't in dynamic discovery. Today, nothing.

I'm not going to test anymore to figure it out. I'll just add all the addresses and call it done. If the person who actually has their VCP complains about it not being done correctly I'll point out that perhaps the person with the VCP certification should have done the work

Architect192

bermovick wrote: »

Well, I take it back. Today scans detects nothing, despite several test luns being created on the netapps.

I really don't get it. Yesterday it was seeing luns on netapp00 even though netapp00's address wasn't in dynamic discovery. Today, nothing.

I'm not going to test anymore to figure it out. I'll just add all the addresses and call it done. If the person who actually has their VCP complains about it not being done correctly I'll point out that perhaps the person with the VCP certification should have done the work

+1000

Deathmage

bermovick wrote: »

Yeah, I missed 2 things regarding jumbo frames: 1) Enabling jumbo frames on the switches connecting the devices together, and 2) apparently you have to configure the jumbo MTU size on the vswitch itself, not only the vmkernel ports. Once those were done pings started working and all the latency problems we were seeing went away.

I'm grabbing screenshots regarding the ESXi servers automatically seeing the netapp luns when scanning without having the addresses added under the dynamic discovery tab. I dunno what to tell you though - perhaps it remembers from having them added before? If it helps, they're in the same layer 2 broadcast domain so perhaps that helps?

There is Atleast 3 to 4 places to enable Jumbo Frames depending upon OS.



Make sure the netapp's controllers are in the same subnet and vlan as the svmotion (iSCSI) vSwitch. I know it sounds silly but hook up a laptop to the vlan and ping them all might be as silly as a cable. It's happened to me before... also if the port has port security it could have shutdown the port.

As for a scolding from a VCP, won't come from me. We all start someplace, I only ask for a open ear to learning. I'm the kind of person to help others understand not just have a mega ego and be rude.

QHalo

Ugh, don't even bother with Jumbo Frames...such a waste of time for so little gain.

As for vSphere and NetApp, you should look over this depending on which version of OnTAP - TR-4068:VMware vSphere 5 on NetApp Clustered Data ONTAP Best Practices | NetApp Storage Best Practices for VMware vSphere | TR-3749

There may be newer documents but those are still just as relevant.