Vsan missing disks

DevilWAH

I am seeing this on or vmware infrastructure



Any ideas how I can get these disks to show healthy again? the server sees them just fine but I don't seem to be able to get them back in the vsan? Could I just remove the groups from the disk and re-add?

This happened after putting in to maintenance mode and a reboot for a 5.5u2 update.

Cheers

scott28tt

This question would get good visibility on the VMware community forum.

DevilWAH

scott28tt wrote: »

This question would get good visibility on the VMware community forum.

Cheers, I have indeed posted on a few forums but not that one so I sill do

Essendon

Not much idea about this, looks like the update's done something to them. The disks on the supported list? Is the disk group all flash? I hope someone like Rawlinson Rivera or someone of his ilk sees your thread on the VMTN forums.

DevilWAH

Essendon wrote: »

Not much idea about this, looks like the update's done something to them. The disks on the supported list? Is the disk group all flash? I hope someone like Rawlinson Rivera or someone of his ilk sees your thread on the VMTN forums.

ITs running on 5.5,

we have 1 X 400SSD and 3 x 1Tbyte spindle. The disks and controllers are on the supported list and the other 2 servers I have updated are identical specs and have been fine.

What would happen if i tried to remove the disks from the group and readd them? The vsan has dropped 3tbytes since the update so i am assuming it is these three disks missing and so not currently in use.

Essendon

Not sure mate, please go through VMware kb's and on cormachogan.com to find out. Dont want to give out erroneous information.

kj0

Do you use twitter? Hit Rawlinson Riviera or Cormac Hogan up. Those two should give a pretty decent answer, and if not, they will lead you to the right place.

https://twitter.com/PunchingClouds
https://twitter.com/CormacJHogan

DevilWAH

Just to say after a few sugestions (no one really seemed to know the exact issue, but I did come across and issue of some file locks) I ended up deleting the whole disk group and adding it back and this seems to have sorted out the issue.

Cheers for all the sugestions guys.

Essendon

I was going to suggest this too but I assumed it was a prod environment so I refrained. Was there anything on the disks before the update? If there was, did you lose any data?

Essendon

I realize this is fixed mate, but I ran this past Travis Wood (a VSAN guy and VCDX) today, but removing a disk group can result in data loss. So rebalance the VSAN cluster first (if running any VMs of course) and then delete disks or entire disk group.

DevilWAH

Essendon wrote: »

I was going to suggest this too but I assumed it was a prod environment so I refrained. Was there anything on the disks before the update? If there was, did you lose any data?

When I did the original maintence mode I did the full evac. And we are in fault tollerence mode 1. So I was pretty confident there was no data to lose. I also put the host back in to full maintence (again full evac) before I removed and readded. No data seems to have gone. There was no data to sync at the time and no desktops have suffered any issues.

DevilWAH

Essendon wrote: »

I realize this is fixed mate, but I ran this past Travis Wood (a VSAN guy and VCDX) today, but removing a disk group can result in data loss. So rebalance the VSAN cluster first (if running any VMs of course) and then delete disks or entire disk group.

How would you go about rebalancing the cluster? Is there a command or process to do this.

One thing I have noticed is that putting hosts in to maintence mode would take hours, since recreating the disk group is is taking 30miniutes Max. So seems there was a knock on issue.

As we are running level 1 redundancy I was aware of the potential dataloss but passing that any lose would be protected aginst.

Essendon

I looked at rebalancing a cluster and looks like it's a new feature in 6.0. I guess it doesnt apply in your case then.

Interesting observation about hosts taking longer than usual to go into maintenance. Why dont you log a ticket about it and get GSS to dig deeper?

DevilWAH

Essendon wrote: »

I looked at rebalancing a cluster and looks like it's a new feature in 6.0. I guess it doesnt apply in your case then.

Interesting observation about hosts taking longer than usual to go into maintenance. Why dont you log a ticket about it and get GSS to dig deeper?

Do you remember a few months back I posted about performance issues and a very bad consultant company who had installed it all? Well we dont have any support direct with vmware at the moment, and the new company that is looking at it although are better also don't have a clue about this issue. So sadly I don't have GSS support to look in to this for us . If I wait for our consultants it takes about 2 months to get any where!

Thank fully there are people such as your selves and others who are kind enough to help out. I am a network engineer so am learning as I go. I know the basics of VMware but its not really "my job" more of a side project as I would rather sort it out my self than wait and see the issues it casues our users.

First company did an install and the performance and stability was abysmal, after 7 months a second company was brought in to have a look and we have a 48 page document of misconfiguration and issues that don't meet firmware best practices.

solution to support 150 basic dedicated desktops on 6 hosts. I sware given 2 months and Google and I could have done a better job, I cant post some of the issues but even a juinour IT engineer would have spotted the mistakes that where made.

Oh the point of all the upgrades is to get all the hosts (6 of them) to the same 5.5 update and patch level before we upgrade to version 6. all the hosts where installed as 5.5 but some had no updates, some update 1 and some update 2. Who installs a new solution with all he hosts patched differently??

Essendon

For sure I remember it being a dog's breakfast! Positive side is - you get the experience by fixing the bad design. Anyways, let us know if we can help with anything else. With VSAN though, I dont have much experience, hence the lack of useful information in this thread..

Thinking about it - what do you think about VSAN? Has it given you what you expected? Any real performance gains or just a PITA?

DevilWAH

Essendon wrote: »

Thinking about it - what do you think about VSAN? Has it given you what you expected? Any real performance gains or just a PITA?

I love the idea, for small/medium business the theory behind it makes a lot of sense, cheap high preformace storage with resilience. However we had a lot of issues with the compatibility of hardware and took a lot to get it working. This is something a number of VMWare engineers from various companies have echoed . It can be a bit of a PITA, however now it is working it seems OK, not amazing but a disk test gives <5ms average latence and >2,000 IOPS (this is on the virtual vdi desk top). So for non disk intensive tasks like basic office PC's I think it could be a good solution. A number of engineers have said Vsan2 (vmware 6) is much much better. They seem to think the version in 5.5 was really a beta and performance, reliability and management has been improved. IF this is true (and I will let you know in a few weeks once we have upgraded) then hats of to VMware. I dont think it will scale very well in to the server estates as getting large SSD's and Spindle inside the hosts will end up getting costly enough that a stand alone SAN would make more sense.

But as I say for small/medium business it would diffidently consider it. Just if you don't have in house expertise get in a company that have a proven track record of deploying it

Essendon

Thanks for the input! I agree ROBOs and smaller shops would be able to make good use of it. You gotta take a look at PernixData's FVP product - I love it waaay more than the VSAN, lets you use spare RAM as the acceleration resource and accelerates both reads and writes with fault tolerant domains. I've got it running in our lab at work and with it we can make our legacy storage system sweat for longer. Best part - no redesign required and lets you move your VMs in and out of accelerated datastores seamlessly.

DevilWAH

Essendon wrote: »

Thanks for the input! I agree ROBOs and smaller shops would be able to make good use of it. You gotta take a look at PernixData's FVP product - I love it waaay more than the VSAN, lets you use spare RAM as the acceleration resource and accelerates both reads and writes with fault tolerant domains. I've got it running in our lab at work and with it we can make our legacy storage system sweat for longer. Best part - no redesign required and lets you move your VMs in and out of accelerated datastores seamlessly.

Cheers I will look that one up.

I have to say we have 6 hosts with 2 disk groups each, each group has 1 X 400gig SSD and 3 X 1tybyte spindle (6tybytes per host usable). we have 180 desktops running (max 60 concurrently logged in to with low usage users).

My 3 year old dell desktop still out preforms it for disk read writes, its usagable but as you can tell not really high preformance you would need for a Database.

dave330i

Essendon wrote: »

Thanks for the input! I agree ROBOs and smaller shops would be able to make good use of it. You gotta take a look at PernixData's FVP product - I love it waaay more than the VSAN, lets you use spare RAM as the acceleration resource and accelerates both reads and writes with fault tolerant domains. I've got it running in our lab at work and with it we can make our legacy storage system sweat for longer. Best part - no redesign required and lets you move your VMs in and out of accelerated datastores seamlessly.

Pernix is pretty cool tech. The 1 issue I have is write cache on volatile RAM. Not a big issue with VDI, but can be a problem with server workload. Plus, there's nothing stopping you from using Pernix and VSAN.

VSAN can turn DC storage into a commodity. That give you a lot of flexibility. It's allowing enterprise DC to become closer to a webscale DC.

I've heard that VSAN 6.0 is close to metro cluster capable.

Essendon

I agree somewhat with the hesitation of putting server workloads on volatile RAM, but your data is protected by virtue of being available on other nodes creating fault-tolerant domains for you.

As for using FVP and VSAN together, I dont think it's a good idea. They are both acceleration methods and both create their own redundant copies. Besides if you would want to use SSDs as the acceleration resource, either technology is going to want to have their own SSD. Secondly, costs go way up because of the need for more acceleration resources and licensing. Thirdly, you now have more moving parts, for lack of a better term, making it harder to track a problem down should one arise. Someone's done this article about using these two together here.