ESXi 6.0 and CommVault Backups

Lexluethar

For people on ESXi 6.0 and using CommVault snaps:



Issue
VMware has identified an issue in a VMware CBT API call that can affect backups for virtual machines hosted on ESXi 6.0. The API call can return incorrect changed sectors, causing incomplete incremental backups.
The VMware ESXi engineering team is working on a fix which is slated for ESXi 6.0 Update 2. There is no current ETA on its availability. In the interim, VMware is currently suggesting to downgrade to ESX 5.5 and VM hardware version 10 as a workaround. Customers may also opt to shutdown VMs before doing an incremental backup or run full backups.
For more information, see the VMware KB article Backups with Changed Block Tracking can return incorrect changed sectors in ESXi 6.0 (2136854) and VDDK 6 release notes.
[h=3]For additional information, see the Commvault Books Online article VMware CBT Issue[/h]Additionally, there is no detection criteria for which VMs have been affected.

iBrokeIT

Wow....

Lexluethar

But you can just shut the VM down to back it up - that's acceptable right? Right.......? I laughed at that really hard.

jibbajabba

Doesn't just a affect CommVault. Thought I am losing it with Veeam in my lab when I run into this. Funny enough, still no update on my ticket so I got the fact that there is no solution here quicker than from GSS.

joelsfood

Yeah, thread about it at Veeam basically boils down to "Disable CBT and wait for VMware to release an update"

kj0

Pretty sure this was an issue in 5.1 or 5.5 as well.

We've got bigger issues with Veeam and vSphere at the moment where vSphere isn't letting Veeam know it has started to create a snapshot, so Veeam times out and requests another, doubling up the snapshots and removing only 1 out of the two when finished.

DPG

No problems here with vCenter 6U1, Esxi 6.0.0b hosts and the latest version of Veeam.

blargoe

What the....????

I'm looking to go to vCenter 6 in the near future, but I think I will be holding off on the ESXi upgrade for a while.

kj0

Couple of our other backup vendors have sent word in about this also.

OctalDump

iBrokeIT wrote: »

Wow....

My reaction, too. The only upside to this is that you can at least blame it on the vendor... well you can as long as there isn't some idiot wanting to push Hyper-V who doesn't understand virtualisation beyond that 30 minute presentation at "CTO world 2013 - Are Clouds scary? Why not build your own!"

jibbajabba

In the real world the customer doesn't care. I work as a senior consultant and if I design and implement a solution and hit bugs, they don't care because SOWs etc. Rarely consider bugs and the customer signed / agreed to a result and if you hit bugs they likely still blame you for not knowing.

blargoe

I expect Update 2 will be rushed through, followed by 2a, 2b, 2c, ... to address that other things that rushing out a fix will break.

Lexluethar

Jibba I agree -while as an Engineer i might feel better in the fact that I didn't directly cause the issue, my user base will care less. They will look at it as an IT issue and the argument can always be said that you shouldn't have upgraded that quickly.

Personally I will always wait until at least U1 if not U2 for any ESXi / vCenter upgrade. We migrated from ESXi 5.1 to 5.5 GA (we were on vCenter 5.5 GA at the time) and got bit twice with major bugs. Unfortunately for us we were at a point in time where we HAD to upgrade or make major infrastructure changes because of the 2tb limit on vmdk files. We had three production volumes at that capacity that either had to be redone as RDMs or we had to migrate to 5.5.

Either way, it's up to the Engineer to do their due diligence - i realize that means different things for different people, for me it means i will never implement a new rev of a product into a production environment until others have vetted it for me.

jibbajabba

Glad someone agrees Lexluethar - Actually this thread made me think and I just asked the right people in my company to see how we are covered for vendor bugs in SOWs ...

I used to work for 888.com and responsible for the infrastructure - no matter what. As a result I always got a call directly from the CEO if stuff went wrong .. I don't even want to think what he would have said if I said "Sorry mate, you cannot do XYZ because of a bug" .. Likely "Don't give a &$%£ - fix it or else .... "

But at the end of the day - I.T. is like working in the army at the front ... you find a solution ... 'a' solution - no matter what .... but it really only works with Outlook issues, not necessarily VMware storage issues

Lexluethar

Spot on - I agree. It's the rub in this industry, especially if you are in the position to make infrastructure decision. While i'm not in the habit of doing the whole cover your own arse things, in instances like that if there is no other option available that I can find and management wants to move in another direction that may cause issues I make sure they are aware of my disapproval with the move.

markulous

EP 4 is released today and that resolves this issue.

kj0

Do you have a link?

markulous

Here is the KB with the issue:

VMware KB: Backing up a Changed Block Tracking enabled virtual machine in ESXi 6.0.x returns incorrect changed sectors

Here is the KB that has the patch notes:

VMware KB: VMware ESXi 6.0, Patch Release ESXi600-201511001