Cloning VMs in ESXi only

Is anyone able to clone VMs using ESX or ESXi only and without any assistance from VC?

I'm calling the VI API to clone a VM on ESXi 3.5.0 U4, but the operation is throwing a NotSupportedFault error. There seems to be some disagreement in the VMWare community if it is or is not possible to clone a VM using only ESX(i) and the VI API. I'm wondering if anybody here knows of a manual or programmatic way to clone VMs without using VC to control ESX(i).

Find more posts tagged with

Comments

astorrs

JDMurray wrote: »

Is anyone able to clone VMs using ESX or ESXi only and without any assistance from VC?

I'm calling the VI API to clone a VM on ESXi 3.5.0 U4, but the operation is throwing a NotSupportedFault error. There seems to be some disagreement in the VMWare community if it is or is not possible to clone a VM using only ESX(i) and the VI API. I'm wondering if anybody here knows of a manual or programmatic way to clone VMs without using VC to control ESX(i).

The problem isn't a lack of VC, it's more likely the fact that the "free" version of ESXi has a read-only implementation of the API and the commands you want to issue are probably considered "write" commands. Or am I way off base and you're still running in 60 day eval mode?

JDMurray

astorrs wrote: »

Or am I way off base and you're still running in 60 day eval mode?

I am still running in 60-day eval mode with U3 and U4 on different machines.

There are a lot of blog articles on how to clone VMs by manually copying and modifying the VM's files using VI Client, SSH, or the "unsupported" console. I need to find an automated way to do it through software only and build it myself.

cnfuzzd

JDMurray wrote: »

I am still running in 60-day eval mode with U3 and U4 on different machines.

There are a lot of blog articles on how to clone VMs by manually copying and modifying the VM's files using VI Client, SSH, or the "unsupported" console. I need to find an automated way to do it through software only and build it myself.

I am not that knowledgeable about this, but does it fail on the U3 machines? I know that ESXi U3 had the bug that granted access to the cli or somesuch, but was fixed in U4. I doubt thats it, but a possibility. If you resolve this, I would be interested in knowing how.

Thanks!

John

JDMurray

ESX and ESXi do not support the cloning of VMs, only Virtual Center does, and this is by design. You can manually clone VMs using VI Client by creating folders in a datastore, copying .vmdk files to those folders, changing the name of the VM in the .vmx and .vmxf files, and associating the .vmdk files with new VMs. I'm just wondering if anyone has developed an automated way (outside of VC) to do this with ESXi.

HeroPsycho

Why not use Converter?

Edit: If that won't work for you, you could surely put a script together using the VI Toolkit for Windows plus the Community Extensions, which has useful cmdlets for this such as Copy-TkeDatastoreFile.

http://vitoolkitextensions.codeplex.com/SourceControl/ListDownloadableCommits.aspx

Edit redux: http://powershell.com/cs/media/p/1655.aspx

JDMurray

I finally came up with a solution for this cloning problem on ESXi.

Cloning requires making a copy of a parent VM's files into a separate folder in a datastore, changing the name of the VM in the .vmx file, removing a few lines from the .vmx file causing a new UUID and MAC address to be auto-generated when the clone is first started, and registering the cloned VM in the ESXi inventory. You also need to make sure that the parent VM is not running or paused during the cloning operation.

For the copy and .vmx file modifications, I wrote a shell script that runs on the ESXi server itself and is run (by a remote client program I also wrote) via an SSH connection. Stopping the parent VM and registering the clone is performed using the VI API called from the same client program.

ESXi runs the very minimal BusyBox Linux, which uses ash for a shell. The scripting language implemented in ash is rather lacking in features one would expect to find in other shells, like bash and ksh. Even performing simple string substitutions is not supported, leaving external programs (like sed) to be used as a workaround. And BusyBox doesn't come with Perl, Python, Java or other shells preinstalled, so you're stuck with ash.

My ESXi VM cloinging solution isn't very elegant, but it works.

tiersten

JDMurray wrote: »

ESXi runs the very minimal BusyBox Linux, which uses ash for a shell.

BusyBox isn't a Linux distribution. It is just a set of utilities that is optimised for size. To further reduce the size, all of the various utilities are actually part of the same binary. It is just linked to different names and it works out what one you actually want based on that. The ash shell used is actually also part of BusyBox. Every byte counts if you're trying to fit everything into a 32MB image.

JDMurray wrote: »

And BusyBox doesn't come with Perl, Python, Java or other shells preinstalled, so you're stuck with ash.

You'd blow through that 32MB limit if you installed any of those because of all the libraries they need.

If you want to just remove a line then you could do it with grep as well. "grep -v" will show you everything except what the pattern matches. The BusyBox implementation of grep supports this option.

JDMurray

Yes, BusyBox is everything except the Linux kernel itself. And I never considered using grep, as sed works wonders for me.

They only thing I couldn't figure out how to do in ash was construct a command to rename files as I copied them. I wanted to loop through files in a directory and create the command cp oldfilename.oldextension newfilename.oldextension for each file to copy, but the args were always reversed no matter what I did. Something odd about variable assignment in a loop that's screwy. I didn't have the patience to figure it out; it turned out that the clone's files didn't need to have their names changed anyway.

tiersten

JDMurray wrote: »

Yes, BusyBox is everything except the Linux kernel itself.

Nearly all the utilities in ESXi are part of BusyBox but it still doesn't count as a BusyBox distribution. It is a combination of installed packages, scripts, directory layout and kernel that make a distribution. An example would be the firmware for most of the cheap SOHO WiFi APs, nearly all of them are based on an embedded Linux distribution called Montavista Linux that uses BusyBox.

JDMurray wrote: »

I wanted to loop through files in a directory and create the command cp oldfilename.oldextension newfilename.oldextension for each file to copy, but the args were always reversed no matter what I did. Something odd about variable assignment in a loop that's screwy.

Possibly. I've hit a few odd bugs in the Busybox ash before which needed an upgrade to the latest BusyBox to fix. Probably not something you really want to be doing though for ESXi.

JDMurray

I had a look through the BusyBox sources and the ash.c file. It appears that some standard shell features simply aren't implemented in ash--for whatever reasons. But with so much pre-existing shell source code to look at, why wouldn't the author(s) implement simple substitution (e.g., s/oldstring/newstring/)? I don't know if ash is currently supported or not.

Jordus

Good greif. You'd think you could just sysprep the VM and copy the vmkd file.

Guess not.

JDMurray

Sysprep is only used for a VM with a Windows guest OS. The cloning operation itself is independent of any guest OS installed (or not) in the VM to be cloned. VC is wonderful for automatically handling things like sysprepping during cloning, but a stand-alone ESXi server won't do any of that for you.

Jordus

JDMurray wrote: »

Sysprep is only used for a VM with a Windows guest OS. .

Well duh.

JDMurray wrote: »

The cloning operation itself is independent of any guest OS installed (or not) in the VM to be cloned. VC is wonderful for automatically handling things like sysprepping during cloning, but a stand-alone ESXi server won't do any of that for you.

Also duh.

My point was, that you seem to have to go to pretty lengthy measures (creating scripts, etc) to simply clone a VM for ESX when for Virtual PC, Virtual Server 2005, and Hyper-V you just copy the VHD and prep the guest OS accordingly.

SCVMM allows for ease of tasks for sysprep and cloning (via templates, etc) but even without enterprise level tools its easy enough.

dynamik

JD's a developer dude. He's trying to write an automated method of cloning the VMs himself. He's not just stuck trying to clone a VM.

tiersten

Jordus wrote: »

My point was, that you seem to have to go to pretty lengthy measures (creating scripts, etc) to simply clone a VM for ESX when for Virtual PC, Virtual Server 2005, and Hyper-V you just copy the VHD and prep the guest OS accordingly.

He is making a script because he wants it to be all automated. If you have the full paid version of ESX or ESXi then you're use VirtualCenter to handle it all for you. What his script is basically doing is the following:

1. Copy entire VM to new directory
2. Change IDs and name
3. Register new VM in the inventory

I don't see that as being particularly lengthy.

Jordus

Reading comprehension tends to pwn me.

JDMurray

I'm basically trying to write my own automation app that's a functional subset of VMWare's Virtual Center and Lab Manger.

Somebody gots to write all the kewl warez for you IT chumps to use.

tiersten

JDMurray wrote: »

I'm basically trying to write my own automation app that's a functional subset of VMWare's Virtual Center and Lab Manger.

Somebody gots to write all the kewl warez for you IT chumps to use.

VirtualMurrayCenter?

JDMurray

tiersten wrote: »

VirtualMurrayCenter?

Ooooo...maybe I should file a trademark application for that one. And maybe for a "Virtual Murray Machine (VMM)" too.

spaat

VMM, Sweet. Good stuff mang. lol

JockVSJock

***BUMP***

Yes, I knew there was going to be a thread on this as I have spent the last few days getting all of the updates/patches for the three xp clients and Win 2003 servers in my VMWare environment.

I was thinking, 'there has to be a better way for this.'

Goggling around didn't find anything free Virtual Centers for Linux either.

tiersten

JockVSJock wrote: »

Yes, I knew there was going to be a thread on this as I have spent the last few days getting all of the updates/patches for the three xp clients and Win 2003 servers in my VMWare environment.

Are you using ESX? Your other thread seemed to imply that you were using VMware Server...

JockVSJock

Just VWMare Server for Linux, under Ubuntu 8.04.

tiersten wrote: »

Are you using ESX? Your other thread seemed to imply that you were using VMware Server...

marc.techguy

JDMurray wrote: »

I finally came up with a solution for this cloning problem on ESXi.

Cloning requires making a copy of a parent VM's files into a separate folder in a datastore, changing the name of the VM in the .vmx file, removing a few lines from the .vmx file causing a new UUID and MAC address to be auto-generated when the clone is first started, and registering the cloned VM in the ESXi inventory. You also need to make sure that the parent VM is not running or paused during the cloning operation.

For the copy and .vmx file modifications, I wrote a shell script that runs on the ESXi server itself and is run (by a remote client program I also wrote) via an SSH connection. Stopping the parent VM and registering the clone is performed using the VI API called from the same client program.

ESXi runs the very minimal BusyBox Linux, which uses ash for a shell. The scripting language implemented in ash is rather lacking in features one would expect to find in other shells, like bash and ksh. Even performing simple string substitutions is not supported, leaving external programs (like sed) to be used as a workaround. And BusyBox doesn't come with Perl, Python, Java or other shells preinstalled, so you're stuck with ash.

My ESXi VM cloinging solution isn't very elegant, but it works.

Do you think you could share your code with everyone? Or send me a copy?

JDMurray

I can't share the actual script I wrote, but I explained what it does, and it's only about a page-and-a-half of script. The tricky part is learning the VI API.