A Dell PowerEdge T710 followed me home, now what?

I got a call last week from my older brother that he got a job cleaning out a medical clinic, the clinic found a new location but left a lot of hardware behind in the old place. He told me I could pick through what I wanted because all of it was going in the trash otherwise. Much of it was truly trash but I found some things of value. Saturday afternoon I came home with, among other things, a Dell PowerEdge T710. It's got 2 quad core Xeon processors, 36GB RAM, among other goodies. What is missing though is any storage. It's got a RAID controller capable of holding 16 2.5" SAS/SATA drives but all storage was removed for data privacy reasons. I ordered caddies and drives to fix that, which cost me about $200, not bad so far. I'm just starting with four SATA spinning disks to get me going, I might add a SSD later.

Other than just sharing my joy on such a find I thought I'd also share some of my thoughts on how to make the most of this. The path I'm considering now is installing ESXi 6.0 and trying out the PCI passthrough so I can also use it as a workstation. I'll need a video card, that's pretty much a given. I might need a USB card too since grabbing the onboard USB might cause problems. Getting sound out is what is stressing me the most.

In the past if I had a computer lacking on-board audio (such as a repurposed server like this), or the on board audio was not supported by my preferred OS, then I'd just grab a PCI sound card from what seems like my bottomless box of old PCI cards. While I'm not lacking PCI sound cards I've discovered an absence of PCI slots in this computer. Looking at my options to get sound I've found that this might not be as simple as picking up a $10 part the next time I find myself in the vicinity of a Best Buy. Maybe it is, I'll get back to that in a bit.

While I also want to discuss the options for turning this server into a virtual machine playground the software aspect can shift with little to no cost. I can try ESXi (which is what I'm going to try first), Hyper-V, as well as other free and non-free virtualization solutions. I may try them all in time. What I want to have in the near future is the ability to still use this as a general purpose workstation through some virtualization or on the metal. I'll probably want to run Windows 10 since the instructors at the university seem to prefer students run that, and I can get a Windows license from the university for nothing (Microsoft has a deal with the university). If for some reason you believe ESXi is not the best choice then make a case for alternatives.

I'll need to make a choice on how to handle video, which plays into the audio choices too. All my displays are currently VGA. I have a TV on the kitchen wall I'm willing to repurpose as a display, I'd like a new TV too. If I go with VGA then I can use my current speaker setup, then I'd need a sound card of some kind. I'll need a new video card so I may as well get one that supports sound from HDMI and/or DisplayPort. This means upgrading my TV and using my current TV as a monitor, or buying a new monitor with speakers (or speaker output anyway, but it seems one comes with the other).

A PCIe sound card is right out, unless I missed something in my research. The PCIe sound cards I've seen are not cheap, and even if they were free it'd cost me a potentially valuable PCIe slot. A USB sound adapter is a possibility as USB passthrough is a common feature for every VM solution I've seen, relieving any hangups I might have with trying to passthrough a PCIe device. Is there a PCIe card that combines an audio output with USB ports? That'd save me a slot. It's not just the loss of the slot, or the absolute cost of the card, it's the relative cost. For lower cost than a dedicated sound card I can get a USB audio device, or upgrade a display to include speakers and use sound from a video card I'd have to buy anyway. Again, I may have missed something.

Another option that crossed my mind is using Bluetooth audio. I have a Bluetooth speaker I use with my iPhone and iPod so the cost would just be the Bluetooth adapter. Experience tells me this might be finicky. Getting a Bluetooth adapter might be handy for other reasons though. I've seen software solutions for situations like this, where a pair of receiver and transmitter software can send audio from one computer to another over a network. This also sounds finicky too, as network lag could mean watching a lecture on the screen and the lips moving does match audio. Not a complete killer but it can be distracting as hell.

The question is not just what my options might be, although I'm sure I missed something in my research so far. The question is what are the BEST options. Have you tried something like this? If so, then what has and has not worked for you?

Find more posts tagged with

Comments

EagerDinosaur

Apologies for taking a negative view, but have you checked the power consumption and worked out what it will do to your electricity bill? Having a second-hand commercial-grade server at home has lot in common with a hungry noisy Great Dane following you home.

MacGuffin

EagerDinosaur wrote: »

Apologies for taking a negative view, but have you checked the power consumption and worked out what it will do to your electricity bill? Having a second-hand commercial-grade server at home has lot in common with a hungry noisy Great Dane following you home.

First, I don't care. Finding this computer probably just saved me $1000. Since dropping my MacBook Pro on a chair edge last year it's been limited to desktop duties, the battery has not kept a charge since. It's a nice computer with a quad core i7 and 16GB RAM but it's still a compromise for trying to do ESXi stuff. I've been using an older laptop for taking to class which has been working reasonably well so far. I was thinking of buying a new computer, desktop or laptop, that would be something reliable and fast enough to do my classwork. If getting this server working costs me $500 - $700 in parts to make a reliable computer out of it then I'll have twice the computer for half the price from what I was looking at. I was thinking of buying something like a Dell XPS, MacBook Pro, or iMac before fall classes. I might still want something new to take to classes but now I can feel comfortable getting a much cheaper tablet or laptop and leave the heavy work for my computers at home.

Second, this may actually be an improvement as far as power consumption and noise. Right now in my basement is a collection of computers tasked with various functions. As each computer (except my battery-less MacBook Pro) has only a single or dual core processor, 4GB or less RAM, no real VM capability, and some hardware quirk or another, they are useful for what I have them do but they also make noise and heat. If this server does what I think it can do then I can retire two or three other computers. This T710 doesn't seem to be all that loud, so if I can get rid of just one noisier computer I can reduce the noise level a bit. Part of this upgrade is the plan to replace a CRT display or two with a flat panel, that will save on the power consumption too. I don't NEED a new computer to upgrade the display but a new computer does kind of force the point since I'd rather not get a VGA capable video card limited to 1920x1200 when I can get a non-VGA card for a few bucks more that can do 4K resolutions. I'm not likely to invest in a 4K display just yet, but I'd like to get one soon, maybe next year. Prices keep going down so maybe I'll buy myself one for Christmas.

Fall classes start in a couple weeks and I'd like to have my basement "lab" fixed up by then. The plan is to have on one desk the "new" stuff with this T710, MacBook Pro, and another computer or two. I'll get a new monitor, or use my current TV as a monitor and upgrade my TV. This would mean putting a couple CRTs back on my "spares" pile or just carting them off to the trash. My second desk will have "old" stuff, with a VGA monitor and two or three computers there. One computer I intend to make a kind of Linux server out of to share a printer (I don't have an AirPrint capable printer but Linux will do the translation for my Apple devices), allow me to play with Drupal and other web stuff, and more. Another old computer runs some old games I like to play, does well as a serial terminal for talking to the Cisco gear stacked next to it, and just generally keeps running well so I keep it around.

I think I have a good idea on what my "lab" should look like in a couple weeks, but I'd like some ideas on some specifics that could save me some time, money, and frustrations.

Have you tried PCI passthrough on ESXi to allow a virtual machine to be used as a desktop PC? How has that worked for you? How did you manage to get audio out of it?

Do TVs make good computer displays? If I want sound from my video card should I look for a monitor with speakers in it, or will a TV with HDMI inputs work as well? Is there a video card that has analog audio out that I'm not aware of? Just generally what can I expect if I try to get sound from a video card and display connected with HDMI or DisplayPort.

What other options should I consider? USB audio seems like a good option, as I can use USB redirection to have a VM see it as a something directly connected to it. Bluetooth audio is an option, anyone use that regularly without problems? There's lots of options here, if you want to share your experiences and frustrations on what you've tried then that could save me (and others with similar goals) from repeating your mistakes.

Welly_59

Have you fired that bad boy up yet? I had a r710 and honestly I couldn't bear to power it up due to the volume

stryder144

I have a 32" Samsung TV that I use as a monitor on my Plex Media Server. I connected my PCIe graphics card, via HDMI cable, to my TV and it produces sound. HDMI is awesome since I only need one card to handle both video and graphics. Worked with Ubuntu and with Windows 7 but I have not tried it with ESXi.

MacGuffin

I found this article that kind of answers some of my questions.

https://www.pugetsystems.com/labs/articles/Multi-headed-VMWare-Gaming-Setup-564/

I don't plan on the "multi" part so much, and not the "gaming" part either, but the ESXi part checks out. Maybe I'd do the "multi" part at some point so I can have something like a Windows desktop alongside a Linux desktop, both having access to a real GPU and such.

They built a machine that could do four headed gaming but I just want one "head" and not lose access to the ESXi console. I'll have to figure out the KVM sharing issue, that kind of depends on the choice of monitor and video card. I'd end up with a "two headed" computer, one being Windows 10 (most likely anyway, might be some other desktop OS) VM and the other the server console. If I want to avoid the use of the onboard VGA for the server console then I'll need two video cards. I'll want to be able to run other virtual machines but they'll be "headless" like how ESXi intended. I should be able to do management of the "headless" VMs from the "headed" VM for a lot of things. Some VM management will have to be from another PC, which is to be expected.

They relied on HDMI for audio, which means needing a display that supports this too. Not a big deal but I'll have to keep this in mind as I select my hardware. HDMI is pretty much a dead end now, people are moving on to DisplayPort, but the basics should be the same if I go with DisplayPort instead.

This is a relatively old article but it does give an impression of what I'm trying to do here. If this works out like I think it will then it might be something I'll try on other computers. Also being an older article it does not mention the use of technologies that came out since, USB-C as one example.

MacGuffin

Welly_59 wrote: »

Have you fired that bad boy up yet? I had a r710 and honestly I couldn't bear to power it up due to the volume

Yes. I dug up an 8GB USB flash drive and installed Ubuntu. It's very quiet, certainly quieter than a lot of other hardware in my basement. I should have my drives by the end of the week so I can start on making this my ESXi playground. I don't expect the addition of the hard drives to add much to the noise level.

On the Ubuntu desktop I put that CPU monitor in the toolbar at the top of the screen. It brings a smile to my face to see 16 virtual LED bars hop up and down as I surf the web on it. The video seems a bit glitchy, the mouse pointer likes to disappear. I don't know if this is lag from booting off a USB2 drive, the crappy video, or what. Either way I expect to have this fixed in time.

jelevated

Just slap Win 10 on there and virtualize Linux in host. Getting ESXi involved in what you're trying to do will be needlessly complex.

MacGuffin

stryder144 wrote: »

I have a 32" Samsung TV that I use as a monitor on my Plex Media Server. I connected my PCIe graphics card, via HDMI cable, to my TV and it produces sound. HDMI is awesome since I only need one card to handle both video and graphics. Worked with Ubuntu and with Windows 7 but I have not tried it with ESXi.

Good to know. What kind of resolution are you getting? 1920x1080? 32 inches is a pretty big screen for my desk. I'm thinking more of like a 20 to 24 inch screen, so long as I can get 1080 lines or better. Beyond 1080 lines and screen prices make a big jump, or so it seems to me, and I'm not sure I want to make that investment yet.

I've been looking around at video cards and it's difficult to tell if the cards support sound or not. I assume that if the card has VGA along with the HDMI that it does not support audio, is that a safe assumption?

I've seen video cards in the $40 range that will give me HDMI but might not support the PCI passthrough I need to make ESXi happy, and likely no audio either. The article I linked to about the "LAN party in a box" says that if the video chip can be seen by the VM then so will the audio, at least that's what I got from it. They also said they had better luck with ATI than nVidia. I think an ATI Radeon or FirePro in the $80 to $120 range should work. It looks like I can get some pretty nice cards for $160 or less, capable of quad 4K.

If I add a USB3 card for about $50, pass that through to the VM, then I should be able to plug in all kinds of stuff and get a speedy connection too.

MacGuffin

jelevated wrote: »

Just slap Win 10 on there and virtualize Linux in host. Getting ESXi involved in what you're trying to do will be needlessly complex.

The goal is to get some ESXi experience as part of this exercise. I will be running ESXi in some form, on the metal or in a VM. I'd rather run ESXi on the metal since that's a more real world environment. Running ESXi in a virtual environment is something I'd consider, especially if I have trouble with ESXi on the metal.

If for some reason I cannot get ESXi to do a proper passthrough of KVM + audio then I'd probably just run ESXi on the server and continue with Windows on another computer. If I can get this to work though then I have the best of both, a speedy Windows PC and an on-the-metal ESXi server to play with.

jelevated

I'm a huge ESXi advocate, I'd virtualize my toaster if there was an OVA for it, but when you start getting into requiring passthru and stuff..it sounds like it should work in theory but you always just run into strange issues. Since its not a dedicated ESXi lab box thats why I lean more towards running Win 10 and virtualizing there instead. I tried getting a simple USB 3.0 card (on the HCL) to work in ESXi 6.0. It works, but it took me several hours and is ultimately kinda slow.

I get what you are saying about ESXi running on the Metal, it really is cool to explore all the different options available and see what your hardware is capable of, but thats only a small subset of ESXi. I and many others received our VMWare Professional Certs on nested ESXi installations (three in my case) which you could easily run in VMWare workstation. Then you have all the horsepower you need to use it as a workstation.

Essendon

How about you check out our Hands on labs, you pay nothing, ever, and use them till your eyes bleed. Link here > VMware Learning Platform

MacGuffin

jelevated wrote: »

I'm a huge ESXi advocate, I'd virtualize my toaster if there was an OVA for it, but when you start getting into requiring passthru and stuff..it sounds like it should work in theory but you always just run into strange issues. Since its not a dedicated ESXi lab box thats why I lean more towards running Win 10 and virtualizing there instead. I tried getting a simple USB 3.0 card (on the HCL) to work in ESXi 6.0. It works, but it took me several hours and is ultimately kinda slow.

I'm certainly open to doing as you propose. Running Windows 10 on the metal and ESXi in virtual machines is certainly an option but how much does that change my hardware dilemma? Let's discuss.

If I run Windows on the metal then I need hardware compatible with Windows. If I run Windows in a VM with PCI passthrough then I still need Windows compatible hardware, but with the added requirement of supporting PCI passthrough. Judging by my research so far just about any video card that supports audio and 4K resolutions will support PCI passthrough for "free". I might have to pick the cards more carefully to make sure I get PCI passthrough support but it looks like any current ATI card will do this. I can get a cheaper video card, spending somewhere in the $50 range instead of $150 range, but that means likely doing without audio on the HDMI port which then opens me up to finding something else to give me audio.

Maybe I can find a cheap video card that does PCI passthrough but no audio, what do I do for audio then? I can get a PCIe sound card, that occupies a slot and the cheapest I've found them are about $50. If I can find one as cheap or cheaper as the other options then I'd consider it, but that means using up a slot I might want in the future. USB audio is an option, and probably less than $10 too. I might get one regardless at that price, might come in handy for other computers too. Using Bluetooth audio would cost me around $20 since I already have speakers, I'd only need the USB adapter. Basically anything that plugs into USB will work either way since USB passthrough is pretty trivial for any virtual machine, and Windows support of these devices is an almost given.

I'm not sure if your USB3 issues apply since my intention is to passthrough the card, do ESXi drivers apply if I do this? That "LAN in a box" article I linked to before seemed to imply such a task was trivial. I don't *NEED* USB3, I guess, but I want it. A USB3 card with USB-A ports is less than $50, with USB-C ports I might have to pay a bit more. Using just the onboard USB2 is an option but it adds more guesswork in the VM configuration and, obviously, limits me to USB2 speeds on anything I plug into it.

If I want cheap and don't care about running ESXi then I can just get a $10 USB audio adapter and be done buying hardware, I can just use my current VGA monitor and the onboard VGA port and use Windows at 1152x864. If I want to run ESXi on the metal and not do a desktop PC VM passthrough then I don't even need the USB audio adapter. If I want to upgrade to a flat panel and/or higher resolution, on the metal or virtual, then I need a new video card. Any video card that does HDMI audio saves me on getting some other audio device, but I'd need a new display which is something I was planning on anyway. What's the extra cost for getting a computer display with builtin speakers? $10? Maybe I should just get a monitor with speakers anyway to keep my options open.

As best I can tell the difference in hardware costs might be $50 between the two since I'd have to get a higher end video card, all else equal. If it comes down to needing to save $50 then I'll put the money in a nicer video card and not get the USB3 card, I can always get that later if I need it. If I get hardware that covers the option of running a desktop PC on ESXi in the "LAN party in a box" style then I have the hardware for running Windows 10 on the metal and ESXi in a virtual environment, the reverse might not be true.

I would like to get the hardware figured out before classes start. I don't know if others have this experience but once that screwdriver comes out I can see hours disappear. If I'm swapping video cards or what not then I'm making a mess of the place and I can't just walk away. If it's a matter of software then I can turn off the computer and pick up where I left off without tripping over computer parts scattered about every horizontal surface (and perhaps a few vertical ones too).

I'd like to get the software parts figured out before fall classes start too but as long I have the hardware in order I won't be tempted to take a screwdriver to things and distract from getting homework done.

So, yes, running Windows on the metal is an option and it might open up my hardware choices a bit and save me a few bucks. If I get the hardware for ESXi and the PCI passthrough doesn't pan out for then I can fall back to Windows on the metal. If someone thinks I'm mistaken on this then I'd like to hear your argument.

MacGuffin

Essendon wrote: »

How about you check out our Hands on labs, you pay nothing, ever, and use them till your eyes bleed. Link here > VMware Learning Platform

I can do that too. I saved a server from the landfill that's probably worth around $1000. I spent $200 on it so far to get some drives and caddies for it, which is something I'd need anyway if I want to move beyond booting from a USB drive or DVD. Another $250 or so to get a video card and display gets me very nice workstation from it. I've read a couple articles on your blog and I see you have your own home VMWare lab, looks like you spent some good money on it too. Has anything changed since you posted those articles you'd like to share? You could save me a few bucks.

Essendon

Yep I did read you gave the server a new lease of life. It might be good to be able to tinker around with it and build a nested lab of your choice. In fact, I had a nested labs for a long time, some of my VCAPs were achieved using the nested lab. I did go all physical for my lab and truth be told I haven't used it to its potential. I've been in a design role for almost as long as I have had the physical lab, spent more time in Visio and Word rather than hands-on. Nothing's changed since I posted the article about my lab. If I could turn back time, I wouldn't buy the lab. VMware's Hands-on Labs are sufficient for most things. If you remain keen, max out the RAM and give the server adequate storage. Your bottleneck will almost always be the SATA disk, CPU and RAM aren't usually the issue if they're both maxed.

MacGuffin

Essendon wrote: »

Yep I did read you gave the server a new lease of life. It might be good to be able to tinker around with it and build a nested lab of your choice.
...
If I could turn back time, I wouldn't buy the lab. VMware's Hands-on Labs are sufficient for most things.

Given that the server is nearly "free" in that the bulk of the cost I'm imposing on myself is the upgrade in display I'm going to at least give the nested ESXi lab a try. If it doesn't work out then I can still repurpose the machine as a workstation. I can still run ESXi in a VM on Windows or Linux since the computer has the power to do so. If I do that then I'd lose the on the metal aspects of this and potentially a hit in performance from running on a desktop OS. I'd also lose the "cool" factor of running a VM with a physical video card.

I didn't realize how much of a difference it makes to have a video card versus the onboard video until I ran an experiment in swapping the one PCIe video card I have from another server to desktop conversion to the T710. All I was doing was surf the web and the difference the video card makes is AMAZING. With the onboard video the computer was unbearable, the mouse pointer kept flickering, rendering web pages took a very long time, and it was just generally aggravating. After the video card switch the computer I took the card from had a flickering mouse pointer and slow performance. The T710 just FLEW in web surfing. I was still running from USB or DVD, which I hope is what is causing any remaining lag I'm still seeing. Any lingering idea of going without a video card just evaporated.

The card I used for my experiment cost me something like $40 when I got it about a year ago. A video card with an ATI FirePro chip would be somewhere around $120. I'll try something like that for a start. If that doesn't do the PCI passthrough like my research leads me to think it would then it's still a very nice video card. I can go with Windows on the metal at that point or put that card in a different machine and try again with a different card.

I'll need to still figure out the display I want. I'm thinking of a TV or monitor with speakers in the 22 to 27 inch range. Any bigger and it might not fit on my desk, or within the budget I set for myself. If I limit my display budget to $150, or perhaps a bit more, I can get something very nice. I'll leave it at that since this is the virtualization forum and discussing displays is getting off topic.

The display and video card should be less than $300 for both. If someone thinks I should go down a different path for video and sound then I'll consider it.

I'll have to remember to check out those hands on labs at some point.

Essendon wrote: »

If you remain keen, max out the RAM and give the server adequate storage.

Maxing out the RAM would mean having 192GB on the T710 and much less money in my bank account. I'll go with the 36GB I have now unless I consistently hit the limits. The smallest drive I have now that will fit on the RAID controller is 200GB, is that "adequate" by your metrics? I have more drives in the mail for me already, and certainly the option to buy more.

Essendon wrote: »

Your bottleneck will almost always be the SATA disk, CPU and RAM aren't usually the issue if they're both maxed.

I'll start with what I have. I'll invest in a SSD once I get an idea on where I'm going with this, such as how much more money I want to spend and how much space I'll need. Thankfully the adding of more drives won't involve too much time with a screwdriver so I shouldn't feel reluctant to add more later, even after classes start.

I mentioned the RAM aspect already, I think I'm good with what I have for a while. I can get more later if my budget allows.

I tried upgrading the CPU in a different server to desktop conversion and I just made a mess, lost many hours and whatever I spent on the CPU (which admittedly wasn't much, and I couldn't send it back because the return window closed on me). That CPU is around here somewhere and I might be able to find a motherboard that can take it. I will say it was a learning experience, I learned to be much more careful on part selection. That's if there is a next time.

A quick word on KVM use... A new KVM with DVI, HDMI, or DisplayPort would be around $100 for a 2-way, and about $400 for a four way. It'd cost nearly double for doing fancier things like support of 4K video or multiple monitors. Given that this is about the same cost as another display, even with the lowest price KVM, then I believe it wise to just buy more displays instead of the KVM. There might be a point where a KVM switch makes sense and I have to figure that out. Where that point is I'll need to figure out as well.

MacGuffin

I got the drives and caddies I ordered today. Of the four drives I got only one is supported by the RAID controller. I'm not too upset by this because the drives were cheap and I knew that there was a possibility of a problem. I'd likely be in a much poorer mood if none of the drives worked. The RAID controller is, as best I can tell, a LSI PERC H700. The only differences I can see, physically, is that the supported drive is 9mm in height and the unsupported are 7mm in height. The supported drive is Samsung branded, unsupported are HGST.

What is it that makes one work and the other not? Can I somehow "force" the controller to use these drives? The RAID controller sees the drives but marks them as "blocked" and "unsupported" in the setup utility.

More importantly, what should I look for in drives to make sure they work with this controller? I intended to get more storage from the beginning but thought I'd start small with these drives. I'm glad I did since I discovered this problem I did not know about before.

I realize that getting a SSD would improve performance and that's likely what I'll get next but how do I know the SSD will work on this RAID controller?

For now I'll play with the drive I got. I can at least get an idea on how well this server will perform.

MacGuffin

I figured out my hard drive problem, a firmware update in the RAID controller opened up support for previously unsupported drives. What is annoying though is the RAID controller still shows an error condition on the drives because the drive firmware fails a check, which potentially masks more serious conditions like over temperature errors. I set up a 1TB RAID with some cheap secondhand laptop drives. I could use some advice on how to make a smooth transition to SSD drives as time passes and money comes in.

I've discovered that doing PCI passthrough with the onboard USB is dangerous if I don't know which port goes where, especially when booting from a USB device and using a USB keyboard. I got that figured out but I do think that getting a USB 3 card would not only add performance on some devices but also avoid such confusion and frustration in the future.

I experimented with the PCI passthrough on the cheap NVidia card I have and I have not had much luck with it. There's a lot of variables that could be causing my trouble. A not terribly expensive ATI card should fix this problem. I did notice that the Ubuntu virtual machine was running slow again. I guess that having a proper video card really does matter on getting good performance on a desktop for even seemingly trivial tasks now.

Other matters will occupy my attention for the next couple days but I'll be shopping for a video and USB card soon. Given how my experiments have been going I may have to buy a handful of both to upgrade my other server to desktop conversions. Even a moderately powerful card by today's standards is quite the upgrade from the secondhand stuff I've collected and put up with so far. This talk on video cards is probably best discussed in another thread. I think I'll start that discussion elsewhere.