after an hour or so of putting the stuff together, I was able to finally start playing around with the ESX 4.0 bits.
I was relieved (and impressed) to find out that vSphere had no problems at all installing and running on my new whitebox:
My whitebox has the following specs:
CPU: Intel Core 2 Quad Q6600 Kentsfield 2.4GHz LGA 775
Board: ASUS P5BV-M LGA 775 Intel 3200 Micro ATX Intel
Drive: Seagate ST3640323AS 640GB 7200 RPM SATA
Memory: CORSAIR 4GB (2 x 2GB) 240-Pin DDR2 SDRAM DDR2
CD-ROM: ASUS 22X DVD Burner Black SATA Model DRW-22B1S
The installation was done from the SATA DVD - no issues at all.
All the on-board components (i.e. the dual NICs) were also detected no problem.
I am just getting going with this thing... Tomorrow I will install vCenter and a few VMs and after that will be testing upgrade scenarios from 3.5->4.0. Will post accordingly.
Hey I have been looking high and low for a vSPhere compatible whitebox. Looks good and I am glad I found your parts list! What did you get for a case? Will you be able to use FT with the CPU you chose? do you have a preferred website you bought your gear? Any update on how it is running? I know you just posted this but am excited to get some stuff ordered.
ReplyDeleteI get all my parts from newegg.com. To be honest I have not tried FT yet - I have just one box running vSphere at present. My other two whiteboxes still run 3.5 and I am planning to bring one of them up to vSphere shortly and then will check out FT (I found a god list btw: http://www.gabesvirtualworld.com/?p=456). My other "whiteboxes" also run the same CPU (Q6600) so I remain hopeful and will post the results of my test
ReplyDeletePaul,
ReplyDeleteDid you have to use the "usb trick" to get the hard drive working properly? Or does that only apply for some motherboards?
Does it matter what type of hard drive you use? So could you have 4x500GB's in your whitebox and ESX will detect all 4?
No, I connected a SATA DVD drive (both the Samsung and Asus I've got worked without any issues).
ReplyDeleteIn the BIOS, make sure you set the drive controller to AHCI mode - otherwise ESX will not detect the drive. I've got two 840 GB drives in one of my whitebox servers (1.6 TB total), and it detects it no problem. But I have not tried four drives yet...
I have been following your vSPhere whitebox build. How are things going with it? Have you been able to do much more with it? I purchased the parts from your parts list. I wanted to thank you for saving a bunch of my time from trying to figure out what hardware worked. Do you know if this hardware list is 3.5 compatible? Any update on your findings?
ReplyDeleteThanks again for the inspiration to build a home lab!
Yes, this hardware list is 3.5 compatible - I have another server built based on it and I have 3.5 running on it no issues. The only thing that took me a while to figure out was that ESX installer did not see the drives until I switched them to AHCI mode in the BIOS.
ReplyDeleteSo far I have not done much with my new vSphere server - Apart from installing it just to make sure it works on my hardware. My plan is to do a few migrations first (from 3.5 to 4.0) to test different scenarios. Also, I do a lot of testing with Site Recovery Manager and it is not compatible with vCenter 4.0 so I will actually be staying on 3.5 probably until H2'09 when there will be a new release of SRM compatible with vCenter 4.0.
I am glad you found it helpful - keep checking back for more info, I will be posting as my testing progresses and will be happy to answer any questions.
Is there any problem with hyperthreading and the q6600? I came across this:
ReplyDeletehttp://ultimatewhitebox.com/cpu/12
I'd like to know how this affects your setup.
good question... You are right, HT is not available with this proc. However, the value of HT has been questioned and there are only certain loads that will benefit from it. I have also seen articles about performance issues due to HT.
ReplyDeleteRemember that a whitebox is purely for testing/training purposes, I would never recommend this type of deployment in prod. Some things will not work on a whitebox - i.e. HT or FT (FT also does not work with this proc).
Are there stability problems? Because yes I want a home test box (maybe two) at the end of the day, but I have several physical hosts that do barely anything (fileserver, im, dns, email) that could shrink down 8 hosts into one with this.
ReplyDeleteI think as long as everything is supported there shouldn't be a problem with this setup. I was looking to have a freenas box with iscsi setup so another whitebox could be thrown in the mix at a later date.
What's your uptime like?
It's ESX... Uptime is great ! :-)
ReplyDeleteFrom the standpoint of uptime/reliability/etc., my whitebox servers have been fantastic. They stay up all the time (except when there is a power blip... I cheaped out on UPS so they are afected by a power outage). Other than not supporting certain features (i.e. HT) due to not being on the official VMware HCL).
When I said I would not recommend a whitebox server like that in prod is that there is no way you could get Support from VMware for a whitebox implementation (they only support hardware from their HCL list)... Other than that it's ROCK SOLID...
You can create a cheap NAS storage server based on Linux (i.e. Ubuntu). Coincidentally I also run one of those at home - check out a blog post in March on how I implemented it.
Paul,
ReplyDeleteGot most of my equipment in and am waiting on the MB and CPU. They should arrive tomorrow. I can't wait to start getting things installed. I will install 3.5 first then upgrade so I know the process in that. I too am a consultant. I currently run a Openfiler server and Freenas and plan to present one of them to the host.
You mentioned something about the DVD drive. Is there something to look for in that? I have an HP SATA DVD drive and figured I would just use that.
Was there anything you had to do to get the onboard NICS to be recognized? I think you said they just worked correct? Also, Have you tried to run ESX in a VM on your setup?
Well it was just the other day when we got a new appliance in and I said, wait a second we bought support for this? Since that is the case I never rely on support as it is.
ReplyDeleteThese whitebox HCL's are very cryptic. It's almost a sure bet to just buy the same stuff as someone that's posting. Also, they don't sell Q6600's from good places like newegg or zipzoomfly. I'm going to make one for work first then make my own after my paid for experience. I do want the best cpu/motherboard for the performance per dollar. Q8200 is easy to find, but I wouldn't mind throwing a little more money at it for HT'ing. I did scratch my head at FT, what does that stand for / do?
@dwilk: Great!
ReplyDeleteAs long as the drive is SATA, it should work no problem. I have tested 2 different SATA DVD drives (Asus and Samsung) and they worked like a charm.
both of the onboard 1Gb NICS are working great - this is the best thing about that motherboard - components work and so you don't have to buy them separately.
What do you mean by "run ESX in a VM on your setup" - did you mean Virtual Center? yes, I run my VC as a vm.
@Tim:
ReplyDeleteYou are doing the right thing - only buy components that have been tested already and know to work with ESX otherwise you may be in for some serious troubleshooting (but, on the other hand... that's a lot of fun and satisfaction if you discover another working component :-)
FT = Fault Tolerance. vSphere allows you to protect virtual machines so they both write in lockstep. If one fails, the other one can pick up the load.
I was asking if you ever had any luck with running ESX or ESXi in a VM on your ESX host. I have seen people running ESX in Workstation for demonstration purposes. I have done ESXi in workstation but ESX is a different animal. I will more than likely try the install of ESX in a VM to see if it will work. This way I don't have to buy another whitebox for HA. I can test things out all on one Box.
ReplyDeleteOne of the main reasons I went with your parts lists is the NICs being detected. That is huge in my book!
I will add my 2 cents on FT. While in the corporate world it will be huge in a test environment I don't think it is really worth it. That was low on my list of home server features. you just right click the VM to enable the feature.
What a great thread going on!!
Just checked an my MB and CPU are out for delivery! Will start the install as soon as it gets here!
Ok. Yes - have run ESX (3.5) as a VM on my VMware Workstation - tyou can Google the docs out there that cover the few things needed to make it work. But I found it slow and unresponsive. Still, good solution for self-training.
ReplyDeleteThat's awesome news - it will be a real pleasure for you to be able to experiment "in your basement" with the different Vmware products.
Alright I have some information to share and maybe your other readers will appreciate this. I purchased all the parts on the list above but used an HP SATA drive for install. ESX 3.5 installed no problem. I left out 4 GB of memory to see how it would handle the "upgrade". That was fine. After the install I booted up and it saw total 8GB memory. I then uploaded some ISOs to local datastore and the speed was great. I then installed a 2003 server and used that for Vcenter. That installed with no issue. I then went to bed and woke up ready to work on installing the iSCSI LUNs from my freenas box. This is were I woke to find some errors on the console screen.
ReplyDeleteThe errors were red and pertaining to the CPU. I had a device might be offline and then I rebooted and upon reboot I received a CPU SMP:437 invalid HT config error. I then went into the BIOS and changed "MAX CPUID" from enable to disabled. This seem to clear up the CPU error for now. I will let it run and see what happens. Otherwise easy build as it saw all hardware! Thanks Paul!
Do you know of an easy way we can compare BIOS settings? Have you updated to the latest? I have not yet.
That's awesome!
ReplyDeleteI am glad it worked out for you.
Also, it is great you provided the advise re: CPUID. btw - I never had to modify that value and did not run into any issues. Perhaps it may be a result of a different BIOS version? Can you tell me what BIOS version your motherboard is at?
I have not done any BIOS changes - left everything at default except I changed the drive access to AHCPI mode. Have you done any other changes?
Paul,
ReplyDeleteI ended up buying this setup:
Intel i7 920
MSI x85 Platinum
12 GB ddr3
2x1TB Samsungs (will raid later after testing)
From what I can gather, it's not worth the hassle with the onboard nics. I installed ESXi 3.5 up4 just fine but it doesn't have a nic to go on, so no further testing until the right nics are purchased.
However, I wasn't able to install ESX 3.5 up3 (had cd on hand.) It could not find my hard drives. Maybe something to do with ichr10? Any thoughts?
It should be a speedy machine, might lack i/o right now but I want to do some testing with it ASAP! I wish I knew a local place to get intel nics (if any thought on that.)
I am not familiar with the MSI x85Platinum board and could not find it on ESX blogs... you may be the first one to try this one. Are there any disk controller options in the board BIOS that you can change? Try that - i.e. try switching to AHCI if such option exists.
ReplyDeleteGreetings: I just finished the 3.5i Fast Track and got my cert but need a whitebox that will run 3.5u4 and also the 4.0 ver. with your listed hdwr, CPU: Intel Core 2 Quad Q6600 Kentsfield 2.4GHz LGA 775
ReplyDeleteBoard: ASUS P5BV-M LGA 775 Intel 3200 Micro ATX Intel
Drive: Seagate ST3640323AS 640GB 7200 RPM SATA
Memory: CORSAIR 4GB (2 x 2GB) 240-Pin DDR2 SDRAM DDR2
CD-ROM: ASUS 22X DVD Burner Black SATA Model DRW-22B1S
Will this work on both vmware vers? Also that DVD burner isn't abilable any longer, will any sata dvd work? What about the case... any requirements? Thanks very much! Z in AZ
Hey
ReplyDeleteSorry for the late publish of your post... just started a new job and times are hectic so I've let my blogging duties slide...
Your configuration looks good. Make sure you double check with the ESX whitebox blogs (Google "ESX whitebox") for parts compatibility.
As far as the DVD, any SATA should work...
So i'm having an issue trying to install ESXi 4 on my whitebox that I just got and put together. I get PSOD errors during the install and they aren't always the same ones...they are different almost everytime I try the install. Here is the ESXi version and hardware:
ReplyDeleteESXi: Label_version 4.0.0 | 171294 - 06/22/09
CPU INTEL|C2Q Q6600 2.40G 775 8M R
MEM 2Gx2|CORSAIR VS4GBKIT800D2 G RT--8GB total
Western Digital Caviar Black 640GB 7200 RPM
32MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive - OEM -- 2 total
DVD BURN SAMSUNG|SH-S223F 22X BK
Almost the same hardware as you have, but the DVD burner and hard drives you purchased were no longer available at newegg.
The only thing that was different prior to running setup was that I had to clear the CMOS due to a chassis intrusion error that was preventing me from getting past the POST. I did set the storage controller to AHCI and the MAXCPU ID was already disabled. I am going to try and get 3.5i and try that also. Will post those results. Any suggestions on why I'm getting these PSODs? Maybe clearing CMOS set a CPU or other setting that is causing this? Any help would be greatly apperciated.
So I bought most of the same hardware as you except for the Seagate hard drives and the dvd drive because they were no longer available on newegg. Here is the ESXi 4 version that I got and the hardware:
ReplyDeleteESXi--4.0.0 | 171294 - 06/22/09
ASUS P5BV-M LGA 775 Intel 3200 Micro ATX Intel Xeon/Core 2 Server Motherboard
CPU INTEL|C2Q Q6600 2.40G 775 8M
Western Digital Caviar Black WD6401AALS 640GB 7200 RPM 32MB--2 total
MEM 2Gx2|CORSAIR VS4GBKIT800D2--8GB total
DVD BURN SAMSUNG|SH-S223F 22X
So I got it all assembled and my first obsticale was a chassis error and then I recieved a fatal error and the system wouldn't finish the POST, and I couldn't get into the BIOS. After about 30 minutes I finally figured out that I had to put jumpers over the chassis open connector on the MB and that fixed the issue (the board came with only one jumper over 2 of the 4 pins). During that process I cleared the CMOS.
Went into the BIOS and set the storage controller to ACHI and looked at the MAXCPU ID, which was set to disabled.
Was able to boot and the ESXi 4 booted off the CD. It loaded all of the .tgz files and then went into its initial install. It showed my proc and the 8GB of mem. At random times during the initialization of different components (vmkernal, vm scheduler, tpm, etc) it will give me a PSOD, sometimes with the same message, and other times with different ones. Most were #pf exception (14) in world ####. The world number would change.
So I got all of the same hardware except the drives and DVD, and I added an extra 4GB of mem. Could either of those things be the cause? Was something reset in the BIOS when I reset the CMOS that I now need to change?
My next step is to try ESXi 3.5 and see if that will install without PSODs. Any ideas you may to resolve this issue would be greatly apperciated. If there is any more information you need let me know. I am eager to get this up and running. Thanks!!!
So I bought most of the same hardware as you except for the Seagate hard drives and the dvd drive because they were no longer available on newegg. Here is the ESXi 4 version that I got and the hardware:
ReplyDeleteESXi--4.0.0 | 171294 - 06/22/09
ASUS P5BV-M LGA 775 Intel 3200 Micro ATX Intel Xeon/Core 2 Server Motherboard
CPU INTEL|C2Q Q6600 2.40G 775 8M
Western Digital Caviar Black WD6401AALS 640GB 7200 RPM 32MB--2 total
MEM 2Gx2|CORSAIR VS4GBKIT800D2--8GB total
DVD BURN SAMSUNG|SH-S223F 22X
So I got it all assembled and the my first obsticale was a chassis error and then I recieved a fatal error and the system wouldn't finish the POST, and I couldn't get into the BIOS. After about 30 minutes I finally figured out that I had to put jumpers over the chassis open connector on the MB and that fixed the issue (the board came with only one jumper over 2 of the 4 pins). During that process I cleared the CMOS.
Went into the BIOS and set the storage controller to ACHI and looked at the MAXCPU ID, which was set to disabled.
Was able to boot and the ESXi 4 booted off the CD. It loaded all of the .tgz files and then went into its initial install. It showed my proc and the 8GB of mem. At random times during the initialization of different components (vmkernal, vm scheduler, tpm, etc) it will give me a PSOD, sometimes with the same message, and other times with different ones. Most were #pf exception (14) in world ####. The world number would change.
So I got all of the same hardware except the drives and DVD, and I added an extra 4GB of mem. Could either of those things be the cause? Was something reset in the BIOS when I reset the CMOS that I now need to change?
My next step is to try ESXi 3.5 and see if that will install without PSODs. Any ideas you may to resolve this issue would be greatly apperciated. If there is any more information you need let me know. I am eager to get this up and running. Thanks!!!
-Josh
Oh man, my bad. I am living overseas and the message that came back stating the comment was waiting for approval was in Arabic, so i didn't know it was waiting for approval. Sorry about posting the same thing 3 times.
ReplyDelete