Part 1: The NUC killer: SuperMicro SuperServer E300-8D
I bought two SuperMicro SuperServer E300-8D a few months ago and they only seems to get better as I get to know them better. The SuperMicro SuperServer E300-8D is replacing my old lab, which was excessively small and inadequate. It was based on Dell PCs, which was limited to only 8GB of memory. With 8GB of memory, there is not much I can get up and running in a state where I was happy with it. Therefore, I was looking for something else. When I started my search, somewhere mid last year, Intel’s NUC was the only real option. Nevertheless, no good option if you ask me. It only supported 32GB of memory and did not have 10Gb/s network cards. I wanted something that could take my lab closer to the enterprise environment I am used to working in. Then came along two SuperServers…
SuperServer v. NUC
Before I dig it to the two SuperMicro SuperServers. Let me settle why I think this is the NUC killer. First of IMHO there is no comparison between the two server types. The NUC is very limited and needs workarounds to get to a somewhat working and supported state. Nevertheless, let me first go over the specification difference.
|CPU||Intel Xeon D-1518 2,2Ghz||Intel Core i5-6260U 1,80GHz||Intel Core i7-6770HQ 2.60GHz|
|Cores/Threads||4 Cores / 8 Threads||2 Cores / 4 Threads||4 Cores / 8 Threads|
|CPU Passmark score||4700||4380||9684|
|Memory||128 GB||32 GB||32 GB|
|Network||6x1Gbe + 2x10Gb SFP+||1x1Gbe||1x1Gbe|
|Disk||4x SATA3 + 1x M.2 + 1x mSATA||1x M.2 + 1x SDXC||2x M.2 + 1x SDXC|
|Peripheral||2x USB 3.0||4x USB 3.0 + 2x USB 2.0||6x USB 3.0 + 2x USB 2.0 + Thunderbolt or USB 3.1 Gen2|
|Expansion||2x PCIe (No Space for HHHL – One can be used with riser)||NO||NO|
|Price (Local prices)||DKK5718 / €769 / $814||DKK3080 / €411 / $439||DKK4825 / €643 / $687|
So the E300-8D is clearly priced higher than any Intel NUC, but for a small premium you get a system that supports up to 128GB memory, have multiple 1Gbe and dual 10Gb SFP+ network interfaces and as a bonus, IPMI and multiple storage options, though there is no room in the case for all the storage options being used simultaneously. If you want a more potent (Passmark score of 8612 – 6c/12t) option, you can go for the E200-8D, which will cost you around DKK7135 / €959 / $1016 and comes in a smaller package at the price of some ethernet ports and expansion options. If you want a clearer comparison of SuperServer vs NUC take a look at Tai Ratcliffs blog post “SuperMicro vs Intel NUC”. Highly recommended.
Every choice comes at a price
SuperMicro put out two SuperServers in a small form factor. Both 1U cases, but with some differences. The hard choice quickly became between the SuperMicro SuperServer E200-8D or E300-8D, but what was the key differences. Well just be looking at the two servers it was obvious that the E300-8D was wider than the E200-8D, but that was not all. Let me quickly run through the list.
|CPU||Intel Xeon D-1528 1,9Ghz||Intel Xeon D-1518 2,2Ghz|
|Cores/Threads||6 Cores / 12 Threads||4 Cores / 8 Threads|
|Memory||4x DDR4 – up to 128GB ECC||4x DDR4 – up to 128GB ECC|
|DISK||6x SATA3 ports (2x support SATADOM)|
1x M.2 (PCI-E 3.0 x4) M key (42 to 110mm)
|4x SATA3 ports (2x support SATADOM)|
1x M.2 (PCI-E 3.0 x4) M key (42 to 110mm)
1x mSATA/Mini PCI-E 2.0 x1
|Network||Dual 10Gbe – Intel SoC|
Dual 1Gbe – Intel I350-AM2
|Dual 10G SFP+ – Intel SoC|
Dual 1Gbe – Intel I210
Quad 1Gbe – Intel I350-AM4
|Expansion options||none||2x PCI-E 3.0 x8 slots (Of which only one is accessible via riser card)|
|IPMI||1 port||1 port|
|USB||2x USB 3.0 ports||2x USB 3.0 ports|
Looking at the list there are three things, which differentiates them from each other.
First there is CPU – E200-8D has more cores and threads than E300-8D.
When it comes to storage they might look almost like each other, but there are key things that sets the E300-8D apart. mSATA port is one thing. I have not used to yet, but I am looking into buying a Samsung 850 EVO mSATA as a capacity tier disk.
More important is it that the E300-8D has two PCI-E expansion card slots. Admittedly, there is a problem with this, as the case is not high enough to accommodate two HHHL PCI-E cards. The solution SuperMicro brings to the table is to use a riser card, but that leaves you with the option to use only one of the PCI-E expansion slots. I am contemplating if I could 3D print a new case lit which could give me the headroom to be able to use the two PCI-E cards and at the same time, it could double as a case, which has room for HDD bays. That would make a vSAN home lab much more fun. Not to talk about all the possible configuration options. Still have some InfiniBand cards lying around somewhere, which could come to good use.
RJ45 vs SFP+
This was the biggest decider for me. Not that the interface technology as such matters, but the price of network switch and cables might have an impact on your decisions. For me at least, the price was what made the E300-8D a no brainer! Let me set the scene. There are only so many 10Gbit Ethernet switches on the market and they come in two flavors. The de facto standard for Ethernet RJ45 or SFP+ which gives you a lot of options in terms of carrier media. There are three price factors that comes into play here. Compute, Switch and cable prices.
Let me first take the route of RJ45 connectivity options. Compute is simple in the case of Supermirco SuperServers the option is E200-8D. Then there is cabling – cheap ethernet Base-T cables with RJ45 connectors. Lastly, there is the option of switch.
Here local availability has a saying, but more importantly IMHO brand and series speaks louder than anything when dealing with network gear. I have had and still have some gear from network vendors such as D-link, Netgear, Linksys etc. essentially the “who is who” of cheap networking . These semi cheap brands that promises more than they can hold up to. I AM DONE WITH THEM! LIFE IS TOO SHORT!
Just so, we are clear there are more on that list as an example anything Cisco small business is there was well. I just have a poor track record with a lot of them. I do not think I am being hard or biased here – My requirements for such devices are simple. Stability first and that is it! They all fail at being stable under load or otherwise. That is just not good enough. So now you know why the price pretty quickly goes up. So what are the options then.
Well enterprise gear is out of the question as the cheapest options comes in at around DKK40000 / €5379 / $5697 for either Dell, HPE or Netgear. In this price range for Cisco, it is still SMB gear. Therefore, with enterprise gear ruled out, I have to circle back and look at SMB gear, even though I am not happy with that. Prices start at around DKK6000 / €809 / $855. Looked at what others was recommending and there seems to be an agreement amongst the vExpert community on twitter that Cisco’s SG350XG is the switch to buy. It comes in at twice as much as the cheapest switch.
|Model||Price for one|
|Compute||E200-8D||DKK7135 / €959 / $1016|
|Cables||Ethernet RJ45 3M/10FT||DKK53 / €7 / $8|
|Switch||SG350XG-2F10||DKK11983 / €1611 / $1707|
|Total||DKK19171 / €2578 / $2730|
The above prices are just for one of each and excluding memory and storage, which would be the same for either E200-8D or E300-8D. A more realistic bill for redundant wiring and two or three E200-8D will put the price (excluding memory and storage) to DKK26571 / €3573 / $3785 or DKK33865 / €4554 / $4823. If you want network switch redundancy, well just at another switch to the setup and see the price skyrocket once again.
The big question is then, what did I fork out to get a new home lab?
Let me start by defining the network side. I wanted enterprise gear, but as you now know, I was not up for actually paying for enterprise network equipment. Over the last year or so I have grown very fund of Ubiquiti. Started out as a flirt with their access-points and ended up getting some more of Ubiquiti’s equipment, as the stability is great and so is performance and management side of things.
So guess what! I bought a Ubiquiti 10Gbit switch, named ES-16-XG (today also sold as US-16-XG. Which is part of the Unifi series that has a simple and easy to use management GUI). It has 12x SFP+ ports and 4x 10Gbe. Perfect for what I need. I does layer three, and I have not found any issues with the specifications list so far. However, I have not used any advanced features so far, so I guess a really review is in order once I start fiddling with it.
Next up is cabling – Some may think that a SFP+ optics are expensive or that cable with SFP+ connector are equally as expensive… and they would be right. At least in local terms. One of the reasons is that the Ubiquiti ES-16-XG is a little picky about which SFP+ optics or cables it will work with. The Ubiquiti ES-16-XG is a wannabe enterprise diva, I guess.
Over at ServeTheHome.com, there is a thread about which cables and SFP+ optics are working and which are not. You can look it up here. I found a DAC (Direct Attached Cable), so I did not have to fiddle with optics and cables. I just needed simple and working. The Dell 2CM32 3m DAC seemed to be the cheapest. I looked at local suppliers and the bill per cable would be around DKK6000 / €800 / $850. Ouch!
A quick search on Ebay and I had found a supplier in the UK, which could do it, much cheaper! I skipped paying via Ebay and went directly to their site and got it even cheaper! These cable are Twinax cables and I know some of you may not like Twinax in your datacenters, but this is not a datacenter I am building merely a home lab and the cables will do just fine.
|Switch||Ubiquiti ES-16-XG||DKK5088 / €685 / $725||computersalg.dk|
|Cable||Dell 2CM32 3m DAC||DKK310 / €42 / $44||itinstock.com|
I bought eight cables, because the price was so good compared to what I else would have to pay. So the total price for networking came to be DKK7568 / €1018 / $1078. Not bad if you ask me. Not more expensive or cheaper than the cheapest on market.
Let me just get straight to it. This build is not the cheapest one could do. You could probably shave some of the price if you compromise somewhere. It could be on capacity or performance. I did reevaluate my need doing this search for the right home lab. Started out looking at Intel only storage, such as the P3700 and the 750 series of PCI-E cards, but ended up buying a 1TB Samsung 960 Pro M.2 SSD.
|Compute||SuperMicro SuperServer SYS-E300-8D||DKK5718 / €769 / $814||plexcom.de|
|Memory||32 GB DDR4 Kingston ECC 2400 MHz (KVR24R17D4/32MA)||DKK2046 / €275 / $291||bj-trading.dk|
|SATADOM||SuperMicro SuperDOM 64GB (SSD-DM064-PHI)||DKK862 / €116 / $123||Computersalg.dk|
|M.2 Storage||SAMSUNG 960 PRO 1TB||DKK3979 / €535 / $567||Dustin.dk|
|Total||DKK12605 / €1695 / $1795|
The quick ones reading this, will notice the lack of capacity storage. You would be right. If you are planning to use any solutions based of storage convergence, capacity tier should be added. Currently I am using what I have at my disposition, small SSDs or 2TB 3,5” HDD. Both of which does not really fit within the case of the E300-8D. As I stated earlier the Samsung 850 EVO mSata does look like a good option for capacity and the speed penalty of mSata and EVO drive should not be a problem. At a price around DKK2500 / €335 / $355 for a 1TB drive, it is also much cheaper than the Samsung 960 Pro. Well I guess one is never truly done building a home lab…
This concludes this blog post. More detailed blog posts will come, when I find the time, on the components discussed here. So look forward to a deep dive on E300-8D, storage components and the Ubiquiti ES-16-XG switch. Before I leave you to decide what is in your coming home lab. I have summarized what I find as the good, the bad and the ugly of the SuperMicro SuperServer E300-8D. Thank for reading this far!
- M.2 NVMe (SATA support stated in documentation)
- Cheap way to 10Gbit network
- Loads of 1Gbe ports
- Up to 128GB of Memory
- Supports SATADOM (Two if you need)
- Extra licenses for a full featured IPMI solution
- Only two USB ports (Cannot do USB dongle, mouse and keyboard)
- Noisy compared to fan-less NUCs
And the ugly
- Only room for one 2.5” drive
- PCI-E ports cannot both be used (U1 is not high enough for a half height PCI-E card – 2U Case option would be nice)