Introduction to ARM Servers

"Intel does not have any competition whatsoever in the midrange and high-end (x86) server market". We came to that rather boring conclusion in our review of the Xeon E5-2600 v2. That date was September 2013.

At the same time, the number of announcements and press releases about ARM server SoCs based on the new ARMv8 ISA were almost uncountable. AppliedMicro was announcing their 64-bit ARMv8 X-Gene back in late 2011. Calxeda sent us a real ARM-based server at the end of 2012. Texas Instruments, Cavium, AMD, Broadcom, and Qualcomm announced that they would be challenging Intel in the server market with ARM SoCs. Today, the first retail products have finally appeared in the HP Moonshot server.

There has been no lack of muscular statements about the ARM Server SoCs. For example, Andrew Feldman, the founder of micro server pioneer Seamicro and the former head of the server department at AMD stated: "In the history of computers, smaller, lower-cost, and higher-volume CPUs have always won. ARM cores, with their low-power heritage in devices, should enable power-efficient server chips." One of the most infamous silicon valley insiders even went so far as to say, "ARM servers are currently steamroller-ing Intel in key high margin areas but for some reason the company is pretending they don’t exist."

Rest assured, we will not stop at opinions and press releases. As people started talking specifications, we really got interested. Let's see how the Cavium Thunder-X, AppliedMicro X-Gene, Broadcom Vulcan, and AMD Opteron A1100 compare to the current and future Intel Server chips. We are working hard to get all these contenders in our lab, and we are having some success, but it is too soon for a full blown shoot out.

Micro Servers and Scale-out Servers

Micro servers were the first the target of the ARM licensees. Typically, a discussion about Micro servers quickly turns into a wimpy versus brawny core debate. One of the reasons for that is that Seamicro, the inventor of the micro server, first entered the market with Atom CPUs. The second reason is that Calxeda, the pioneer of ARM based servers, had to work with the fact that the Cortex-A9 core was a wimpy core that could not deal with most server workloads. Wikipedia also associates micro servers with very low power SoCs: “Very low power and small size server based on System-on-Chip, typically centered around ARM processor”.

Micro servers are typically associated with low end servers that serve static HTML, cache web objects, and/or function as slow storage servers. It's true that you will not find a 150W high-end Xeon inside a micro server, but that does not mean that micro servers are defined by low power SoCs. In fact, the most successful micro servers are based on 15-45W Xeon E3s. Seamicro, the pioneer of micro servers, clearly indicated that there was little interest in the low power Atom based systems, but that sales spiked once they integrated Xeon E3s.

Currently micro servers are still a niche market. But micro servers are definitely not hype; they are here to stay, although we don't think they will be as dominant as rack servers or even blade servers in the near future. To understand why we would make such a bold statement, it is important to understand the real reason why micro servers exist.

Let us go back to the past decade (2005-2010). Virtualization was (and is) embraced as the best way to make enterprises with many heterogeneous applications running on underutilized servers more efficient. RAM capacity and core counts shot up. Networking and storage lagged but caught up – more or less – as flash storage, 10 Gbit Ethernet, and SRIOV became available. But the trend to notice was that virtualization made servers more I/O feature rich: the number and speed of network NICs and PCI-e expansion slots for storage increased quickly. Servers based on the Xeon E5 and Opterons have become "software defined datacenters in a box" with virtual switching and storage. The main driver for buying complex servers with high processor counts and more I/O devices is simple: professionals want the benefits that highly integrated virtualization software brings. Faster provisioning, high availability (HA), live migration (vMotion), disaster recovery (DR), keeping old services alive (running on Windows 2000 for example): virtualization made everything so much easier.

But what if you did not need those features because your application is spread among many servers and can take a few hardware outages? What if you do not need the complex hardware sharing features such as SRIOV and VT-d? The prime example is an application like Facebook, but quite a few smaller web farms are in a similar situation. If you do not need the features that come with enterprise virtualization software, you are just adding complexity and (consultancy/training) costs to your infrastructure.

Unfortunately, as always, the industry analysts came with unrealistic high predictions for the new micro server market: in 2016, they would be 10% of the market, no less than "a 50 fold jump"! The simple truth is that there is a lot of demand for "non-virtualized" servers, but they do not all have to be as dense and low power as the micro servers inside the Boston Viridis. The "very low power", extremely dense micro servers with their very low power SoCs are not a good match for most workloads out there, with the exception of some storage and memcached machines. But there is a much larger market for servers denser than the current rack servers, but less complex and cheaper than the current blade servers, and there's a demand for systems with a relatively strong SoC, currently the SoCs with a TDP in the 20W-80W range.

Not convinced? ARM and the ARM licensees are. The first thing that Lakshmi Mandyam, the director of ARM servers systems at ARM, emphasized when we talked to her is that ARM servers will be targeting scale-out servers, not just micro servers. The big difference is that micro servers are using (very) low power CPUs, while scale-out servers are just servers that can run lots and lots of threads in parallel.

The Current Intel Offerings
Comments Locked

78 Comments

View All Comments

  • jhh - Tuesday, December 16, 2014 - link

    SPARC and Power have had trouble keeping up with Moore's law, as neither sold enough to amortize R&D to push out innovation at the same rate as Intel. As Moore's law comes to an end, this will stop being a unique Intel advantage. It just might be too late for both of them. One can see the pressure on IBM, with their opening the Power architecture in similar ways to ARM. Both POWER and SPARC have to keep up to porting drivers to their Unix implementations, while the device manufacturers either write drivers for Linux or don't get volume. I just can't see either POWER or SPARC being cost effective over the long run. And, when others see the same thing, they aren't going to be excited about porting application software to those platforms.

    ARM needs to have a good performance/power and performance/cost ratio to get people excited to buy something other than Intel. They are certainly getting enough volume from the low-end to make investment on high-end parts. So far, I'm not excited enough to recommend any ARM proof-of-concept though.
  • Kevin G - Wednesday, December 17, 2014 - link

    IBM always had a licensing model similar to ARM with PowerPC cores. The only thing really new here is that IBM is licensing out there flagship POWER chip in the same manner. Despite Intel having a process advantage, IBM was able to keep up in performance. (The 45 mm based 8POWER7 was generally faster than the 32 mm 10 core Westmere-EX.) There will always be a market for top performance but you are correct that sustaining on just that customer base is unwise.

    IBM does realize that their software licensing model to subsidize hardware R&D was not sustainable. So while you can't run AIX, you can get a POWER8 box for less than $3k now.
  • OreoCookie - Wednesday, December 17, 2014 - link

    Really, just $3000? Wow, how times have changed, I remember ~12 years ago that a single Alpha CPU cost that much (the department I was working for had a workstation fail, fortunately under warranty, because otherwise they would have had to pay for 2 new CPUs and new RAM worth about 15,000 German Marks).
  • Ratman6161 - Wednesday, December 17, 2014 - link

    "The general lower cost of Linux and open source software" While it's true that the cost of a Linux OS including support is lower than an equivalent Windows OS, in the larger scheme of things the cost of Windows and even VMware becomes little more than background noise in the total cost of operations. Try pricing out an Oracle DB for example and you find that the cost of that software dwarfs the price of the hardware it's running on as well as whatever the OS is costing. Ditto with most "enterprise software".
  • lefty2 - Tuesday, December 16, 2014 - link

    Intel has another big advantage over ARM, which everyone seems to have forgotten about, and that is software compatibilty. 64-bit ARM server software is still a work in progress. The stuff that's being worked on at the moment is open source. Once that's finished you still have to convince clients to convert their proprietary software to ARM.
  • JohanAnandtech - Tuesday, December 16, 2014 - link

    Don't you think that the open source software that has been/is ported now is enough? Apache/PHP/MySQL, Memcached and Hadoop...that is a massive server market. And there is little stopping Microsoft to invest in ARM software too. Just VMware might be a bit tricky, but I don't think the software is a problem.
  • Kevin G - Wednesday, December 17, 2014 - link

    Actually VMware has said some less that flattering about ARM. Xen is the main hyper visor on ARM for the moment.
  • goop666666 - Thursday, December 25, 2014 - link

    Yeah, recompiling is so very hard. Essentially what you're saying is that Intel is for legacy systems and software that is poorly written. That is a large enough market, but doesn't apply to hyperscale deployments, which are the future.
  • gostan - Tuesday, December 16, 2014 - link

    great article by Johan as always.

    but the argument is muted. we have heard this tune before.

    the hardware might be cheaper. the power bill might be cheaper. wait until you see the software maintenance cost. custom software needs 'custom' pricing.

    besides, arm has no cutting edge fab process to back them.
  • JohanAnandtech - Tuesday, December 16, 2014 - link

    You do not need expensive software to create a server market these days. Just look how many webservers are running the LAMP stack.

Log in

Don't have an account? Sign up now