Hardware

Homemade Cluster Hardware.

  • January 2nd 2017
  • 9 min read

Introduction.

This morning, I decided to build a real cluster for my researchs. Why?

In 2012 I started to working projects related with bigdata technologies and from 2014 in 100% bigdata projects.

In all this time I have had the necessity to test the performance in real clusters. In my last project, I am working 100% with AWS, so we create and destroy clusters continuously without buy any type of hardware.

But what about my personal projects? The Cloud and AWS is a fantastic tool to do it, but:

  1. I am not a company with a big budget, so I can’t expend hundreds of euros per month only for enjoy. If I do it, my wife kill me.
  2. I am a romantic, so I prefer hardware.

Target.

With the proliferation of new card-sizes single-boards computers, there are a lot of people that is starting to create clusters using this kinds of cheap computers. But is really the best option?

In this post, I am going to try to build a cluster with a budget of €500 (around $520/£426)

To do it, I am going to compare:

I discarded other computers focus in robotics so they don’t have enough memory, performance, etc. Also, I excluded solutions that exceeded €100, because I want buy at least 5 computers.

It isn’t going to be a production environment, but the idea is to use the cluster for distributed and big data application.

So I am going to analysis the following list of  points:

  • CPU / GPU: The performance per thread and the number of cores is important.

  • Memory: It’s important to have enough memory per every job/task executed or node working. 1G of RAM is not a lot, but it is the minimum requirement that I used to exclude a lot of card size computers. Different use cases need different amount of memory. For example:

    • For Elasticsearch cluster to index the OpenStreetMap full planet network, we need a lot of memory.
    • For a Spark or AKKA cluster to process the OpenStreetMap full planet to create a network, where every paralleling block size is around 32M of ram, we don’t need a lot of memory.

  • Storage: Always important to store temporal data, also important if the cluster is a storage system like Hdfs, Hbase, Cassandra or Elasticsearch. The minimum requirement will be 32GB and at least 30MB/s.
    Maybe, the best option will be to use an external USB pendrive.
    I am going to consider few different options, comparing the transfer rate in MB/s:

    • microSD write speed (read is faster)

      • UHS-II Speed Class 3 : theoretical maximum rate of 240Mbps (30MB/s)
      • UHS-I Speed Class 1 :  theoretical maximum rate of 80Mbps (10MB/s)
      • Speed Class 10 : theoretical maximum rate of 80Mbps (10MB/s)
      • Speed Class 6 : theoretical maximum rate of 48Mbps (6MB/s)
      • Speed Class 4 : theoretical maximum rate of 32Mbps (4MB/s)
      • Speed Class 2 : theoretical maximum rate of 16Mbps (2MB/s)

    • USB for external drivers:

      • USB 2.0 theoretical maximum rate of 480 Mbps (60MB/s)
      • USB 3.0 theoretical maximum rate of 5 Gbps (640MB/s)
      • USB 3.1 theoretical maximum rate of 10 Gbps (1280MB/s)

    • Thunderbolt theoretical maximum rate of 10 Gbps (1280MB/s)

    • eMMC: Few times faster than the SD.

    • eSATA: theoretical maximum rate of 6 Gbps (768MB/s)

  • Network: I think that 1Gbps Ethernet is the minimum requirement.
  • OS: I need to be able to execute a full distribution of Linux. I prefer Ubuntu.
  • Power: Different systems need different power supply. It is better if the consume is low and it is not necessary one power supply per node.
  • Physical space for allocation: I don’t want my office full of hardware without cases!
  • Price: Of course, one of the must important.

I am going to exclude from the comparison all components not related directly with the CPU. For example, USB HUB, Ethernet Switch and cables, etc.

Comparison

Rasperry Pi 3Odroid XU4Odroid C2Banana Pi M64PINE A64+ 2GBMini PCUP Squared
CPU1.2GHz 64 Bit Quad-Core ARMv8 Cortex A53Samsung Exynos-5422 : Cortex™-A15 and Cortex™-A7Amlogic ARM® Cortex®-A53(ARMv8) 1.5Ghz1.2GHz Allwinner 64 Bit Quad-Core ARM Cortex A531.2 Ghz 64-Bit Quad-Core ARM Cortex A53AMD APU Sempron 3850 Quad CoreCeleron™ 2C 2.4 GHz N3350
Memory1GB SDRAM LPDDR2 @ 900 MHz2GB LPDDR32Gbyte DDR3 SDRAM2GB LPDDR32GB LPDDR34GB DDR32GB LPDDR 4
GPUBroadcom VideoCore IV 3DMali-T628 MP6Mali™-450 GPUDual core Mali 400 MP2Dual core Mali 400 MP2AMD Radeon™ RIntel Gen 9 HD<
Video OutputHDMI 1.3HDMI 1.4aHDM 2.0HDMI 1.4HDMI 1.4aHDMI1x Full HDMI 1.4b – 4K @ 30 hz, 1x DP 1.2 – 4K @ 60 hz
microSDMicroSDHC (up to 32 GB)MicroSDHC ??MicroSDHC ??microSDXC ( up to 256GB )microSDXC ( up to 256GB )
USBUSB 2.0 PORT (x4)USB 3.0 (x2) USB 2.0 (x1)SB 2.0 PORT (x2) USB OTG (x1)USB 2.0 PORT (x2) USB OTG (x1)USB 2.0 PORT (x2)USB 3.0 (x2) USB 2.0 (x4)USB 3.0 Type A(x3) USB OTG Micro B (x1) USB 2.0 (x2)
eMMCeMMC5.0eMMC5.08G-32GB eMMC
SATA SSD--250GB SATA3 (SATA 6Gb/s)Full-size SATA3 (SATA 6Gb/s)
Ether100Mbit/s1Gbit/s1Gbit/s1Gbit/s1Gbit/s1Gbit/s1Gbit/s x2 dedicated
SORaspbian,Ubuntu,Android Things, etc.Android and LinuxUbuntu 16.04 or Android 5.1 Lollipop based on Kernel3.14LTSAndroid and LinuxAndroid and LinuxWindows / LinuxWindows / Linux
PowerRec. 5V@2.5A Power rating: 800 mA (4.0 W)5V@4A input via DC Power5V@2A via DC Power and/or Micro USB (OTG)5V@2A via DC Power and/or Micro USB (OTG)5V@2A via DC Power and/or Micro USB (OTG)5V@4A via DC Power and/or Micro USB (OTG)
Size85.60x56.5mm82x58mm82x56mm92x60mm127x79mm285x88x320mm86x90mm
Board priceUS$35US$74US$40US$74US$29.00-
CaseUS$12-US$10
StorageUS$15 - MicroSDHC Memory Card 32 GB Class 10 UHS-1 U1--
Total PriceUS$62US$101US$67US$101US$56US$168US$168
Total Price€59€96€64€96€53€160€160
Total Price£50£82£54£82£46£137£137

Rasperry Pi 3

Official site: www.raspberrypi.org/ Wiki: https://en.wikipedia.org/wiki/Raspberry_Pi​

It could be the most famous card-size computer, with the biggest community.

The last version in this moment is the Raspberry Pi 3, with 1.2GHz 64-bit quad-core ARMv8 Cortex-A53 CPU, a VideoCore IV 3D GPU and only 1GB SDRAM @ 400 MHz it is not the more powerful computer in the list compared.

Also, a big handicap is the 100Mbps Ethernet and the storage system.

Other extra features: 40 pins GPIO, 802.11n wireless and bluetooth 4.1

Banana Pi M64

Official site:https://www.banana-pi.org/m64.html Wiki: https://en.wikipedia.org/wiki/Banana_Pi#Banana_Pi_M64

This computer board has the same 1.2GHz 64-bit quad-core ARMv8 Cortex-A53 CPU and an equivalent Dual core Mali 400 MP2 GPU.

The big difference is the 1Gbps ethernet, the 2GB LPDDR3 RAM the microSDXC and the 8GB eMMC included in the board.

The problem, the price. It’s more than the double of the Raspberry Pi 3. So per every Banana I can buy two Raspberries :)

Pine A64+ 2GB

Official Site: https://www.pine64.org/?page_id=1194 Wiki:https://en.wikipedia.org/wiki/Pine64 This computer board has the same 1.2GHz 64-bit quad-core ARMv8 Cortex-A53 CPU and an equivalent Dual core Mali 400 MP2 GPU.

The big difference is the 1Gbps ethernet, the 2GB LPDDR3 RAM and the microSDXC.

The price is really good. Better than the Raspeberry and cheaper: $30

The bad part, no wifi or bluetooth without an extra $10. Also, old HDMI 1.4

Odroid XU4 and C2

Official site: http://www.hardkernel.com

Wiki: https://en.wikipedia.org/wiki/ODROID

Both are good alternatives.

Both with 2GB Ram, 1GBit/s Ethernet eMMC5.0 that is faster (expensive and difficult to find) than a microsd.

XU4 has a older CPU with a 32bits architecture, but the performance is good enough. Also, has a USB 3.0 port, so you can use really cheap 128GB 100MB/s scandisk pendrives. The problem is that it is one of the more expensive boards.

C2 is the newest one, with new A53 CPU with a 64bits architecture but with USB 2.0 ports.

Mini ITX homemade cheap PCs

Selected configuration: https://uk.pcpartpicker.com/user/angelcervera/saved/6xfmGX

It is going to be more powerful than any other combination, but I can not buy more than 3 servers.

In this field there are a lot of option, from £65 with only 2GB of RAM and 32GB of ROM to £140 with 4GB of RAM and 60GB SSD SATA3:

UP Squared

Site: http://www.up-board.org/upsquared/specifications-up2

The specification are really good. All advantages of a Intel PC and all advantages of a card-size board.

But this comparison it to find a low cost server (no for robotics) and at the end, the price is the same that a Mini ITX.

Conclusion.

Really, there is not a winner.

In my opinion:

  • ​If you have enough money or your wife does not manage it: A custom miniPC is the best option.
  • If you don’t want expend a lot, Pine A64+ 2G with a good microSD car is the best option.

Other conclusion is that Rasberry Pi is the more important in the market with the biggest community, but looks that it is not the best relation price/specs.

Reference: