Homemade Cluster Hardware.
This morning, I decided to build a real cluster for my researchs. Why?
In 2012 I started to working projects related with bigdata technologies and from 2014 in 100% bigdata projects.
In all this time I have had the necessity to test the performance in real clusters. In my last project, I am working 100% with AWS, so we create and destroy clusters continuously without buy any type of hardware.
But what about my personal projects? The Cloud and AWS is a fantastic tool to do it, but:
- I am not a company with a big budget, so I can’t expend hundreds of euros per month only for enjoy. If I do it, my wife kill me.
- I am a romantic, so I prefer hardware.
With the proliferation of new card-sizes single-boards computers, there are a lot of people that is starting to create clusters using this kinds of cheap computers. But is really the best option?
In this post, I am going to try to build a cluster with a budget of €500 (around $520/£426)
To do it, I am going to compare:
- Rasperry Pi 3
- ODROID XU4
- ODROID C2
- Banana Pi M64
- PINE A64+ 2GB
- mini ITX homemade cheap PCs
- UP Squared
I discarded other computers focus in robotics so they don’t have enough memory, performance, etc. Also, I excluded solutions that exceeded €100, because I want buy at least 5 computers.
It isn’t going to be a production environment, but the idea is to use the cluster for distributed and big data application.
So I am going to analysis the following list of points:
CPU / GPU: The performance per thread and the number of cores is important.
Memory: It’s important to have enough memory per every job/task executed or node working. 1G of RAM is not a lot, but it is the minimum requirement that I used to exclude a lot of card size computers. Different use cases need different amount of memory. For example:
- For Elasticsearch cluster to index the OpenStreetMap full planet network, we need a lot of memory.
- For a Spark or AKKA cluster to process the OpenStreetMap full planet to create a network, where every paralleling block size is around 32M of ram, we don’t need a lot of memory.
Storage: Always important to store temporal data, also important if the cluster is a storage system like Hdfs, Hbase, Cassandra or Elasticsearch. The minimum requirement will be 32GB and at least 30MB/s.
Maybe, the best option will be to use an external USB pendrive.
I am going to consider few different options, comparing the transfer rate in MB/s:
microSD write speed (read is faster)
- UHS-II Speed Class 3 : theoretical maximum rate of 240Mbps (30MB/s)
- UHS-I Speed Class 1 : theoretical maximum rate of 80Mbps (10MB/s)
- Speed Class 10 : theoretical maximum rate of 80Mbps (10MB/s)
- Speed Class 6 : theoretical maximum rate of 48Mbps (6MB/s)
- Speed Class 4 : theoretical maximum rate of 32Mbps (4MB/s)
- Speed Class 2 : theoretical maximum rate of 16Mbps (2MB/s)
USB for external drivers:
- USB 2.0 theoretical maximum rate of 480 Mbps (60MB/s)
- USB 3.0 theoretical maximum rate of 5 Gbps (640MB/s)
- USB 3.1 theoretical maximum rate of 10 Gbps (1280MB/s)
Thunderbolt theoretical maximum rate of 10 Gbps (1280MB/s)
eMMC: Few times faster than the SD.
eSATA: theoretical maximum rate of 6 Gbps (768MB/s)
- Network: I think that 1Gbps Ethernet is the minimum requirement.
- OS: I need to be able to execute a full distribution of Linux. I prefer Ubuntu.
- Power: Different systems need different power supply. It is better if the consume is low and it is not necessary one power supply per node.
- Physical space for allocation: I don’t want my office full of hardware without cases!
- Price: Of course, one of the must important.
I am going to exclude from the comparison all components not related directly with the CPU. For example, USB HUB, Ethernet Switch and cables, etc.
|Rasperry Pi 3||Odroid XU4||Odroid C2||Banana Pi M64||PINE A64+ 2GB||Mini PC||UP Squared|
|CPU||1.2GHz 64 Bit Quad-Core ARMv8 Cortex A53||Samsung Exynos-5422 : Cortex™-A15 and Cortex™-A7||Amlogic ARM® Cortex®-A53(ARMv8) 1.5Ghz||1.2GHz Allwinner 64 Bit Quad-Core ARM Cortex A53||1.2 Ghz 64-Bit Quad-Core ARM Cortex A53||AMD APU Sempron 3850 Quad Core||Celeron™ 2C 2.4 GHz N3350|
|Memory||1GB SDRAM LPDDR2 @ 900 MHz||2GB LPDDR3||2Gbyte DDR3 SDRAM||2GB LPDDR3||2GB LPDDR3||4GB DDR3||2GB LPDDR 4|
|GPU||Broadcom VideoCore IV 3D||Mali-T628 MP6||Mali™-450 GPU||Dual core Mali 400 MP2||Dual core Mali 400 MP2||AMD Radeon™ R||Intel Gen 9 HD<|
|Video Output||HDMI 1.3||HDMI 1.4a||HDM 2.0||HDMI 1.4||HDMI 1.4a||HDMI||1x Full HDMI 1.4b – 4K @ 30 hz, 1x DP 1.2 – 4K @ 60 hz|
|microSD||MicroSDHC (up to 32 GB)||MicroSDHC ??||MicroSDHC ??||microSDXC ( up to 256GB )||microSDXC ( up to 256GB )|
|USB||USB 2.0 PORT (x4)||USB 3.0 (x2) USB 2.0 (x1)||SB 2.0 PORT (x2) USB OTG (x1)||USB 2.0 PORT (x2) USB OTG (x1)||USB 2.0 PORT (x2)||USB 3.0 (x2) USB 2.0 (x4)||USB 3.0 Type A(x3) USB OTG Micro B (x1) USB 2.0 (x2)|
|SATA SSD||-||-||250GB SATA3 (SATA 6Gb/s)||Full-size SATA3 (SATA 6Gb/s)|
|Ether||100Mbit/s||1Gbit/s||1Gbit/s||1Gbit/s||1Gbit/s||1Gbit/s||1Gbit/s x2 dedicated|
|SO||Raspbian,Ubuntu,Android Things, etc.||Android and Linux||Ubuntu 16.04 or Android 5.1 Lollipop based on Kernel3.14LTS||Android and Linux||Android and Linux||Windows / Linux||Windows / Linux|
|Power||Rec. 5V@2.5A Power rating: 800 mA (4.0 W)||5V@4A input via DC Power||5V@2A via DC Power and/or Micro USB (OTG)||5V@2A via DC Power and/or Micro USB (OTG)||5V@2A via DC Power and/or Micro USB (OTG)||5V@4A via DC Power and/or Micro USB (OTG)|
|Storage||US$15 - MicroSDHC Memory Card 32 GB Class 10 UHS-1 U1||-||-|
Rasperry Pi 3
It could be the most famous card-size computer, with the biggest community.
The last version in this moment is the Raspberry Pi 3, with 1.2GHz 64-bit quad-core ARMv8 Cortex-A53 CPU, a VideoCore IV 3D GPU and only 1GB SDRAM @ 400 MHz it is not the more powerful computer in the list compared.
Also, a big handicap is the 100Mbps Ethernet and the storage system.
Other extra features: 40 pins GPIO, 802.11n wireless and bluetooth 4.1
Banana Pi M64
This computer board has the same 1.2GHz 64-bit quad-core ARMv8 Cortex-A53 CPU and an equivalent Dual core Mali 400 MP2 GPU.
The big difference is the 1Gbps ethernet, the 2GB LPDDR3 RAM the microSDXC and the 8GB eMMC included in the board.
The problem, the price. It’s more than the double of the Raspberry Pi 3. So per every Banana I can buy two Raspberries :)
Pine A64+ 2GB
Official Site: https://www.pine64.org/?page_id=1194 Wiki:https://en.wikipedia.org/wiki/Pine64 This computer board has the same 1.2GHz 64-bit quad-core ARMv8 Cortex-A53 CPU and an equivalent Dual core Mali 400 MP2 GPU.
The big difference is the 1Gbps ethernet, the 2GB LPDDR3 RAM and the microSDXC.
The price is really good. Better than the Raspeberry and cheaper: $30
The bad part, no wifi or bluetooth without an extra $10. Also, old HDMI 1.4
Odroid XU4 and C2
Official site: http://www.hardkernel.com
Both are good alternatives.
Both with 2GB Ram, 1GBit/s Ethernet eMMC5.0 that is faster (expensive and difficult to find) than a microsd.
XU4 has a older CPU with a 32bits architecture, but the performance is good enough. Also, has a USB 3.0 port, so you can use really cheap 128GB 100MB/s scandisk pendrives. The problem is that it is one of the more expensive boards.
C2 is the newest one, with new A53 CPU with a 64bits architecture but with USB 2.0 ports.
Mini ITX homemade cheap PCs
Selected configuration: https://uk.pcpartpicker.com/user/angelcervera/saved/6xfmGX
It is going to be more powerful than any other combination, but I can not buy more than 3 servers.
In this field there are a lot of option, from £65 with only 2GB of RAM and 32GB of ROM to £140 with 4GB of RAM and 60GB SSD SATA3:
Type Item Spending CPU AMD 3850 1.3GHz Quad-Core Processor £29.70 @ Eclipse Computers CPU Cooler ARCTIC Alpine M1 Fluid Dynamic Bearing CPU Cooler £5.35 @ Amazon UK Motherboard Asus AM1I-A Mini ITX AM1 Motherboard £26.99 @ Eclipse Computers Memory Patriot 4GB (1 x 4GB) DDR3-1600 Memory £22.07 @ Amazon UK Storage Western Digital Caviar Blue 250GB 3.5” 7200RPM Internal Hard Drive £25.49 @ Amazon UK Case CiT MTX-005B Mini ITX Tower Case w/300W Power Supply £27.94 @ Ebuyer Prices include shipping, taxes, rebates, and discounts Total £137.54 Generated by PCPartPicker 2017-01-07 11:44 GMT+0000
DIY combination 2:
Using a barebone:
£90 - Mini PC assembly as SmartTV Box: Plater® Z83 Windows Mini PC Full 4K HD H.265 Smart TV Box Intel AtomTM x5-Z8300 Processor HD Graphics DDR3 2GB System Disk 32GB Gigabit Ethernet WiFi 2.4G/5.8G
£65 - Other Mini PC assembly as SmartTV Box : Z83II Mini PC Intel Atom x5-Z8350 Quad Core Windows10 64bit 2.4G+5.8G Gigabit Ethernet TV BOX 2GB RAM 32GB ROM
The specification are really good. All advantages of a Intel PC and all advantages of a card-size board.
But this comparison it to find a low cost server (no for robotics) and at the end, the price is the same that a Mini ITX.
Really, there is not a winner.
In my opinion:
- If you have enough money or your wife does not manage it: A custom miniPC is the best option.
- If you don’t want expend a lot, Pine A64+ 2G with a good microSD car is the best option.
Other conclusion is that Rasberry Pi is the more important in the market with the biggest community, but looks that it is not the best relation price/specs.
- A good example and inspiration DIY case in https://makezine.com/projects/create-an-enclosure-for-a-6-node-raspberry-pi-cluster/