Cloud Benchmark at Slush 2015

  • December 3, 2015
Try UpCloud for free! Sign up now and receive a completely free trial. Sign up now


Slush 2015 is a 15.000 attendee startup event in Helsinki where more than 1700 startups and 800 investors arrived for the two-day event on 11th and 12th of November. Like last year, UpCloud was one of the key partners of Slush and we were showcasing our cloud hosting service at our stand. For this year’s event, we wanted to show off some benchmarks and we created a benchmarking game comparing UpCloud, AWS EC2, and DigitalOcean.

Traditional benchmarks are boring, so we created a Commodore64-style game theme, which includes live setup of a cloud server, running the benchmarks and then finally destroying the cloud server. The benchmark game runs from the start (server creation) to finish (server destroyed) in a timeframe of couple minutes. We managed the cloud servers in real-time using Ansible and its modules for the cloud hosting providers. During the two days at Slush, the benchmarks were run 167 times and we collected all the results.

Let’s show the results first and then go through what they mean:

Cloud Server Benchmark Results (averages)
Benchmark UpCloud DigitalOcean AWS EC2
Server setup 48.17 s 81.73 s 78.00 s
CPU (cyclictest) 15.19 120.26 66.59
RAM (mbw) 8352.01 MB/s 4966.74 MB/s 7567.29 MB/s
Network (wget MB/s) 56.09 MB/s 153.77 MB/s 121.21 MB/s
Disk I/O (fio, 4k block randwrite) 83990.50 IOPS 7202.81 IOPS 17049.99 IOPS
Disk I/O (fio, 4k block randread) 99854.57 IOPS 59484.3 IOPS 59087.82 IOPS
Price (USD) $40/mo $40/mo $240/mo
Cloud Server Benchmark Results Consistency (standard deviations)
Benchmark UpCloud DigitalOcean AWS EC2
Server setup 2.56 s 11.32 s 3.45 s
CPU (cyclictest) 2.55 73.93 3.04
RAM (mbw) 130.00 MB/s 549.84 MB/s 740.04 MB/s
Network (wget MB/s) 2.41 MB/s 33.25 MB/s 6.83 MB/s
Disk I/O (fio, 4k block randwrite) 5211.52 IOPS 1277.83 IOPS 9098.21 IOPS
Disk I/O (fio, 4k block randread) 926.22 IOPS 9062.07 IOPS 8328.36 IOPS
Statistics reminder: 95.5% of results are within 2 standard deviations of the average.
The Benchmarked Instances
UpCloud DigitalOcean AWS EC2
Instance *custom 2 CPU, 4GB RAM, 60GB MaxIOPS $40/mo (2 CPU, 4GB RAM, 60GB SSD) c3.large + EBS provisioned IOPS 60GB (with IOPS maxed)
Location Frankfurt Frankfurt Frankfurt
Price (USD) $40/mo $40/mo $240/mo
*UpCloud’s $40/mo instance has 4CPU, 4GB RAM and 100GB MaxIOPS storage so we opted to use a less powerful custom instance to match AWS EC2 and DigitalOcean.


We’ve marked the winner of each category with the color green. The averages give a view of the performance and the standard deviation is a good way to measure the consistency; ie. how close to the average you can expect each test run to be. UpCloud really shines in both of these only losing in randwrite IOPS consistency whereas our average results are phenomenal compared to AWS EC2 and DigitalOcean.

The only benchmark where UpCloud is not the winner is network; this is due to the fact that we limit each cloud server to a port speed of 500Mbps (it can be upgraded to 1Gbps upon request). This is a service quality measure as we don’t want any server affecting the network performance of others by accident.

Server setup
Server setup is the time in seconds from starting the Ansible playbook to when Ansible tells us that the instance is responding over SSH. You can expect similar results when using the control panels of these providers to create servers, as Ansible is using the standard APIs of each provider. UpCloud is fastest in deploying a cloud server thanks to our quick MaxIOPS storage system that allows quick cloning of the OS template. Interestingly, DigitalOcean is the slowest with an average deploy time of 120 seconds.

An example Ansible playbook for UpCloud. For AWS EC2 and DigitalOcean, use their Ansible modules instead for setting up servers.

- hosts: localhost
  connection: local
  serial: 1
  gather_facts: no
    server_name: "{{ SERVER_NAME|default('') }}"
    tag_name: "{{ TAG_NAME|default('www') }}"
    - name: Create UpCloud server
        state: present
        hostname: "{{ server_name }}"
        title: "{{ server_name }}"
        zone: de-fra1
        core_number: 2
        memory_amount: 4096
          - { size: 60, os: Ubuntu 14.04, address: "virtio:0" }
      register: upcloud_server

    - name: Wait for SSH to come up
      wait_for: host={{ upcloud_server.public_ip }} port=22 delay=5 timeout=320 state=started


CPU ~ cyclictest
We used cyclictest to measure the average latency of the CPU, or the time it takes to complete a CPU cycle. This is essentially a measure of how overbooked the physical CPUs are; how many virtual CPUs are “lining up” to get their turn at crunching numbers at the actual physical CPU. Thus, a smaller number is better as it is a sign of how responsive the CPU is, and what kind of performance you are getting for your money.

UpCloud demonstrates much smaller latencies with a much better consistency. Create a cloud server at our Frankfurt site right now, and you will get similar results with a very high probability. AWS can also show a very good consistency in their results, unlike DigitalOcean who had a very high standard deviation in its results.

Exact command to reproduce test is shown below. We used the avg_lat value.

apt-get install rt-tests
sudo cyclictest -D 10s -q


RAM ~ mbw
Mbw is a measure of how fast data can be moved in and out of RAM. The faster the better. We didn’t expect huge differences between cloud providers but for some reason, DigitalOcean seems to be lacking. Given the equal size of RAM, this is an important measure as moving data in and out of RAM is one of the common bottlenecks in computing in general. Given an equal amount of RAM, it is certainly better to get good results from Mbw – you wouldn’t like paying for less, right?

Exact command to reproduce test is shown below. We used the avg_copy value.

apt-get install mbw
mbw -n 250 -t 0 200


Network ~ wget
Measuring network performance is a bit tricky as it isn’t easy to get completely objective results. UpCloud loses here as we currently limit cloud servers to a 500Mbps connection (can be upgraded to 1Gbps upon request). The reason behind this decision is one of service quality, we do not want any single customer or their cloud server affecting the network performance of others.

Exact command to reproduce test is shown below. We used the average speed value for this test.

wget -4 -O /dev/null


Storage I/O ~ fio
Fio is the traditional benchmark for disk I/O. We are measuring random read and write IOPS to compare the disk performance of 60GB SSDs. UpCloud performs superbly, especially in random writes. Thanks to our proprietary MaxIOPS system, our competitors can not match or even approach our disk I/O performance that reaches 100 000 IOPS. The speed of storage is very noticeable; your databases and file servers can see a huge performance boost on top of MaxIOPS. The performance boost isn’t related to only database servers, as regular web servers perform much better with no bottlenecks in storage. This is something we’re especially proud of.

Exact command to reproduce test is shown below. We used the IOPS value.

apt-get install fio

# device depends on the provider; make sure you run against the correct disk

sudo fio --minimal -name=temp-fio --bs=4k --ioengine=libaio --iodepth=64 --size=1G --direct=1 --rw=randwrite --filename=/dev/"+str(device)+" --numjobs=12 --time_based --runtime=5 

sudo fio --minimal -name=temp-fio --bs=4k --ioengine=libaio --iodepth=64 --size=1G --direct=1 --rw=randread --filename=/dev/"+str(device)+" --numjobs=12 --time_based --runtime=5 


UpCloud can offer clear price and performance benefits against AWS EC2 and DigitalOcean as shown in our benchmark test. We can also offer much predictable performance as can be seen from the standard deviations. Performance and predictability together mean that you don’t have to be afraid of sudden slowdowns and reserve extra resources to cope with that – again adding to the cost of your infrastructure. Performance in general means that you don’t need as many resources in the first place, to begin with. For example, try running a database on top of our MaxIOPS and you will realize what we are talking about.

The results of these benchmarks show that you get more performance for your money on UpCloud, so why not take us for a test drive? We have free trials available here and you can always contact [email protected] if you’d like to more extensive testing or have some questions answered.

Cover photo by Sami Heiskanen