This is a summary from our 2014 whitepaper, ManyCraft: scaling Minecraft to Millions. Some of the numbers are outdated as Minecraft has become even more popular.

Introduction

Minecraft is a popular game with more than 20 million users. In multiplayer mode however, only a few thousand users can play together. Our measurements show that even reducing the map to a uniform flat land, a server cannot host more users.

We conducted experiments with many simulated users to understand these limitations. As expected, the simulations show that bandwidth consumption, CPU load, and memory usage increase with the complexity of the world and the number of connected players.

Unexpectedly, with a minimalistic map - a uniform flat land and totally inactive players - server resource utilization remains high. Merely maintaining the presence of the players is a resource intensive activity.

We have designed and implemented Manycraft which can scale to millions of users using exaQuark, a distributed infrastructure for scaling virtual worlds.

Attempts to scale games

Scaling multiplayer games usually requires partitioning the game - zoning, sharding, and instancing. These methods do not allow an unlimited number of simultaneous users in a unique contiguous space.

Other solutions propose to improve scalability using p2p architectures. Distributed Scene Graphs are used to scale complex 3D landscapes in virtual worlds such as Second Life. Minecraft has simple graphics so it doesn’t have the same requirements for 3D scene computation. Colyseus and Donnybrook are middlewares, used to develop multiplayer games built on top of Mercury (a p2p overlay of range-queriable DHTs). They have been tested for Quake III with up to 900 simultaneous players. However, DHTs have been proven to fail under frequent player movements. These solutions scale at most up to several hundred users - far fewer than existing Minecraft servers.

Separating the static elements of the scene from the avatar movements is a more promising approach. However, despite good theoretical or simulated results, these solutions have failed to be widely deployed. This may be because their p2p architecture is impractical: adoption by users is slowed by the extra steps required to install the software; developing and maintaining code for multiple versions of several platforms is costly and cumbersome; and overall performance is impacted by the slowest peers.

We have developed exaQuark, a cloud based solution to scale virtual worlds by handling the avatars movements. exaQuark offers a streaming API where avatars report theirs positions and get notified of the events in the neighbourhood. Static elements handled separately from interactions and actions. The typical example is HybridEarth, which relies on exaQuark for the avatars while the landscape is Google StreetView.

An overview of the Minecraft architecture

A Minecraft map is made of simple cubic blocks. All blocks have the same size and only vary in their type (wood, coal ore, stone). Players are represented by two block high entities. There are several playmodes: creative, adventure or PVP.

In creative mode, where the purpose is to build things, players add and remove blocks. This mode is very popular as it can be played alone. But in order to present their creations, players either host a server on their own computer and allow others to connect, or record a video and share it on the internet. The first solution is cumbersome and costly: the user needs to maintain a server for a few friends that might visit, and the second does allow players to visit the world.

Inside Minecraft each player is represented by an entity. To play together they connect to the same server.

Minecraft server

A Minecraft server hosts the world and delivers the content to the connected players. Server’s time unit is the tick, which lasts for 50ms. Within each tick, the server updates world physics and notifies each player about changes that occurred in her awareness area. The elements constituting the map evolve. Dynamics such as water flowing, plants growing or night and day alternation, are computed when they are part of someone’s field of vision. Also, “mobs” (simulated entities) react to the environment by moving, attacking, etc.

Minecraft client

After connection, the player is spawned at a position chosen by the server. During the game, the client sends to the server actions: position updates, block destruction, etc. Players are notified about events occurring in their awareness area.

Minecraft protocol

Protocol messages can be divided into the following categories: map, entity and control.

• Map - Map related messages are generated when a player changes position or when blocks in the awareness area are changed. The server only sends the differences with respect to the last configuration.
• Entities - When a player or a mob moves, the new position is sent by the server to every player that should be aware of it. Entity related messages also bear chat, actions or other entity attributes.
• Control - During each tick, the server sends keep alive messages to clients. There are also cyclic events, such as day-night alternation, that generate messages at regular time intervals.

Evaluating Minecraft Scalability

The maximum number of simultaneous players for online Minecraft multiplayer servers is in the thousands. We have generated simulations of Minecraft to find out why they unable to scale further.

Experimental Setup

To estimate the load generated by the players’ presence we reduce both environment complexity and in game interactions:

• the world contains no mobs, only blocks and players. so the server has no animal behaviours to compute
• the map is Superflat and made of a single layer of rocky blocks, eliminating world’s physics as no plants are growing and no water is flowing.
• our simulated players don’t chat, nor mine, nor craft.
• we set the view distance to its minimum value. This is a well-known trick to reduce server’s load.

We ran our experiments on four identical machines hosted in the same datacenter. Each machine had the following specs:

- 8-core Intel Xeon at 2.8GHz
- 24 gigabytes of RAM
- Ubuntu 12.04
- Oracle Java 7
- Gigabit-ethernet connections

The average latency between machines is around 0.8ms. One of the machines ran a CraftBukkit server. A CraftBukkit server is a Minecraft server modded to accept Bukkit plug-ins. The other machines ran several simulated players using text-only Minecraft clients, with no graphical rendering.

Players’ movements followed the model of avatars movement in virtual worlds: a player is either slowly random walking or moving fast in a given direction. Players move at the same speed as human driven Minecraft players and similarly update their positions 20 times per second.

Measurements

In our experiment we add players to the server one by one, waiting 5 seconds between two insertions. This prevents players having their connection denied by the server. After inserting a batch of one hundred we wait several minutes for the system to stabilize before taking the measurements. This ensures that the initial cost of entering the world does not impact the measurements for resource utilization.

Throughput

In Figure 1 the x-axis is the number of simulated players connected while the y-axis is the server throughput in megabytes per second. These two curves clearly show that the bandwidth consumption grows linearly with the number of players.

Each Minecraft client sends ∼3.5Kbytes/s and receives ∼3Kbytes/s. For a server connected to the backbone through gigabit ethernet – a common setting for servers in datacenters – we can assume a full duplex with maximum throughput of 500 Mbits/s. This indicates that bandwidth should start to be a limiting factor for approximately 20K players.

With a more complex, modifiable and interactive map the throughput per user should be greater and the maximum number of player significantly reduced.

Memory Usage

In figure 2 the x-axis is the number of players while the y-axis is the amount of RAM used by the server in gigabytes. The server runs on a Java virtual machine, so the slight deviations could be caused by the garbage collector. At this scale, the memory usage of the server can be approximated to grow linearly with the number of connected Minecraft clients. A server that can allocate 24 gigabytes to the server reaches its RAM limitation for approximately 1500 connected players.

CPU Load

To evaluate the impact of entity dynamics we used both moving avatars and stationary avatars (they connect but remain inactive). Figure 3 shows the results of the two measurements: the x- axis is the number of players while y-axis is the aggregate percentage of the CPU load of all cores: a percentage above 100% indicates that more than one core is used by the Minecraft server. Given that we used an 8-core server, the maximum load it can stand is 800%.

We weren’t sure that our model for avatar movements reflect the actual behaviour of Minecraft players, but we compared CPU load between moving avatars vs. stationary avatars and found little difference anyhow.

Both measurements show that the CPU load can be approximated to grow linearly. Therefore, with a very simple map, the maximum load supported by our 8-core CPU should be just above 1000 players.

Conclusion

One of the prominent outcomes of this experiment is that the map complexity might lower the maximum number of simultaneous connected players but is not the major cause of this limitation. The other lesson is that there is not a unique bottleneck; when the number of players grows all resources become exhausted at once.

The only way to go far beyond ten thousand players – the actual observed limit – in a Minecraft map is to distribute the load generated by the entities amongst many machines.

Manycraft

Manycraft enables an unlimited number of players to interact in a static Minecraft map. To join the system the player runs a Manycraft node on her own computer.

The Manycraft node is a Minecraft server “modded” with Bukkit in order to divert the entity messages to exaQuark. In return, exaQuark sends the events occurring in the player’s neighbourhood. The local Minecraft server has the static map preloaded and its main duty is to deliver the map to the player. The load generated by the avatars has been shifted entirely to exaQuark.

exaQuark is a scalable distributed infrastructure for virtual worlds, designed to support an unlimited number of moving objects updating their position at arbitrary high frequencies. In exaQuark the set of moving entities is distributed onto many servers, each taking care of a group of entities based on their geographical proximity. The positions are indexed using a distributed Delaunay triangulation for efficient neighbourhood access.

The same indexing structure is used to partition the space into zones following entity density. As entities move, these zones dynamically change their shape. In summary: exaQuark servers handle a relatively constant number of entities, so scaling them is predictable and easy. To free the application layer from complexities, exaQuark provides an API for developers.

// example: players send updates of their positions and actions
{
  method:"update",
  urlid:"http://manycraft.net/u6nMnT",
  iid:"john.smith",
  lng:52.5, lat:15.3, angle:40, alt:40,
  appdata:{
     manycraft: {
       sneaking:false,
       pitch:0.147
     }
   }
 }

When connecting, a client is assigned a proxy, the entry point to exaQuark for the entire session. Each proxy is connected to a subset of all entities and, for each of them, maintains the communication with the corresponding zone and neighbourhood. And, in order to scale, exaQuark spawns as many zone servers and proxies as needed.

exaQuark clients send three types of commands: update entity state, message to all and private message. On the other side, they receive four types of notifications: the full list of neighbours, neighbourhood updates, message from another client, and remove neighbouring entities.

exaQuark has been designed independently of any particular system and messages have an appdata field to carry data specific to the virtual world to scale.

Manycraft Node

To join a Manycraft world, the player downloads the node preloaded with the corresponding Minecraft static map. The node is a regular Minecraft server and a proxy. Once the node is running on her machine, she connects her regular Minecraft client to the local Minecraft server through the proxy. The local server receives the packets from the client and serves the blocks of the map as the player moves. The proxy sends player movements and actions to exaQuark.

exaQuark sends back the movements and actions occurring at nodes of neighbouring players. These notifications are translated and processed by the internal Minecraft server which generates the packets for the client. Remote players movements and actions are then rendered by the client.

Players are in the same map - they see each other and interact. They are together in the world.

Bridging Minecraft over exaQuark

The essential task of a Manycraft Node is to route the relevant Minecraft traffic from and to exaQuark. The proxy inspects the messages issued by the client. All the messages, including player movements, are forwarded without modification to the Minecraft server. On position updates the Minecraft server sends the surrounding blocks. Messages concerning player actions are translated and sent to exaQuark which sends back notifications about the neighbours. These messages are translated into Minecraft messages so the neighbours entities can be placed on the map as mobs. This way, the server attached to a player has all the information corresponding to the awareness area.

Chat and player actions are translated into exaQuark message commands.

exaQuark notifications inform a node about actions and state changes of remote players in the neighbourhood. Remote players are represented in the Minecraft server as mobs. exaQuark notifications trigger movements and actions of these mobs. Upon notification of entity arrivals/departures in player’s awareness area the mobs are created and removed accordingly in the Minecraft server.

Manycraft Scalability

The Minecraft server embedded in the node supports only the load of one client. The traffic with exaQuark is a subset of the normal Minecraft traffic and remains low. Finally, the translation and routing processes of the proxy do not generate any significant load.

Compared to a regular Minecraft client, the CPU load of the user’s computer is only increased by the low activity of the embedded server. The memory depends on the size of the preloaded map but remains constant and low. Because in average the number of neighbours provided by exaQuark is constant, resources used by the proxy remain constant. The external communication of a Manycraft node happens only with exaQuark and the throughput is similar to a Minecraft client, minus the incoming map description messages. All in all, compared to classical setting — a Minecraft client connected to a remote multiplayer server — the load of the user’s machine is lower regarding bandwidth and slightly higher regarding CPU and memory usage.

Manycraft is a distributed system composed of nodes connected through exaQuark infrastructure.