Disclaimer: this post does not intend to provide a highly scalable solution for developing a massive multiplayer online game. The hypothetical scenario described here should be able to support a few thousand users. Existing online gaming frameworks take care of the heavy infrastructure considerations by implementing proven design patterns. These battle-tested abstraction frameworks allow developers to concentrate on implementing their game's specific logic.
Designing a Massively Multiplayer Online (MMO) game is not trivial. Latency should be relatively very low, usually less than 200ms, and preferably less than 100ms. Players and objects are moving constantly, and these changes should be available to all clients. After all, you wouldn't want to be "killed" because your location wasn’t updated as fast as an axe thrown by your opponent.
In a game with thousands or hundreds of thousands of users, you'll probably need multiple frontend servers to manage the client connections and continuously route game events to the clients.
In this post I'll create a hypothetical game with players in constant movement in the game world. Since I'm targeting novice developers, we're not going to use any of the existing gaming frameworks. I want to keep things simple, so that novice gaming developers can start creating multiplayer online games without needing to master a gaming framework.
- Latency - no greater than 200 ms for an event to reach relevant clients from the server. If the latency is too high events aren’t properly synchronized and become awkward from the users' perspective.
- World- there are no different worlds, areas, or rooms in our game. All users move in the same, big space. The users should see all users or objects in their vicinity —whatever objects enter into their view. The user’s view, is the part of the world that is visible to the user.
- Code - we’re avoiding any MMO stack or gaming framework as described above.
This image simulates our "world" in which the users are randomly positioned. The red and blue borders around the green circle user represents the user’s "view" (the visible portion of the world). The space between the red and blue borders represents the space beyond the user's view where the user will still receive updates, such as players entering or exiting the user’s view.
MMO games are nothing new and there are few frameworks available that implement the actor design pattern in Java, .NET and other languages. The common implementation of these actor design pattern frameworks is that they keep the actor instances (AKA Activations) and their state in-memory. The state is spread on all the servers. The framework also abstract the underlying communication between the servers, so that if two players are activated in two different physical machines, they will be able to communicate one with the other. These frameworks also handle other common scenarios like servers failures, reactivation, load balancing, and much more.
Using an actor design framework requires a large amount of RAM machines so that the service can maximize the amount of users' state retained. My approach is to use one big cache server (like Redis) to hold all of our users' state. Additional information is available here regarding which Redis configuration to use for your scenario
Proposed solution layout:
- Clients connect to our front-end servers using websockets enabling the user to send and receive events to and from the servers.
- Node.js service keeps the open clients' web socket connections. This service will run on each of our front end machines. We'll use Node.js cluster feature to fork processes according to the number of cores available in our VM. Since our service does not contain state we don’t need to worry about affinity.
- We'll use the socket.io node module to manage websocket connections.
- Redis setup: we'll implement 2 main features:
- Caching - Redis will be our sole state server, retaining all users' state.
- Pub/Sub - using the publish and subscribe mechanism to synchronize all of the servers with events coming from clients. This feature will be used to broadcast location change events from a client connected to one server to all of the clients connected to the rest of the servers.
- We will run our solution locally but when we come to publish our game we can use an Azure VMs (either Windows or Ubuntu machines). We can start with one and scale out with more instances according to our needs.
The number of users we can support is heavily dependent on the nature of our game. Some important important considerations include: the size of the world, the size of the view, the density and the way users are spread in the world, the number of events each client sends and the frequency, etc. One front end (FE) server might support enough users for our purpose in one configuration while in other game scenario might require additional FE servers.
In our approach- one Redis machine quickly becomes a bottleneck as additional users are added to the game. As well since Redis is single-threaded, at some point, it will be inevitable to switch to a solution where the state is either clustered (Redis cluster) or split between servers using an existing framework as described above.
A fully implemented code can be found here. You're welcome to review the code. Instructions regarding how to install and setup the application to run locally on your machine