Vendetta Online was conceived as a single-server, non-sharded game universe of potentially unlimited scale. Although our engine would work equally well with sharded designs (sharding is much easier), this historical point is mentioned to help illustrate the background of our architectural choices. The server-side needed to be able to scale effectively to a very large simultaneous-player capacity, while the client-side needed playability across a broad range of available hardware. We began development in 1998, during the modem era, so a strong emphasis was placed on network protocol efficiency; in later years this has helped us to pack a great deal more activity into a smaller amount of per-client bandwidth, a feature of particular value on mobile wireless networks. (The server has been experimentally implemented and used on Amazon EC2).
Like many live MMO products, Vendetta Online is still undergoing a process of continual refinement, so some aspects of architecture may fluctuate over time. That being said, here are some basics:
The Server- The Core Packet Router, uses tightly optimized C++ with administrative Lua on top, and handles player connection aggregation and logins. Per-user packet queues allow variable rate shaping, packet re-ordering and optimization (newer update replaces unsent older update). Realtime datastream compression further improves bandwidth usage, compression tables may be genetically optimized to common packet use cases. Connections to Players, Kourier (Erlang), Sector Daemons, and the SQL server are maintained. Features built-in webserver with admin functionality.
- The Sector Daemon actively controls an area of game geography, written in C++ and Lua. Vendetta Online is made up of over 7,000 different "sectors"; only those with players in them are "awake" and running at any given time. The SD can control any physically sized region, and handles the server-side of intensive tasks such as collision detection, physics, and low-level AI (PRM pathfinding, etc). SDs are distributed across a large cluster based on monitored per-machine CPU and memory load, with SDs transparently started as needed by players in the game, and shut down after "idle" parameters are met. Any number of sectors are possible, allowing for a game universe of near-infinite scale.
- Kourier is the moniker given to our Erlang-based subsystem, used by us to control large-scale gameplay and AI. For example, an individual Sector Daemon may handle the low-level AI for a given set of NPCs, perhaps a hundred or so in a specific region, having them react to various input criteria and other areas requiring high performance. Kourier, on the other hand, manages the high-level behaviour across tens of thousands of NPCs, commanding them on a larger scale and managing the behaviour of armies, nations, economies, and the like. Erlang is a highly-scalable functional language and runtime system, originally developed for use with always-on phone switches made by Ericsson. Distributed usage and lightweight threading are two of its core properties, making it ideal for our application. Using this, we can maintain persistent threads for every NPC, as well as many other game cases, with minimal overhead and great scalability. Should the NPC be unneeded, such as being located in a non-running sector, the thread can be hibernated, reducing the overhead even further. Vendetta Online usually has between 4,000 and 30,000 NPCs in production at any given time; although we have tested the Kourier system with over 300,000 NPCs at a 10x simulation rate, without issue (each NPC using two unique threads, on a single quad-core machine with 4GB of ram), and we expect it to scale far beyond that point. Kourier also includes an advanced binary logging system, capable of recording thousands of events per second, which can be used as a kind of chronological database, allowing one to view only what is pertinent to the task at hand.
