Dev Blog 4/24/2021 - Beta 7 Performance and Feature Updates

[jcsnider]

Beta 7 Performance and Feature Updates (a.k.a  'The Leafling Wrap-up')

4/24/2021

 

Preface 

People are always asking how many simultaneous players can Intersect handle this writeup will go on to tackle that question somewhat. Unless you've been living under a rock you probably know that Leafling launched on Steam in December. Despite a rocky start, Leafling's launch was very successful with peak online players hitting a highs in the 190s-200 range despite lots of performance issues and constant server crashes.

 

This was the first opportunity to really work on performance profiling, debugging, and improving an engine with thousands of daily users so I volunteered to help improve performance and fix bugs within Leafling given that the solutions would be back ported into Intersect Beta 7. A lot of the qol, looting, npc logic, and other misc updates were done by @Cheshire - so big thanks to her, and other performance fixes came from @panda as well, so everything below is entirely a team effort.

 

At this point in time Leafling is running very well, the server can run for days on end without crashing, and game performance is as good or better than ever. All the updates that we've made to Leafling (and some unintended ones) are already in B7, or will make it into 0.7.0.112.

 

This thread will go on to list and detail most of the changes made in Beta 7 since December. I figured that we outta get these documented/published somewhere here on AGD. 

 

Special thanks to @Aesthetic for letting us use Leafling as a guinea pig, who never complained, even when we released chains of updates that seemed at the time to be making things worse and worse.

 

There is a TLDR at the bottom if you find that you don't care about the specific details.

 

 

Thread Pools & Multi-Processing (Ref 1)

• Server architecture redesigned around thread pools
  • Thread pools will spin up threads as needed and each thread can run on a different cpu core allowing for full usage of todays processors with 8, 10,. 12, 16, or more cores.
  • Each thread pool can be configured, you define the minimum and maximum allowed threads.
• There are three main thread pools in Beta 7
  • Logic Pool: Handles map updates, and allows multiple maps to update/process at the same time.
  • Network Pool: Handles sending, receiving, and processing packets.
  • Database Pool: Handles most database operations such as saving players, writing logs, and more.

 

 

Network & Packet Updates (Ref: 1, 2)

• Serialization: Replaced Ceras as the library we used for packet serialization with MessagePack-CSharp because some of Ceras' unsafe memory allocations would crash under Mono (on Linux and Windows)
• Compression: All packets are compressed now because there are some users who still have <1mbit up/down here in the US, and without compression we would effectively DDOS those players with general game/entity/combat updates.
• Auto MTU Negotiation: The Intersect server will communicate with clients upon connection and negotiate the best MTU value possible instead of assuming 1400 or so. Honestly no noticeable performance difference came from this but it doesn't hurt to have.
• Reliable vs Unreliable Sending:  Some packets are now sent unreliably, which means that we don't care if they get lost like combat action messages. Every packet we can send unreliably takes load off the client and server with keeping track of packets being received and the order they arrive in.
• Batched Updates: For some packets like vital, stats, and status updates, the server will wait (maximum 50ms) and send grouped together in one larger packet instead of hundreds or thousands of updates in real time. This leads to the same information making it to the client without a perceived delay and cuts down on network traffic and processing requirements significantly.

 

 

Database Updates (Ref: 1)

• We're finally using Entity Framework (our db interface) properly!
• Db connections are created for short operations (like logins) and then disposed of.
• Resolved rollback issues where players would lose exp and levels.
• Resolved an issue where players who have logged out would be kept in memory forever.
• Huge performance improvements/gains.

 

 

Client Updates (Ref: 1, 2)

• Cycle through targets with auto target (default tab key)
• Esc key closes open windows
• Enter key progresses through dialog windows where applicable
• Entity box (hud) is static
• Memory leak fixes in entity box & party menu
• Fixed crashes/performance issued caused by Tooltips
• Server announcements are shown on screen

 

 

Map Item/Looting/Bag Updates (Ref: 1)

• Server config.json will allow you to set a loot radius in which loot will show up in the loot window and can be grabbed from.
• Dragging items from inventory into bags work instead of right clicking, and you can drag into specific/desired slots.
• When withdrawing/depositing/selling/dropping items the existing quantity is automatically populated into the numeric input box instead of 0.

 

 

Event, Player, and NPC Upgrades (Ref: 1, 2)

• Dozens of event processing, conditional checking improvements.
• Change name event command
• Reset radius for npcs
• Movement type static for npcs (never move, even when engaged)
• Individualize loot option for npcs (intended for bosses) where all attackers and their party members can receive individual drops. -- Now togglable via the Npc editor
• Use caching for event positions allowing quicker lookups
• General optimizations to the player update function

 

 

Combat Updates (Ref: 1)

• Stat calculations fixed
• Npc targeting/logic updated to be smarter
• Server config option to disable combat messages being shown in the chatbox
• Hotbar will autocast if a hotbar key is held down

 

 

API Updates (Ref: 1, 2)

• Users
  • Search and sort users list endpoint results by name or email address
  • Change user names
  • Delete users
  • Track registration dates, last ip address, and total playtime
• Players
  • Search players list endpoint results by name and email address
  • Sort players list endpoint results by name or rank
  • Change player names
  • Delete players
  • Track creation dates and total playtime

 

 

Chat Logging (Ref: 1)

• Thanks @Mighty Professional for this one!
• If enabled chat messages will be logged to the logging database.
• Chat logs can be retrieved and viewed using sqlite viewer or via api endpoints

 

 

Metrics Collection (Ref: 1)

• If enabled in the server config, the server will keep track of how fast or slow it's running.
• Includes stats on active maps, players, entities (players + npcs), events, map update times, thread pool utilization, packet counts, bandwidth, and more!
• Metrics can be fetched via the API and used in external applications for realtime performance monitoring.

 

 

TLDR; Key Takeaways

• We've now had an intersect game with over 190 simultaneous players, and I'm confident that it can handle more!
• The Intersect server is now optimized to take much better advantage of multi-core and multi-cpu systems.
• There are a few new features and quality of life updates.
• I'm sure I missed a dozen or more changes in this list, we changed so much in short amount of time in order to optimize performance.
• We're still cleaning up a bit from all these changes, but Beta 7 is already far superior to Beta 6 if you are planning on publishing your game to have dozens or hundreds of players online at a time.
• Excited for other games to launch on Steam and test Intersect's capabilities further!

 

 

 

Going Forward

The plugin system is still a goal of ours to get running well, right now it's in bad shape but we will be getting back to working on it. Outside of that the next major feature I intend to tackle is Instancing. I'll also be sharing more details on my side project (Intersect Dashboard) very soon with closed alpha/beta signups.

 

[Cheshire]

1 hour ago, jcsnider said:

even when we released chains of updates that seemed at the time to be making things worse and worse

 

🤫

[Kamus]

i love all the changes and the things that all of the dev team are making

I'll still following all the major updates for make their respective spanish post for hispanic community 😎👌

[jcsnider]

Modified OP to state that all of those changes and more will be in 0.7.0.112 which is building now.