Dev Blog 11/28/2017 - Jsonification!

[jcsnider]

Jsonification!

Nov. 28th 2017

 

News and Updates

After the buggy Beta 4 release, and 7 (going on 8) patches we're finally making some decent headway into the Beta 5 todo list. I'm working on jsonification -- I'll explain that more later. Panda is working on some database changes and Kibz is back to work with some quality of life modifications. (Infinitely many npc drops and the drop rate going below 1% for starters!)  I will let them discuss everything else they're working on at their own pace. Let's jump into what I'm doing.

 

 

Jsonification (verb: The act of replacing .xml files with .json alternatives)

Intersect uses .xml files for all of our configuration purposes. We use xml for basic server settings, color definitions, language translations, and even the dynamic ui system. Xml has been fine, but I recently discovered that by using Json we could eliminate a ton of code from the engine along with several other benefits. The biggest benefit for switching to json is that when we add more options/translations/colors/etc they will appear in your existing translation/config file and not require you to add it manually or start your config again from scratch! (This is huge for people trying to keep up with translations every update)

 

Json files are smaller than xml files

Json doesn't use nearly as many characters for layout, so the file sizes are smaller. Pictured below is our current server config.xml file (first) compared to the new config.json file (second).

             

 

Language files are Jsonified too!

Translations will have to be re-done just one more time! In the future, if we change the engine strings then changes will appear in your language files automatically so you only have to translate the new stuff instead of the whole file every single update.

 

XML Language Example (Old)

   

 

Json Language Example (New)

 

 

Dynamic UI is up next!

This is giving me a good opportunity to revisit Dynamic UI and fix how it works. Instead of the massive files that take forever to load and are scary to work with, I'm separating each window/panel into different json files. 

 

 

The same benefits apply to dynamic ui -- after this update, if we add more options to the ui elements they will appear in your modified files, instead of forcing you to restart your ui design! After this update I promise we'll get some dynamic ui documentation out there!

 

No migration this time

Sadly these files have changed too much for there to be an easy migration path. I know this will be particularly hard on those with existing translations and dynamic ui layouts. For translators I'm going to release the new language files early.  For dynamic ui'ers I will make a post on how to most-easily port your modifications to the new format when Beta 5 is released.

Client.English.json

Editor.English.json

Server.English.json

Migrator.English.json

 

 

Whats next?

After this I will start working on the fun stuff like fixing animation autorotations, resource layering, etc. We will also be (finally) implementing item extra effects like lifesteal and cooldown reduction. Beta 5 is still a ways away -- but I wanted to get this dev blog out to inform people with translations/dynamic ui changes of the new format that's headed this way.

 

As always, feel free to post comments and questions below! We are excited to hear your thoughts on our progress!

 

-The Intersect Development Team

[Ambard]

Well done! Looking forward to a fresh start. The .json file format and the new ui system look great too.

[Aon Reo]

It's great to see this change. JSON is so much easier to manually read than XML. I've never had to parse XML but I do work with JSON results from web API's often and XML configuration files all the time. Very exciting, keep it up!

[jcsnider]

3 minutes ago, Aon Reo said:

It's great to see this change. JSON is so much easier to manually read than XML. I've never had to parse XML but I do work with JSON results from web API's often and XML configuration files all the time. Very exciting, keep it up!

Yes, being able to easy serialize almost any Intersect class/structure will be a big benefit of already using this Json library. I expect to see REST services in the future that process and distribute json objects   (like character info, item info, etc, etc)

[Refur]

Nice news. Do you recommend still working the UI with xml? I mean，the json files will have the same properties? @jcsnider

[emptyaccount]

Magnificent! Love this Jcnification  aha

[jcsnider]

6 hours ago, Refur said:

Nice news. Do you recommend still working the UI with xml? I mean，the json files will have the same properties? @jcsnider

Yes, all of the properties will be the same, just a little bit of a different format.

 

For those with existing ui changes, you've be able to throw your modified file into a comparison tool with the default ui file, see what's different, and then manually move those changes to the json equalivants.

[Agoraphobic]

Keep up the great work!

[Skaveron]

Will the binary blobs in the database be replaced with serialized JSON as well? That would make web apps so much easier  

[jcsnider]

1 hour ago, Skaveron said:

Will the binary blobs in the database be replaced with serialized JSON as well? That would make web apps so much easier  

Its looking like @panda is attempting to create the logic that would create columns and extra tables to store all of the game objects' values. That'd be the best case scenario.*** (I'm on edge about this, we'd then have 3 or 4x the tables and the data base would be a mess to navigate and upgrade)

 

If that doesn't happen then yes, at the very least, in Beta 5 the binary blobs will be replaced with serialized json.

[Cheshire]

3 hours ago, jcsnider said:

(I'm on edge about this, we'd then have 3 or 4x the tables and the data base would be a mess to navigate and upgrade)

Upgrading would be messy maybe, but at the same time data would be far more accessible for development and tools outside the engine itself. (Also, searchable by queries allowing mass updates of stat weights, balancing revamps etc.) I'd say that's more important long term than upgradability for short-term patches that will become infrequent.

 

Also, you can alter tables and add values with a simple query on the database, so how is that harder than loading the old data, transposing it to a new class and then saving it to the DB like the current database works? yeah it's harder than having the Json lib figure it out for you but at least you'd be making proper use of the database system (and would allow for indexing on MySQL)

[Aon Reo]

6 hours ago, jcsnider said:

Its looking like @panda is attempting to create the logic that would create columns and extra tables to store all of the game objects' values. That'd be the best case scenario.*** (I'm on edge about this, we'd then have 3 or 4x the tables and the data base would be a mess to navigate and upgrade)

 

If that doesn't happen then yes, at the very least, in Beta 5 the binary blobs will be replaced with serialized json.

 

Overall I think storing all the game objects structure in tables is really the best choice. Everyone who wants to make web or mobile apps will benefit from this. I know if I was to develop a game with Intersect I would definitely want to develop some kind of companion app. It might require a lot of work developing the db and queries but I think you can find ways to mitigate the pain of upgrading. You can for example, abstract access to the db from the actual implementation. It would be great to see this change. Just my two-cents.

 

Edit: I guess if we have some access to an API in the future like you mentioned in your earlier post this wouldn't really matter that much! Whatever works, just one way to look it!

[jcsnider]

2 hours ago, Joyce said:

Upgrading would be messy maybe, but at the same time data would be far more accessible for development and tools outside the engine itself. (Also, searchable by queries allowing mass updates of stat weights, balancing revamps etc.) I'd say that's more important long term than upgradability for short-term patches that will become infrequent.

 

Also, you can alter tables and add values with a simple query on the database, so how is that harder than loading the old data, transposing it to a new class and then saving it to the DB like the current database works? yeah it's harder than having the Json lib figure it out for you but at least you'd be making proper use of the database system (and would allow for indexing on MySQL)

 

2 hours ago, Joyce said:

Upgrading would be messy maybe, but at the same time data would be far more accessible for development and tools outside the engine itself. (Also, searchable by queries allowing mass updates of stat weights, balancing revamps etc.) I'd say that's more important long term than upgradability for short-term patches that will become infrequent.

 

Also, you can alter tables and add values with a simple query on the database, so how is that harder than loading the old data, transposing it to a new class and then saving it to the DB like the current database works? yeah it's harder than having the Json lib figure it out for you but at least you'd be making proper use of the database system (and would allow for indexing on MySQL)

 

I understand the technical superiority of using our database as its intended to be used. I also understand how it would improve the lives of many developers (and potentially myself down the line). 

 

I'm okay with these changes being made. I do have concerns (Joyce mentioned a couple of them) for when this is done. But I have far more concerns regarding the transition itself. I foresee challenges that I cannot throughly explain in a forum post - this change would require cascading system overhauls all throughout the engine.  

 

Regardless, while I'm open to this happening, it's nothing something I really want and right now I see game breaking bugs and larger engine upgrades that could be done that'd take far less time. 

 

As a result, I'm not willing to commit the time to make this happen. But @panda might be going all the way. We haven't had a chance to talk so I'm not sure. I'd be rooting for him if I were you. 

[Miharukun]

Uhm, So what I need to do with the Json file? put it on the game file or..?

[jcsnider]

For now you don't need to do anything. If you want to make a translation for the new system you can start translating the new files above. This post is more meant to just give everyone a heads up on what I'm doing  

[jcsnider]

Just got done with the Jsonification of the UI.  Instead of 2 massive xml files (Menu.xml and InGame.xml) we now have 46 smaller/targeted files. Load times are quicker when opening each window and making modifications to specific windows should be better because it'll be easier to find the options that you want to edit.

 

 

 

[panda]

23 hours ago, jcsnider said:

 

 

I understand the technical superiority of using our database as its intended to be used. I also understand how it would improve the lives of many developers (and potentially myself down the line). 

 

I'm okay with these changes being made. I do have concerns (Joyce mentioned a couple of them) for when this is done. But I have far more concerns regarding the transition itself. I foresee challenges that I cannot throughly explain in a forum post - this change would require cascading system overhauls all throughout the engine.  

 

Regardless, while I'm open to this happening, it's nothing something I really want and right now I see game breaking bugs and larger engine upgrades that could be done that'd take far less time. 

 

As a result, I'm not willing to commit the time to make this happen. But @panda might be going all the way. We haven't had a chance to talk so I'm not sure. I'd be rooting for him if I were you. 

It's going to be a significantly slower process than that in all honesty. I still fully expect blobs to get stored, just less of them, and in the current iteration of the code I don't actually intend to change any of the table structures, only how we communicate with them, eventually I would be extracting more data from the blobs into their own columns. We shouldn't have any more tables that what we already have once I'm done though. 

My reason behind not going and converting all the blobs completely is because 1. json is easily parseable and human-readable (maybe even without us documenting its structure, and definitely once source is released) and 2. any dynamic lists would actually require extra tables, and I'm not sure it's worth the increased database complexity (and that code I'm much less willing to write, at least given the current state of the source base).

[Skaveron]

5 hours ago, panda said:

My reason behind not going and converting all the blobs completely is because 1. json is easily parseable and human-readable (maybe even without us documenting its structure, and definitely once source is released) and 2. any dynamic lists would actually require extra tables, and I'm not sure it's worth the increased database complexity (and that code I'm much less willing to write, at least given the current state of the source base).

 

Honestly, I'd be happy enough with JSON  

[jcsnider]

I kinda agree. If you do entirely JSON then you have to make 1 query to get what you need and then parse, and everything works the same way.

 

If you go half and half then you have to read all of the custom column values and then still do json parsing after the fact.

[panda]

3 hours ago, Skaveron said:

 

Honestly, I'd be happy enough with JSON  

 

3 hours ago, jcsnider said:

I kinda agree. If you do entirely JSON then you have to make 1 query to get what you need and then parse, and everything works the same way.

 

If you go half and half then you have to read all of the custom column values and then still do json parsing after the fact.

So while this is true you have to read in columns for several tables regardless, and the more import part is that you can't run search queries if it depends on a variable stored in the JSON blob. I'm definitely leaving lists in the JSON blobs, and will not move out fields into columns if they don't really make sense to be the target of a search query.

[BugSICK]

is there a specific time when this be ready?

[jcsnider]

Hopefully December sometime.. maybe Janurary. 

[nizate]

Man, I have to go learn json now lol... The change sounds awesome! I look forward to it!

[BugSICK]

any updates ?

[jcsnider]

All devs are taking a break for the Holidays. We're traveling and spending time with friends/family. Likely won't see any updates until after the New Year.