RFC: Save Games as Game-Deltas
-
Overview
This project would be to convert how save games are stored, and in turn how the 'GameData' object represents the current game state. Instead of storing the current state in the singular 'GameData' object, and serializing that to disk, instead the game state would be the aggregate of a "replay" of a sequential series of game deltas. To store save games, we would store the base map name and then the series of game deltas.
To illustrate this difference, instead of storing something like "there is are 2 infantry in Manchuria, 3 artillery in Japan", we would store "the map is revised, delta 1 = move 1 infantry from western russia to russia, delta 2 = move 1 artillery from US west coast to central US"
Another analogy would be how banks store account balances. Banks do not store "you have $5", they instead store: "your bank account opened with $10, you withdrew $10 on 10/02, and you deposited $5 on 11/02". By "replaying" this ledger, you wind up with the same "I have $5".
This idea was touched on in this thread:
https://forums.triplea-game.org/topic/2794/triplea-3-0-design-proposal-discussion/50Background: GameData serialization & and key benefits
Compatibility & Freedom to Change code
Currently save games are a dump to disk of GameData. The dump to disk is a mapping of the internal java classes, objects & variables of that represent the GameData object and everything the GameData object references. Changing any variables, methods, classes that are already serialized breaks the process of reading this data back from disk into running code. This means that a simple update of a variable name crashes the load-game process. Worse yet, save games are kept around for months, which means we are extremely limited to be able to update GameData and anything it references without breaking save games.
A nice aspect of having game deltas is that they would be represented in plain text. This means we'd have a single parser module that could read a variety of legacy game-delta formats and would be able to convert that text information into the latest java representation of the game code. This is a case of adding a layer of indirection between how the data is represented on disk compared to how it is represented when the program is running.
Further, the game keeps a separate tally of the history. Having a stack of game deltas would be very natural for changing to intermediate game steps, to going back in history and forward very naturally (without using what is otherwise a parallel data structure)
Save game file size
Another key benefit, the save game file size would be a lot smaller. Currently the dump of java objects is a lot of excessive information and is not a nice and easily compressible text representation of the game. Currently a late stage game can easily be 5MB when saved, going to a text based format should get us down to a couple hundred KB or so. The game-save information is often transmitted over network, during game, repeatedly - so using game deltas will help cut that lagginess down.
Better network communication
Further, the network layer for multiplayer games can be simpler when we transmit game deltas as actions that a user has taken. This is a nice & generic & simple way to communicate about game state changes compared to how it is currently done.
-
well then ya
-
-
Backwards compatibility is good, but I think if a new system is needed, let save games be broken. Possibly build a conversion utility so legacy games may be converted to/from the new version.
-
"represented in plain text" - and there was reference elsewhere of having save games in text. Is the only purpose of this to make games easily human-readable? My thought is, if objects and such are easier for the program to read, then leave the data in objects - but build in a function to convert such objects to human readable.
I expect there may be some other reason for plain text that I'm not thinking of, or some complication with the system I just described.
- Re: network communication and sending deltas - mostly, not an issue. But suppose there were a tournament game for high stakes, and one player's data somehow got corrupted. If data were tracked only by deltas, the error would only be discovered when a reference was made to a unit that didn't exist on one player's board, if then. By that point, the game might have progressed well beyond the point of data error.
Given that various safeguards would make that quite rare, is that acceptable? Or should players have an option (or should it be standard) that at least one complete game state be transmitted per sending?
-
-
Re: compatibility - it breaks easily & a lot
Backward compatibility is very good indeed. Though, it is not the be-all & end-all. This won't be the first time we've had to deal with incompatible save versions. So, while we are willing to release versions of TripleA that are incompatible, if we were not very diligent about maintaining backward compatibility, it would be broken all the time (on a weekly basis). If we were to start releasing more often again, once every 3 months, nobody wants save games to be incompatible every 3 months. That is not even considering the intermediate releases which would all largely be incompatible with one another.
That is to say it is expensive to maintain compatibility on a near constant basis. The efforts we have to take to maintain compatibility are also impactful themselves and disallow a large class of changes that would make all future development more efficient. So not only are we prevented from speeding up development for future efforts, but we have to pay an ongoing tax on all future development to maintain compatibility.
Literally 2.6 would already be launched if it were not for Java Serialization in TripleA. That is just the tip of the iceburg of serialization slowing things down for development.
The root of evil - Industry Perspective On Java Serialization
In 2018, Oracle's chief architect had this to say about Java Serialization:
[Serialization] was a horrible mistake in 1997," he said. "Some of us tried
to fight it, but it went in, and there it is. ...We like to call serialization
'the gift that keeps on giving,'The same article mentions that there are plans for Java Serialization to be removed entirely from Java. This other set of slides does a nice job of describing some of the aspects why Java Serialization is considered harmful: https://stuartmarks.files.wordpress.com/2021/04/java-serialization-20191105a.pdf
The TripleA Java Style Guide says on line 3: "Generally follow: Effective Java". Effective java is something of a bible for writing good Java code. It was assigned as required reading in one of the CS courses I took. In Effective Java, one of the recommendations Joshua Bloch wrote: “Prefer alternatives to Java serialization”
Java Serialization vs Plain Text
The reason for plain text is because it is a good alternative to Java Serialization. In some senses, serializing to plain text is serialization. Java serialization is a specific implementation of serialization that uses a binary format which is deeply tied to very nitty details of the code.
Hence:
A nice aspect of having game deltas is that they would be represented in plain text. This means we'd have a single parser module that could read a variety of legacy game-delta formats and would be able to convert that text information into the latest java representation of the game code. This is a case of adding a layer of indirection between how the data is represented on disk compared to how it is represented when the program is running.
The net effect is the internal code representation would no longer be deeply tied to binary data that is saved in a save game file. The code being tied to this representation means we cannot change it. What is worse, nothing tells you that it is unsafe to change, you only find out either in a code review, or when a user reports an error that a save game won't load, or when you do the full launch of 2.6 to find out that bots are not working.
Instead of Java Serialization, plain text would allow us to have a very explicit module that has a lot of flexibility for how it reads this data and converts it to the latest code representation of game state. This would be explicit, easily tested, and more easily understand. Further, instead of the tentacles of Java serialization reaching into the deepest layers of code, we would be a lot more free to change up the structure. As an analogy, it's a case where you can't change the wind shield wiper blade of a car becauseit is tied to the engine block which is tied to every other part of the car except for the paint on the rear doors. This is not at all an intrinsic property of software, in general, software is actually easy to change.
Bottom line, the advice of Oracle's Chief Architect is: "Reinhold said, users [Java developers] would be better off using JSON or XML."
https://adtmag.com/articles/2018/05/30/java-serialization.aspx#:~:text=Serialization is brittle%2C it pokes,use in simple use cases.That is essentially what we are doing, the plain text format of game deltas very likely would be JSON.
Game Data Integrity
Data integrity is perhaps a bigger problem today than it would be going forward. TCP/IP generally comes into play here though and helps ensure that game data transmissions are accurate (so you don't get too many cases of bits flipping without being corrected). Though, the game engine does not have anything to prevent game data state transmissions from occurring in the wrong order. Generally though the impact is on game clients which just get a ton of lag. TripleA treats the servers copy of GameData as authoritative and this will get transmitted to clients. Clients then need to wait for this download for the UI to complete its movements.
Network deltas would be far smaller and could be compressed before being transmitted. Instead of needlessly sending the full game state, only the needed deltas would be transmitted. This is the differnce of transmitting 2kB of data vs 5MB. We could add checksums to the deltas to help ensure that they are received in order (we could use sequence numbers or checksums in a very block-chain like manner).
Java Serializations Cost to TripleA
It's hard to fully explain how [explitive deleted] expensive Java Serialization has been to TripleA. I'd estimate that it has reduced development speed by about 60-80%. For example, a robust testing framework is key for fast development. A simple test like, "create two units and have them do combat & validate that they have the correct attack & defense powers" is expensive. Each unit in-code needs to have a full game-data reference. Because of this, instead of just simply creating two units with about 5 lines of code, instead you have to load in a full Game-XML to get that full game context. The test case that could otherwise run in a few milliseconds now needs a few full seconds (which is bad considering we should be targetting to have several tens of thousands of test cases and we want them to run in under 20s). Then.. if you want to vary the rules a bit, you have to get yet another full game-XML.
What winds up happening is we tend to not have fully exhaustive tests, which means we spend more time validating & inspecting code changes to ensure we have not broken anything, and we spend way more time manually testing after every code update (very expensive) & fixing corner case regressions.
Also, it's worth considering that these test XMLs would be several hundred lines long, a thing to maintain in of itself. And.. why do units need a full reference to GameData? Because of serialization.
This is just one example. Serialization prevents us from even moving source code files. Rather than being able to re-group related files together, they are stuck in the folder where they were originally places 15 to 20 years ago.
Java serialization prevents a lot of valuable updates from occurring. This rigidity prevents new collaborators from having an efficient and valuable contribution which means even less development occurs on TripleA beyond the inefficiencies and pure overhead that is incurred as a time cost.
Performance Impact of Java Serialization
Java serialization makes heavy use of Java Reflection. Java reflection is notoriously slow. IIRC in 2.5 we greatly sped up map parsing. War of the Relics used to take about 45s to load. I t now loads in under 2s. A big part of the speed up was in getting rid of Java Reflection.
The AI spends the majority of its time doing battle simulations (as much as 80% of the compute time for AI is doing this). The majority of time spent in a battle simulation is simply copy GameData, which uses Java Serialization & makes heavy use of Java Reflection.
If we were able to instead copy GameData using plain text, we are potentially removing all of this delay. That would mean AI would play the game about 4 times faster.
TL;DR
This is all about getting rid of Java Serialization.
-
Considering Alternatives
Serialization Format
The choices are (AFAIK);
- Java Serialization
- Google Protobuf
- Plain Text (XML / JSON / YML)
Java Serialization is evil. Protobuf is not a good fit. Which leaves plain text. Not a lot of alternatives to consider per-say.
Game Data Representation - Full state vs Deltas
It's worth considering that the game data already keeps track of a game history, which is essentially already a stack of game deltas. Rather than having GameData be the current state plus a set of game deltas, instead we can have GameData be computed based on the set of game deltas.
So, rather than serializing the game data plus the game history, instead we could serialize the game history only and then compute the game data.
The code already has a concept of a
Changeobject, which is effectively a game delta. The architectural design to use these change objects in a ledger-like system is not at all fully baked into triplea, it's done very piecemeal. If we do bake it fully, then we can get the benefits of a GameData to be the sum of deltas, rather than a game-data being the set of current data values plus a set of deltas. -
I'll have to read up more on Java serialization, thanks for the extremely useful links provided. Especially the wordpress with those specific examples.
Noted on other points, too many to acknowledge individually.
-
@aardvarkpepper FWIW, this is all part of an effort to remove components that are not compile-time safe. Meaning you have to run the code before you get to see it break, rather than just having it break at compile time.
Previously XML was mapped to game objects via reflection. We have removed that.
Network data transfer is done via a customized implementation of Java Serialization that uses Java Reflection sprinkled in. The 'network relay' project attempts to address that.
The remaining piece is serialization of save games. This would be the project to address that.
