TripleA Project Update: 2.6 Status & Changes to Release Strategy
-
2.6 Status
The 2.6 lobby is getting close to being fully set up. It is now running and there is some supporting configuration for clients to be able to interface with it properly that is still needed. Once that is in place the broken map downloads for 2.6 clients will be fixed.
Getting all that done will be a pretty big milestone for being able to launch 2.6. If anyone would like to get up close and nitty with what is left for us to launch 2.6, check the 'MVP' column of the tracking project: https://github.com/orgs/triplea-game/projects/2
Release Cycle Changes
To cut to the chase, instead of releasing every 2 years, every time the code is changed we will be launching (automatically) a full release versions. Players downloading the game will always be downloading the latest release version.
With 2.6 we will also be changing the release model of TripleA. The 2.6 launch was meant to be about 5 months behind 2.5, and that was more than 2 years ago now. Unable to call that anything other than a failure, we seemingly need to do something different and better.
TripleA way back in the day (when we were still in SVN) used a 'unstable' 'stable' release model. When we migrated to Github and automated the release process, we evolved this to be a series of prerelease versions that were generally stable, followed by blessing specific versions that we considered as releases.
Along the way we would introduce incompatible changes, break things, and generally operate in a codebase that was very hostile to getting things done. There have been various incompatibility issues that we had to address & various issues with how to do infrastructure (the servers) that have held up 2.6. We have also dealt with various bugs that were came in from merges that happened more than a year ago.
So, to help resolve this we will be changing to a constant release model. In this model, after every change, we will cut a brand new (full) release. That means whenever you download the game client, you will always be downloading the very latest version.
Constant Release FAQ
Why?
If every time we change the code is a release, there is no way for us to get behind on releasing, ABR - "always be releaasing". Really this is to solve the problem where we have previously needed 1 to 2 years between releases, and this is just too long. We have always wanted to release about every 3 to 6 months. Despite good intentions, something was pretty deeply broken that was slowing us down.
Will I have to download a new game version all the time?
No, we will continue incrementing the "minor" version of the game periodically when it is recommended to update your game engine. So, every 3 months or so we'll bump to a "2.7", and so forth which will then trigger the game to notify players that they should upgrade. We'll do this when there are a large number of bugfixes and other improvements where it is worthwhile that everyone pick up the latest.
Meanwhile, we'll continue keeping compatibility when we update the game, which means all of the coming game versions will continue working together, and players will (as today) not notice that they are not quite exactly on the same version of the game.
Will Developers be Constantly Breaking the Game?
Maybe. It's a damn'ed spaghetti code base in there, so.. it's gonna happen.
With luck, the constant release model means that we can deploy fixes just as quickly as we can break things. So.. any fixes will be fast to follow, and that is a positive.
We'll be asking developers to make heavy use of 'feature-flag' gating behind any feature work. This means that while new code is shipped, it will be disabled for any of the vanilla users. We will build in feature-toggles into the options (which exist today in the 'testing' settings in engine preferences) to enable these new features, so they will be opt-in while they are being worked on. This kind of technique should give a lot of stability while we are working on the game.
Further, we'll emphasize that with changes going live, that they need to be solid. It'll be easier to ask developers to simply spend more diligence per change to help make sure the updates are solid.
-
I wonder if going from the current release to this new one would rather warrant a 3.0 versioning instead of 2.6. I mean, the 2.5 to 2.6 looks like such a bigger jump than any of the 2.0 to 2.1 to 2.2 to 2.3 to 2.4 to 2.5 and also than them all combined.
-
@lafayette is there any concern about players not being on the same subversions unless we force auto updates as well?
-
@cernel It could be merited to call 2.6 a 3.0; I don't know if I have that strong of an opinion overall that it is worth doing. The 'major' version (IIRC) is still meant to connote save-game compatibility. I would somewhat tend to think that we might be well just numbering the releases (eg: 1, 2, 3) and avoid the 'major.minor' system altogether. Though, a lot of ink has been spilled over version numbering. I don't think it'll be worthwhile to revisit version numbering for another release or two.
is there any concern about players not being on the same subversions unless we force auto updates as well?
By 'subversion' do you mean the build version, eg: "2.5.123" vs "2.5.300"? If so, players have been on different subversions for quite some time already, and the subversions are compatible with one another. So, generally, no, not super concerned (though there will be a few challenges involved nonetheless).
As far as 'force auto update', we'll in theory do that slightly more often than every 2 years! If we have the latest version being released all the time, we can then toggle an auto-update. As-is, the latest was something we questioned and then could easily get into a non-releasable state. This prevented us from triggering the 'force update' function, and so it was triggered a lot less often than we would have liked.
-
@lafayette cool, in terms of the auto updates i was referring to an update on the launch of the client but it sounds like u are way ahead of me as per usual
hey just want to make sure there's nothing you need in terms of the start screen i think the consensus was to go back to using the standing logo rather than the current text treatment. please let me know if there's anything you need from me in that regard.
also more than happy to make something fancier if we like.please also let me know if there's anything else i can do to pitch in with the release efforts
-
Seems as if saved games may be a problem. Not so much for forum play, as most will probably already have triplea and be compatible, but in the lobby it might be.
Idk how many saves get used in the lobby. I would guess most would be from experienced users, so maybe not that big of an impact. Someone new would trigger everyone needing to update. Might be best to promote the older version player saving when possible.
Idk. As you said, a different approach needs to be tried. Just have to see how it goes and solve problems when they arise.
-
@lafayette said in TripleA Project Update: 2.6 Status & Changes to Release Strategy:
So, to help resolve this we will be changing to a constant release model. In this model, after every change, we will cut a brand new (full) release. That means whenever you download the game client, you will always be downloading the very latest version.
I like this, as it finally removes connotations bound to the terms "stable" and "unstable". Since the move to Github I have used the terms "current release" and "pre-release" towards end-users. Pre-releases have usually never been more "unstable" than "releases". In very rare cases when an undesired side-effect of a pull-request occurred, it had been removed by one of the next pre-release quite quickly. And savegame-incompatibilities have occurred rarely and carefully prepared and explained.
So switching to a constant release model is more or less underlining how the engine has evolved and will further evolve.
-
@beelee regarding save-game compatibility.
AFAIK current save-game compatibility goes back to 2.0; A 2.5 saved game should play fine on 2.0 and vice versa. There is a needless compatibility check based on version alone in the game (that was overly opinionated and removed in 2.6) that does prevent newer saves from being launched on older engines (eg: 2.5 save being played on 2.4). There is nothing physically otherwise preventing a 2.4 engine from playing a newer save, except that version check which we have removed in 2.6 (so 2.7 saves should be playable in 2.6)
Save games are difficult to keep compatible, really just very constraining for how much we can change the code around. This is a bad thing as it's not obvious when save games would break, and not being able to update the code structure means we have pretty inflexible designs that we tend to have to build on (which is dirty and makes things worse) compared to cleaning it up and doing things right. It's pretty high on the list to change how save games are done so we would not have these problems.
There is a build check that runs before we merge any code that validates existing saves can still be loaded by the engine. This has been really helpful to keep let us know when save game compatibility is lost. So, we have pro-actively been working to ensure that save games keep working.
There was a check in the engine that verified a save-game version was at or before the current engine version. This check is removed in 2.6 since very often there is nothing physically wrong with a future save (it can be played just fine on older 2.x engines).
All that is to say, as long as we keep version compatibility in the code, it does not matter which version a save is coming from, or which version the save is read into.
Further, if we do break compatibility, we'll have a partition for the lobby and will have multiple lobbies running. When we bump the version number to '2.7' for example, there will be a new '2.7' lobby and any '2.7' clients will automatically be routed there. Meanwhile the '2.6' lobby would continue to be there and so there is a capability there to keep similar clients together. Presumably when we break save-game compatibility it will be a '3.0' version.
That is to say, we can & will run multiple lobbies in parallel, one for each 'major.minor' version. Requests are automatically routed to the right lobby, and so players on a similar version will be kept in a cohort. We have the option too as well to force an upgrade by turning off an older lobby and that should generate a message to the client that they should update. This will likely be done when there are network incompatibilities. When we jump to a '3.0', we may do something like try to build an automatic save-game upgrader, or we'll just suffer incompatibility and keep a '2.x' lobby running to allow those games to be played out.
We have on the road map to change how save game compatibility is done so that it will be far easier to maintain. I suspect when we break save-game compatibility it will be so that we can move to this system. TBD how we will do that exactly. Overall though, there should be enough mechanisms in place where the compatibility will be explicit and we won't have cases where accidental version differences cause problems.
Generally, I'm not worried about save game compatibility for now and doing lots of releases. We've been able to maintain save-game compatibility for a good 3 years now, it's something we can continue until we are ready to re-write how save games are done.
-
@lafayette said in TripleA Project Update: 2.6 Status & Changes to Release Strategy:
There was a check in the engine that verified a save-game version was at or before the current engine version. This check is removed in 2.6
Right on so this error should no longer happen then ?
This is trying to run 2.5 with a 2.6 save.
-
@beelee Yeah, unless something is not as expected; 2.6 and beyond should no longer show that message (either the save will run, or it won't due to some other error; but the engine will stop refusing to even to try and load the save game)
-
Lobby is now upgraded to 2.6 (it was previously running 2.4)
Anyone joining the lobby from now on will be joining the 2.6 lobby (regardless of your game client version). 2.5 (game clients) and 2.6 (lobby) are fully compatible, so no worries.For reference, these are the config updates that made this happen (cc: @RoiEX ) :
https://github.com/triplea-game/triplea/pull/11194
https://github.com/triplea-game/triplea/pull/11191This unlocks a big step for releasing 2.6; Before had we released 2.6, the 2.6 players would have been joining a ghost town of a lobby (everyone would have been in the 2.5 lobby and nobody in 2.6). All players will very soon be in the 2.6 lobby regardless if they are upgraded (to 2.6) or not - thereby resolving that concern for the 2.6 migration.
-
@lafayette i can't connect to the lobby with 2.6 or 2.5. It times out
-
@beelee Looks like we are low on memory.
Logs show:
Nov 06 08:30:24 prod.triplea-game.org systemd[1]: Started lobby_2.6. Nov 06 08:30:35 prod.triplea-game.org systemd[1]: lobby_2.6.service: Main process exited, code=exited, status=1/FAILURE Nov 06 08:30:35 prod.triplea-game.org systemd[1]: lobby_2.6.service: Failed with result 'exit-code'.Before this I saw an error for out of memory. I'm resizing the linode to have 4GB instead of 2GB and will turn off the 2.5 lobby as well to free up more memory (not that anyone can connect to it anyways).
With luck should be back up and running pretty soon.
-
@beelee hmm can't edit above here's error
-
@beelee Looks like memory might have been a red herring. DB password on 2.6 lobby is incorrect.
-
@lafayette ok I'm getting a 502 now
-
Looks like there were at least two issues:
(1) DB password got borked somewhere in the config. Repeatedly got 'incorrect password' even though it appears to be passed in to the system correctly. Eventually I was able to get around this.(2) Old/bad code config was killing lobby startup. These needs a code rebuild. Waiting for that to happen. (patch: https://github.com/triplea-game/triplea/commit/24173591773b6ec7cbc54d26e17229778faf842b)
Once the patch is build, I expect to have to deal with bad lobby db password again. Once that is dealt with.. should be back to normal.
-
@lafayette i can get in to both now. No people or bots in 2.5
-
@beelee How are you getting into 2.5? hehe
Seems like 2.6 lobby is up now. Did have to deal with password issue. For reference: I wound up having to set the password directly in 'application.yml', the env variable from the systemctl file did not go through cleanly. -
@lafayette same way as always : ) Ice in there now
