TripleA Project Update: 2.6 Status & Changes to Release Strategy

[ubernaut]

@lafayette is there any concern about players not being on the same subversions unless we force auto updates as well?

[LaFayette]

@cernel It could be merited to call 2.6 a 3.0; I don't know if I have that strong of an opinion overall that it is worth doing. The 'major' version (IIRC) is still meant to connote save-game compatibility. I would somewhat tend to think that we might be well just numbering the releases (eg: 1, 2, 3) and avoid the 'major.minor' system altogether. Though, a lot of ink has been spilled over version numbering. I don't think it'll be worthwhile to revisit version numbering for another release or two.

@ubernaut

is there any concern about players not being on the same subversions unless we force auto updates as well?

By 'subversion' do you mean the build version, eg: "2.5.123" vs "2.5.300"? If so, players have been on different subversions for quite some time already, and the subversions are compatible with one another. So, generally, no, not super concerned (though there will be a few challenges involved nonetheless).

As far as 'force auto update', we'll in theory do that slightly more often than every 2 years! If we have the latest version being released all the time, we can then toggle an auto-update. As-is, the latest was something we questioned and then could easily get into a non-releasable state. This prevented us from triggering the 'force update' function, and so it was triggered a lot less often than we would have liked.

[ubernaut]

@lafayette cool, in terms of the auto updates i was referring to an update on the launch of the client but it sounds like u are way ahead of me as per usual

hey just want to make sure there's nothing you need in terms of the start screen i think the consensus was to go back to using the standing logo rather than the current text treatment. please let me know if there's anything you need from me in that regard. also more than happy to make something fancier if we like.

please also let me know if there's anything else i can do to pitch in with the release efforts

[beelee]

Seems as if saved games may be a problem. Not so much for forum play, as most will probably already have triplea and be compatible, but in the lobby it might be.

Idk how many saves get used in the lobby. I would guess most would be from experienced users, so maybe not that big of an impact. Someone new would trigger everyone needing to update. Might be best to promote the older version player saving when possible.

Idk. As you said, a different approach needs to be tried. Just have to see how it goes and solve problems when they arise.

[Panther]

@lafayette said in TripleA Project Update: 2.6 Status & Changes to Release Strategy:

So, to help resolve this we will be changing to a constant release model. In this model, after every change, we will cut a brand new (full) release. That means whenever you download the game client, you will always be downloading the very latest version.

I like this, as it finally removes connotations bound to the terms "stable" and "unstable". Since the move to Github I have used the terms "current release" and "pre-release" towards end-users. Pre-releases have usually never been more "unstable" than "releases". In very rare cases when an undesired side-effect of a pull-request occurred, it had been removed by one of the next pre-release quite quickly. And savegame-incompatibilities have occurred rarely and carefully prepared and explained.

So switching to a constant release model is more or less underlining how the engine has evolved and will further evolve.

[LaFayette]

@beelee regarding save-game compatibility.

AFAIK current save-game compatibility goes back to 2.0; A 2.5 saved game should play fine on 2.0 and vice versa. There is a needless compatibility check based on version alone in the game (that was overly opinionated and removed in 2.6) that does prevent newer saves from being launched on older engines (eg: 2.5 save being played on 2.4). There is nothing physically otherwise preventing a 2.4 engine from playing a newer save, except that version check which we have removed in 2.6 (so 2.7 saves should be playable in 2.6)

Save games are difficult to keep compatible, really just very constraining for how much we can change the code around. This is a bad thing as it's not obvious when save games would break, and not being able to update the code structure means we have pretty inflexible designs that we tend to have to build on (which is dirty and makes things worse) compared to cleaning it up and doing things right. It's pretty high on the list to change how save games are done so we would not have these problems.

There is a build check that runs before we merge any code that validates existing saves can still be loaded by the engine. This has been really helpful to keep let us know when save game compatibility is lost. So, we have pro-actively been working to ensure that save games keep working.

There was a check in the engine that verified a save-game version was at or before the current engine version. This check is removed in 2.6 since very often there is nothing physically wrong with a future save (it can be played just fine on older 2.x engines).

All that is to say, as long as we keep version compatibility in the code, it does not matter which version a save is coming from, or which version the save is read into.

Further, if we do break compatibility, we'll have a partition for the lobby and will have multiple lobbies running. When we bump the version number to '2.7' for example, there will be a new '2.7' lobby and any '2.7' clients will automatically be routed there. Meanwhile the '2.6' lobby would continue to be there and so there is a capability there to keep similar clients together. Presumably when we break save-game compatibility it will be a '3.0' version.

That is to say, we can & will run multiple lobbies in parallel, one for each 'major.minor' version. Requests are automatically routed to the right lobby, and so players on a similar version will be kept in a cohort. We have the option too as well to force an upgrade by turning off an older lobby and that should generate a message to the client that they should update. This will likely be done when there are network incompatibilities. When we jump to a '3.0', we may do something like try to build an automatic save-game upgrader, or we'll just suffer incompatibility and keep a '2.x' lobby running to allow those games to be played out.

We have on the road map to change how save game compatibility is done so that it will be far easier to maintain. I suspect when we break save-game compatibility it will be so that we can move to this system. TBD how we will do that exactly. Overall though, there should be enough mechanisms in place where the compatibility will be explicit and we won't have cases where accidental version differences cause problems.

Generally, I'm not worried about save game compatibility for now and doing lots of releases. We've been able to maintain save-game compatibility for a good 3 years now, it's something we can continue until we are ready to re-write how save games are done.

[beelee]

@lafayette said in TripleA Project Update: 2.6 Status & Changes to Release Strategy:

There was a check in the engine that verified a save-game version was at or before the current engine version. This check is removed in 2.6

Right on so this error should no longer happen then ?

This is trying to run 2.5 with a 2.6 save.

[LaFayette]

@beelee Yeah, unless something is not as expected; 2.6 and beyond should no longer show that message (either the save will run, or it won't due to some other error; but the engine will stop refusing to even to try and load the save game)

[LaFayette]

Lobby is now upgraded to 2.6 (it was previously running 2.4)
Anyone joining the lobby from now on will be joining the 2.6 lobby (regardless of your game client version). 2.5 (game clients) and 2.6 (lobby) are fully compatible, so no worries.

For reference, these are the config updates that made this happen (cc: @RoiEX ) :

https://github.com/triplea-game/triplea/pull/11194
https://github.com/triplea-game/triplea/pull/11191

This unlocks a big step for releasing 2.6; Before had we released 2.6, the 2.6 players would have been joining a ghost town of a lobby (everyone would have been in the 2.5 lobby and nobody in 2.6). All players will very soon be in the 2.6 lobby regardless if they are upgraded (to 2.6) or not - thereby resolving that concern for the 2.6 migration.

[beelee]

@lafayette i can't connect to the lobby with 2.6 or 2.5. It times out

[LaFayette]

@beelee Looks like we are low on memory.

Logs show:

Nov 06 08:30:24 prod.triplea-game.org systemd[1]: Started lobby_2.6.
Nov 06 08:30:35 prod.triplea-game.org systemd[1]: lobby_2.6.service: Main process exited, code=exited, status=1/FAILURE
Nov 06 08:30:35 prod.triplea-game.org systemd[1]: lobby_2.6.service: Failed with result 'exit-code'.

Before this I saw an error for out of memory. I'm resizing the linode to have 4GB instead of 2GB and will turn off the 2.5 lobby as well to free up more memory (not that anyone can connect to it anyways).

With luck should be back up and running pretty soon.

[beelee]

@beelee hmm can't edit above here's error

[LaFayette]

@beelee Looks like memory might have been a red herring. DB password on 2.6 lobby is incorrect.

[beelee]

@lafayette ok I'm getting a 502 now

[LaFayette]

Looks like there were at least two issues:
(1) DB password got borked somewhere in the config. Repeatedly got 'incorrect password' even though it appears to be passed in to the system correctly. Eventually I was able to get around this.

(2) Old/bad code config was killing lobby startup. These needs a code rebuild. Waiting for that to happen. (patch: https://github.com/triplea-game/triplea/commit/24173591773b6ec7cbc54d26e17229778faf842b)

Once the patch is build, I expect to have to deal with bad lobby db password again. Once that is dealt with.. should be back to normal.

[beelee]

@lafayette i can get in to both now. No people or bots in 2.5

[LaFayette]

@beelee How are you getting into 2.5? hehe
Seems like 2.6 lobby is up now. Did have to deal with password issue. For reference: I wound up having to set the password directly in 'application.yml', the env variable from the systemctl file did not go through cleanly.

[beelee]

@lafayette same way as always : ) Ice in there now

[LaFayette]

@beelee 2.5 clients and 2.6 should both be connecting to 2.6 lobby.. It's not easy to tell which lobby you are in.

I'm getting a 502 bad gateway trying to connect now. Does not make sense since 2.6 lobby looks available on the server.

[LaFayette]

Not seeing a 502 anymore.
Though, it does look like 2.6 bots and 2.5 clients do not work together : |
We don't have much choice other than to have everyone download 2.6