Bot Crash 2026-09-20
-
Picking up from this thread: https://forums.triplea-game.org/topic/4357/triplea-server-costs/40
What happened:
- infrastructure needed an update to deploy SSH keys, which ran the infrastructure deployment. That is the server configs, all server config settings are applied at that time
- part of server configs, is server updates, those were run
- one server update, is to a system (a service) called 'docker', docker is currently running all of the tripleA applications
- a config setting to automatically restart services when they are updated, or any of their dependencies are updated is live in the system
- docker was updated, per the config setting, it was full restarted.
- as a result, all tripleA systems were rebooted (including bots), which would have ejected players from bots.
With the reboot, some bots may have come back up, I expect for some to crash again over time and would still be interested in error reports.
Since then, docker is now on a blacklist for services to be restarted and a secondary config was added that would allow applications running on the docker service to potential survive a reboot of docker as well. 3rd, there is a PR up to autosave when a bot shuts down.
Historically bot restarts had been for more direct causes. This was more "rare" as there are typically no package updates for docker.
I'm taking a quick look at the autosaves to see what the history is there.
-
L LaFayette referenced this topic
-
Bot Autosaves restored
- Autosaves in bots were missing in our latest codebase
- Autosaves were moved from their own dedicated "load autosave" button to the "select game" button in the hosting menu
- An update to "live-load" that list so that new maps were picked up dynamically regressed the feature and didn't add the autosaves to the "select game" list.
- As a result, there was no way to select autosaves.
This is now fixed via: https://github.com/triplea-game/triplea/pull/15002
The autosaves now do appear in the "select game" list, towards the bottom:
-
cc: @beelee / @cernel / @eyeswideshut ^
-
@LaFayette Cool. However, it does not seem to me that the place chosen for the auto-saves is very polished (kind of buried, and I guess might favour them being overlooked once again as a removed feature rather than a problem as opposed to have a dedicated space if not working). I think it would be better having a "Load Bot-Save" button right under the "Select Game" button.
By the way (minor detail), they are not actually "towards the bottom": they are before all games starting with lower case (which I guess are also something that should not be (?), all names ideally better starting upper case). I mean, this is what I see:
However, I do recall that bots are supposed to be supported only for a few years more and eventually phased out in favour of better (and cheaper) ways, and I'm sure bot-users are just fine as they are (and anyway I'm not a bot-user, so I'm not really the person who should give user-experience suggestions on the matter).
A problem with bots is that the very few users who report any problems are usually persons savvy enough to be able to host, so any problem with bots is likely not getting reported, unfortunately. I remember I was told this problem (auto-saves missing) the days after the release (months ago), and maybe I reported it somewhere or someone else did (don't remember). Maybe there is an issue still open in GitHub about this.
-
Cernel said:
Maybe there is an issue still open in GitHub about this.
-
@Cernel thanks, that issue is closed now.
-
-
The bots were stuck in a shutdown deadlock. A system exit call triggers the shutdown hook, and if that also invokes system exit then the system deadlocks. This could be triggered through the "admin shutdown" of a running game.
This update would fix the situation:
https://github.com/triplea-game/triplea/pull/15017That update should roll out to bots in between games (there's an automatic restart between games, the bots have pulled the latest code version, they'll activate the new version on the next game restart)
-
L LaFayette locked this topic
-
Do please let me know about other bot crashes:
- new post
- DM me here
- DM me on discord
- Create a github issue
Any of those work. I'm locking this thread as it's meant to be specific to the Sep-20 incident.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login