TripleA Server Costs
-
Did some work on the bots. ..
Turns out we need EXIT_GAME_ON_QUIT (or whatever it was...) to be true. Memory leaks killed the bots quickly and they crashed out hard after a few games.
I did some other things like put memory limits in and added swap. Each bot is now running with 128M, which is less, but there's more padding for when there is memory pressure. Hopefully wont' be taking down the whole bot instance any longer.
Let me know if bots drop out.. should be a bit better now.
-
Sweet
Yea I saw that on git. Quit seems counter intuitive but hey I was just a carpenter lol
-
-
@beelee Yes, all TripleA hosts are seemingly now working except only the third one (Bot_203_Jersey is missing.) of Jersey (@lafayette).
-
It seems that the problem I had has happened to these guys. I didn't talk to them, just saw their conversation
-
-
@lafayette Bot_302_Toronto was in progress with only 1 (unresponsive) player inside.
Maybe the guy fell asleep or something. Anyway, I did a shutdown (restarted) the bot.
And, yeah, all servers except California were down to 2 bots only (instead of the 3 each they should have).
Unfortunately, after I clicked on "Shutdown" on Bot_302_Toronto, it never came back, so now the Toronto server is down to 1 bot.
I remember that, if you clicked "Shutdown" on a bot, the bot would restart and return available in lobby after a few seconds.
Did it not come back because the bot was broken or did I fuck up because now shutting down a bot just kills it and does not come back, and I didn't know? Should any moderator shutdown apparently broken bots or not?
-
The bots just mass-restarted a few minutes ago.
The good news is that now they are all 15 up and running.
However, I'm under the understanding that everyone who was in a bot at the moment of the mass-restart was disconnected and unable to save its game. If so, that is rather brutal a way to have the bots all back...
I don't actually know for certain what happened: I never play in bots and was not in a bot when that happened. I just saw all bots disappear together in lobby list and then come back all together with also those missing. This is what was going on in lobby right after that happened:
-
Do you know what triggered the mass restart ? That's not good with everyone getting disconnected.
It's actually really bad.
@lafayette you probably want to check this out
-
The lobby restarted. No bots should have. I'm checking in whether there were some one-offs. In general, nobody should have been disconnected from a bot game and we need to be 1000% clear whether a player was simply disconnected from lobby or the bot game. "I was disconnected" could mean the lobby, and that would be expected.
No autosaves should have been lost either.
Also, why are we talking about lobby issues in a "TripleA Server Costs" thread?
edit: updated "no bots did" to "no bots should have"
-
was just replying to Cernels screenshot
-
-
There were server changes, they apply wholesale and part of that are server package upgrades. 'Docker' which runs the bots was part of that and was restarted. No autosaves should have been lost though, except no autosaves are expected to be cut at the time of the restart either.
So, this event is fairly rare, there was a misconfig that looped the bots into the blast radius, and the autosaves would have been slightly dated (ie: beginning of round rather than at moment of restart).
There's 3 fixes that can be applied here:
- don't restart docker, that can be added to a blacklist for restart-on-package-upgrade
- decouple the docker containers from the docker service, we can add a config that detaches the containers and bounces the service, and then re-attaches them (by default the containers are taken down and then rebooted)
- add a shutdown hook to the game code for bots, to capture an autosave when the game shuts down.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login