TripleA Server Costs
-
@beelee My understanding is that TripleA has 5 servers on which the bots are located. The servers are all working but currently have 2 bots each instead of the 3 each @lafayette had them having. I've no idea if that is because 3 bots per server are just too much (so it has to be 2 as it currently practically is) or there is a problem or some problems with the bots which @lafayette (or an other developer) may fix eventually (maybe with help from @eyeswideshut).
Not sure if we should make a problem report in GitHub about this or if we should notify anyone anywhere when we see that some bots are broken.
-
Yea I'm not certain either but like you, I thought we were supposed to have 15.
The reason I mention it again, besides potentially paying for a service we aren't receiving, there's usually only 1 available bot or sometimes none.
I'm glad the lobby still popular but I was hoping to play on it again since new maps are now supported.
Anyway, it seemed noteworthy to mention again
-
15 bots are back.
Setting update has been deployed:
https://github.com/triplea-game/infrastructure/commit/54f11b2da26c9b9a7a18079a0489fdc76dc2c756The act of deploying an update would have booted the missing bots. They would have been crash/disconnected due to something within our control (system config, code issues, something we own).
I don't know if that setting will have an impact on the lobby bots. I do see that they are not full on restarting when a game is over, that seems more stable.
The restart of lobby bots was otherwise handled by systemctl rather than docker. Maybe a key difference, maybe not, I'm not sure.
I'll be interested if bots drop off again.
-
Hi Dan
sadly we had a bot malfunction. I was in Cali 103 and the game kinda just stopped. But wasn't frozen. I checked lobby and my bot was gone
as you can see no 103 and it looks as if we're down to 12.
Here's in game that is still showing 103 though after it dropped from lobby.
A couple guys said they had froze up at the same time.
Anyway, just a heads up.
Edit
Just checked and now they're back.
Edit 2 jusst checked in a gain about 3 1/2 hours and they are gone once more.
No Cali 103. Idk when they left again. Was about a 30 minute window first time I checked and it came back.
-
Did some work on the bots. ..
Turns out we need EXIT_GAME_ON_QUIT (or whatever it was...) to be true. Memory leaks killed the bots quickly and they crashed out hard after a few games.
I did some other things like put memory limits in and added swap. Each bot is now running with 128M, which is less, but there's more padding for when there is memory pressure. Hopefully wont' be taking down the whole bot instance any longer.
Let me know if bots drop out.. should be a bit better now.
-
Sweet
Yea I saw that on git. Quit seems counter intuitive but hey I was just a carpenter lol
-
-
@beelee Yes, all TripleA hosts are seemingly now working except only the third one (Bot_203_Jersey is missing.) of Jersey (@lafayette).
-
It seems that the problem I had has happened to these guys. I didn't talk to them, just saw their conversation
-
-
@lafayette Bot_302_Toronto was in progress with only 1 (unresponsive) player inside.
Maybe the guy fell asleep or something. Anyway, I did a shutdown (restarted) the bot.
And, yeah, all servers except California were down to 2 bots only (instead of the 3 each they should have).
Unfortunately, after I clicked on "Shutdown" on Bot_302_Toronto, it never came back, so now the Toronto server is down to 1 bot.
I remember that, if you clicked "Shutdown" on a bot, the bot would restart and return available in lobby after a few seconds.
Did it not come back because the bot was broken or did I fuck up because now shutting down a bot just kills it and does not come back, and I didn't know? Should any moderator shutdown apparently broken bots or not?
-
The bots just mass-restarted a few minutes ago.
The good news is that now they are all 15 up and running.
However, I'm under the understanding that everyone who was in a bot at the moment of the mass-restart was disconnected and unable to save its game. If so, that is rather brutal a way to have the bots all back...
I don't actually know for certain what happened: I never play in bots and was not in a bot when that happened. I just saw all bots disappear together in lobby list and then come back all together with also those missing. This is what was going on in lobby right after that happened:
-
Do you know what triggered the mass restart ? That's not good with everyone getting disconnected.
It's actually really bad.
@lafayette you probably want to check this out
-
The lobby restarted. No bots should have. I'm checking in whether there were some one-offs. In general, nobody should have been disconnected from a bot game and we need to be 1000% clear whether a player was simply disconnected from lobby or the bot game. "I was disconnected" could mean the lobby, and that would be expected.
No autosaves should have been lost either.
Also, why are we talking about lobby issues in a "TripleA Server Costs" thread?
edit: updated "no bots did" to "no bots should have"
-
was just replying to Cernels screenshot
-
-
There were server changes, they apply wholesale and part of that are server package upgrades. 'Docker' which runs the bots was part of that and was restarted. No autosaves should have been lost though, except no autosaves are expected to be cut at the time of the restart either.
So, this event is fairly rare, there was a misconfig that looped the bots into the blast radius, and the autosaves would have been slightly dated (ie: beginning of round rather than at moment of restart).
There's 3 fixes that can be applied here:
- don't restart docker, that can be added to a blacklist for restart-on-package-upgrade
- decouple the docker containers from the docker service, we can add a config that detaches the containers and bounces the service, and then re-attaches them (by default the containers are taken down and then rebooted)
- add a shutdown hook to the game code for bots, to capture an autosave when the game shuts down.
-
@beelee As I said, I don't know what actually happened. I was not in a bot. Just in lobby seeing all bots disappear and reappear and the talking in lobby which I mostly showed here with the screenshot I posted. Maybe @eyeswideshut knows more about this. (The best would be that somebody who was playing in a bot at the moment comes here and tells us what he/she experienced: if he/she lost part of its game and how much of it was save if it was, but that is very unlikely to happen.)
To be clear, I'm not saying that was all negative: there were 10 out of 15 bots in the lobby and a few users were unable to play games for lacking available bots, so this restart actually fixed all bots, a third of which were broken, and brought all them back fresh and available.
Personally, I was thinking to suggest a count-down starting 1 minute before this event, with the bot itself telling that it will restart itself in 1 minute and inviting everyone inside to save the game or letting the shut-down games remaining open on the user's side as dead games which however allow taking a save. However, I've seen that @lafayette proposed a bunch of solution which are likely better and make my suggestions unnecessary (but I don't fully understand them).
-
No autosaves should have been lost though, except no autosaves are expected to be cut at the time of the restart either.
In the last screenshot I posted, @eyeswideshut said "cernel my understanding is that autosave was removed from triplea".
I don't know if autosaves work: I've never playtested any bots since the release.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login