TripleA Server Costs

[Cernel]

@LaFayette said:

Good points, we need to also keep in mind that the bots need to be available for peak lobby activity.

@lafayette Yes, it looks like that at peak 15 bots are easily all taken. I've seen it happening despite the 2 additional bots courtesy of @eyeswideshut.

Unfortunately, the lobby is back having only 10 out of 15 TripleA bots working. Every server seems missing 1 of its 3 bots. The missing bots are "101", "202", "303", "403" and "501".

Maybe @eyeswideshut can help you making the bots more stable? It made a post with information about that right before this one.

CC: @frigoref

[beelee]

@Cernel

There is currently only one available bot

[EyesWideShut]

@Cernel OK! So I think I solved the bot crashing problem on my end. It was the script function, and the fix was actually very simple.

The problem was that I had this command in the Docker startup script -- and it's probably in all the bots:

--env EXIT_ON_GAME_END=true \

That tells the bot to exit when the game ends. Since the container was configured with --restart always, I was expecting Docker to bring it back, but this was creating problems with the bot disappearing from the lobby.

I removed that command from the startup scripts.

For example, the relevant part went from:

sudo docker run
--network host
--pull always
--restart always
--env BOT_PORT_NUMBER=3301
--env BOT_NAME=Bot_Revised_Ladder
--env BOT_COMMENT=bit.ly/RevLad2
--env LOBBY_URI=https://prod.triplea-game.org
--env EXIT_ON_GAME_END=true
--env BOT_MAX_MEMORY=256m \

to:

sudo docker run
--network host
--pull always
--restart always
--env BOT_PORT_NUMBER=3301
--env BOT_NAME=Bot_Revised_Ladder
--env BOT_COMMENT=bit.ly/RevLad2
--env LOBBY_URI=https://prod.triplea-game.org
--env BOT_MAX_MEMORY=256m \

And I did the same for the second bot on port 3302.

The two bots are now running normally, so I'm going to see whether they remain in the lobby after their next games. Hopefully that's all it was! check to see if the other bots have this function..

[beelee]

I'm consistently seeing only 10 triplea bots in the lobby. If triplea is paying for 15, then this is something that should be addressed sooner than later imo

[Cernel]

@beelee My understanding is that TripleA has 5 servers on which the bots are located. The servers are all working but currently have 2 bots each instead of the 3 each @lafayette had them having. I've no idea if that is because 3 bots per server are just too much (so it has to be 2 as it currently practically is) or there is a problem or some problems with the bots which @lafayette (or an other developer) may fix eventually (maybe with help from @eyeswideshut).

Not sure if we should make a problem report in GitHub about this or if we should notify anyone anywhere when we see that some bots are broken.

[beelee]

@Cernel

Yea I'm not certain either but like you, I thought we were supposed to have 15.

The reason I mention it again, besides potentially paying for a service we aren't receiving, there's usually only 1 available bot or sometimes none.

I'm glad the lobby still popular but I was hoping to play on it again since new maps are now supported.

Anyway, it seemed noteworthy to mention again

[LaFayette]

15 bots are back.

Setting update has been deployed:
https://github.com/triplea-game/infrastructure/commit/54f11b2da26c9b9a7a18079a0489fdc76dc2c756

The act of deploying an update would have booted the missing bots. They would have been crash/disconnected due to something within our control (system config, code issues, something we own).

I don't know if that setting will have an impact on the lobby bots. I do see that they are not full on restarting when a game is over, that seems more stable.

The restart of lobby bots was otherwise handled by systemctl rather than docker. Maybe a key difference, maybe not, I'm not sure.

I'll be interested if bots drop off again.

[beelee]

@LaFayette

Hi Dan

sadly we had a bot malfunction. I was in Cali 103 and the game kinda just stopped. But wasn't frozen. I checked lobby and my bot was gone

as you can see no 103 and it looks as if we're down to 12.

Here's in game that is still showing 103 though after it dropped from lobby.

A couple guys said they had froze up at the same time.

Anyway, just a heads up.

Edit
Just checked and now they're back.

Edit 2 jusst checked in a gain about 3 1/2 hours and they are gone once more.

No Cali 103. Idk when they left again. Was about a 30 minute window first time I checked and it came back.

[LaFayette]

Did some work on the bots. ..

Turns out we need EXIT_GAME_ON_QUIT (or whatever it was...) to be true. Memory leaks killed the bots quickly and they crashed out hard after a few games.

I did some other things like put memory limits in and added swap. Each bot is now running with 128M, which is less, but there's more padding for when there is memory pressure. Hopefully wont' be taking down the whole bot instance any longer.

Let me know if bots drop out.. should be a bit better now.

[beelee]

@LaFayette

Sweet

Yea I saw that on git. Quit seems counter intuitive but hey I was just a carpenter lol