TripleA Server Costs

[Cernel]

All 15 TripleA automated hosts are now working (so that's a total of 17 with @eyeswideshut's ones). Did anyone do anything or was it just something automated? Just curious.

[EyesWideShut]

@Cernel Thanks for flagging this. For the Revised Ladder bots, I set them up to run inside persistent tmux sessions on the AWS server, so they remain active even after I disconnect from SSH or close my computer. I also created separate run scripts for the bots and a startup script that launches them together, which makes them easier to restart and keep online. So far, that setup seems to be working and the Revised Ladder bots are still running, although I’m continuing to monitor them in case one freezes or disconnects. I am paying about $10 a month for both headless servers which I understand is consistent with what triplea in general is paying for the bots.

[Cernel]

@EyesWideShut said:

I am paying about $10 a month for both headless servers which I understand is consistent with what triplea in general is paying for the bots.

I guess this means that you are paying $5 per bot? What I understand @lafayette said is that TripleA is paying $5 per server of 3 bots, which would be $1.66 per bot.

[EyesWideShut]

@Cernel ah my bad. so I'm paying more. My Bots are cloud-based for better or for worse. They seem to be working fine but I haven't had crashes that I'm aware of.

[EyesWideShut]

@Cernel When the Revised Ladder bots were crashing frequently, I moved them onto Amazon Web Services and reworked how they were launched and maintained.

AWS is a cloud hosted Linux server, so the bots run remotely rather than depending on my own computer, home internet connection, or an open SSH window. I connect to the server through SSH, but the bots themselves run inside persistent tmux sessions.

tmux is useful here because it separates the bot process from the temporary SSH connection. I can disconnect from the server, close my laptop, or lose my local connection without terminating the bot. Each bot has its own named session, which also makes it easier to inspect or restart one without affecting the other.

The main upkeep commands are:

tmux ls

This shows whether the bot sessions are still present.

tmux attach -t revised_ladder
tmux attach -t revised_ladder2

These reconnect me to the live console output for either bot so I can inspect errors, freezes, or lobby disconnects.

To leave a session while keeping the bot running:

Ctrl+B, then D

If the bot is frozen rather than fully gone, I can attach to the session, stop the process with Ctrl+C, and relaunch it using the relevant startup script.

I also set up the launch process so that if a bot process actually disappears or exits, it is automatically started again. In other words, the server does not simply leave the bot offline after a crash. The surrounding startup or monitoring process detects that the bot is no longer running and relaunches it. That does not necessarily fix every possible freeze, since a frozen process may still technically exist, but it does handle ordinary crashes and unexpected exits without requiring me to log in immediately.

So the overall setup has three layers:

AWS provides the continuously available cloud server.
tmux keeps the bot sessions alive independently of SSH.
The restart logic relaunches a bot when its process exits or disappears.

This is what I implemented when the bots were crashing a lot. Since then, the AWS and tmux setup has generally been stable, and the automatic restart behavior has reduced the amount of manual intervention needed.

[Cernel]

@LaFayette said:

Good points, we need to also keep in mind that the bots need to be available for peak lobby activity.

@lafayette Yes, it looks like that at peak 15 bots are easily all taken. I've seen it happening despite the 2 additional bots courtesy of @eyeswideshut.

Unfortunately, the lobby is back having only 10 out of 15 TripleA bots working. Every server seems missing 1 of its 3 bots. The missing bots are "101", "202", "303", "403" and "501".

Maybe @eyeswideshut can help you making the bots more stable? It made a post with information about that right before this one.

CC: @frigoref

[beelee]

@Cernel

There is currently only one available bot

[EyesWideShut]

@Cernel OK! So I think I solved the bot crashing problem on my end. It was the script function, and the fix was actually very simple.

The problem was that I had this command in the Docker startup script -- and it's probably in all the bots:

--env EXIT_ON_GAME_END=true \

That tells the bot to exit when the game ends. Since the container was configured with --restart always, I was expecting Docker to bring it back, but this was creating problems with the bot disappearing from the lobby.

I removed that command from the startup scripts.

For example, the relevant part went from:

sudo docker run
--network host
--pull always
--restart always
--env BOT_PORT_NUMBER=3301
--env BOT_NAME=Bot_Revised_Ladder
--env BOT_COMMENT=bit.ly/RevLad2
--env LOBBY_URI=https://prod.triplea-game.org
--env EXIT_ON_GAME_END=true
--env BOT_MAX_MEMORY=256m \

to:

sudo docker run
--network host
--pull always
--restart always
--env BOT_PORT_NUMBER=3301
--env BOT_NAME=Bot_Revised_Ladder
--env BOT_COMMENT=bit.ly/RevLad2
--env LOBBY_URI=https://prod.triplea-game.org
--env BOT_MAX_MEMORY=256m \

And I did the same for the second bot on port 3302.

The two bots are now running normally, so I'm going to see whether they remain in the lobby after their next games. Hopefully that's all it was! check to see if the other bots have this function..

[beelee]

I'm consistently seeing only 10 triplea bots in the lobby. If triplea is paying for 15, then this is something that should be addressed sooner than later imo

[Cernel]

@beelee My understanding is that TripleA has 5 servers on which the bots are located. The servers are all working but currently have 2 bots each instead of the 3 each @lafayette had them having. I've no idea if that is because 3 bots per server are just too much (so it has to be 2 as it currently practically is) or there is a problem or some problems with the bots which @lafayette (or an other developer) may fix eventually (maybe with help from @eyeswideshut).

Not sure if we should make a problem report in GitHub about this or if we should notify anyone anywhere when we see that some bots are broken.

[beelee]

@Cernel

Yea I'm not certain either but like you, I thought we were supposed to have 15.

The reason I mention it again, besides potentially paying for a service we aren't receiving, there's usually only 1 available bot or sometimes none.

I'm glad the lobby still popular but I was hoping to play on it again since new maps are now supported.

Anyway, it seemed noteworthy to mention again