Lobby Upgrades, June-25-2026
-
Upgrading the server OS running lobby, it's currently unavailable. Hoping for this to be wrapped up in the next 30 minutes.
-
Upgrade is a bit more involved than anticipated. I'm in process of re-configuring the database.
-
@LaFayette random question im not sure if this actually different or not but it seems different. now when i go to the lobby and start a new game it prompts me to select a port and actually seems to be locally hosting the game is this changed format he past and/or intentional or am doing something wrong?
-
@ubernaut that's unrelated.
Upgrade update: The database re-config for lobby is not going that well. Anyone that registered in the last two weeks will likely need to re-register (lost data). I'm making progress though on getting lobby back up and running.
Key steps:
- [done] new server running
- [done] DNS updated to point to new server
- [done] nginx config (this does SSL / https for us)
- [WIP] postgres18 database running
- [WIP] lobby running
- [TODO] restore users table in database
-
@LaFayette im reading that as nothing changed in that regard and im doing something wrong
-
@ubernaut You're able to actually connect to the lobby? It should be dead/gone right now. I need to focus on getting things set back up properly - sorry if i misread something in the meanwhile.
-
@LaFayette i didnt check it today maybe i should have kept my mouth shut lol
-
Progress, setup tasks are done. Currently working on a routing misconfig causing a 404 when logging in. Once resolved, I believe everything should be back up and running
- [done] new server running
- [done] DNS updated to point to new server
- [done] nginx config (this does SSL / https for us)
- [done] postgres18 database running
- [done] lobby running
- [done] restore users table in database
-
Possibly final update, I think everything is sorted now - lobby is back up.
Key changes:
- changed to a new server on a less SKU, saves $19/month. I think we can get away with the less powerful server and save cash.
- new server is running Ubuntu 24 (OS upgraded)
- we did lose about 2 weeks worth of database changes, anyone who registered in that time will need to re-register their account
- I added a new daily backup feature for database, it'll help prevent data loss like that going forward
-
-
@beelee good catch. I cleaned up a bit too many DNS entries. I restored those, it takes about 10 minutes for DNS to update globally. Should be back pretty soon.
-
-
@beelee yes, working on that currently. The 'forgot password' feature of the lobby should be busted too as well. I'll be tackling that after the support server is restored.
One thing that is truly lost, a good chunk of DB data, any users registered to the lobby in the last two weeks will need to re-register.
-
Maps server is back up & running, it's now re-indexing maps, the map list is growing as it goes on.
Now to fix the forgot password feature of the lobby...
-
Mail delivery from lobby and dice server should no longer go to spam. Though, lobby "forgot password" emails are still not working. I'll be un-SNAFU'ing that at near future date.
-
Lobby can now send emails too.
To boot, the email did not go into spam, so I think those various configs are also properly in place.
We're left with servers all patched up, latest, with autodeployments enabled, running services via docker-compose, and controlled with 'infrastructure-as-code' via the 'infrastructure' repository.
-
Right on.
I noticed a couple disconnects today. I assume that was to the above work ? They won't be a daily occurrence ?
-
@beelee I think we need to be clear on the 'disconnects'
Are we talking just the lobby disconnecting and reconnecting, or disconnecting from an actual game?
There was one global, everything restarted event today, that will not be terribly common.
Otherwise the lobby disconnects should be overall harmless - and it's the lobby being rebooted to pick up updates.
If the games disconnected, like players are in a game playing on a bot, and are booted (apart from the one global-everything-restarted even), that would be a new issue that should be identified as such.
Otherwise, we might very well have just updated the lobby more times in the last 2 days than we have in the last 10 years! We were running lobby versions way behind the game version most of the time.
-
Yea i just noticed guys saying disconnected and having to restart the game and get the save. Last 2-3 hours I guess
-
@beelee thanks, hard to know what specifically it means. Whether D/C from lobby, bot (in-game), and if bot - whether that was a bot issue or something related to the lobby reboot.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login