Lobby Upgrades, June-25-2026
-
Progress, setup tasks are done. Currently working on a routing misconfig causing a 404 when logging in. Once resolved, I believe everything should be back up and running
- [done] new server running
- [done] DNS updated to point to new server
- [done] nginx config (this does SSL / https for us)
- [done] postgres18 database running
- [done] lobby running
- [done] restore users table in database
-
Possibly final update, I think everything is sorted now - lobby is back up.
Key changes:
- changed to a new server on a less SKU, saves $19/month. I think we can get away with the less powerful server and save cash.
- new server is running Ubuntu 24 (OS upgraded)
- we did lose about 2 weeks worth of database changes, anyone who registered in that time will need to re-register their account
- I added a new daily backup feature for database, it'll help prevent data loss like that going forward
-
-
@beelee good catch. I cleaned up a bit too many DNS entries. I restored those, it takes about 10 minutes for DNS to update globally. Should be back pretty soon.
-
-
@beelee yes, working on that currently. The 'forgot password' feature of the lobby should be busted too as well. I'll be tackling that after the support server is restored.
One thing that is truly lost, a good chunk of DB data, any users registered to the lobby in the last two weeks will need to re-register.
-
Maps server is back up & running, it's now re-indexing maps, the map list is growing as it goes on.
Now to fix the forgot password feature of the lobby...
-
Mail delivery from lobby and dice server should no longer go to spam. Though, lobby "forgot password" emails are still not working. I'll be un-SNAFU'ing that at near future date.
-
Lobby can now send emails too.
To boot, the email did not go into spam, so I think those various configs are also properly in place.
We're left with servers all patched up, latest, with autodeployments enabled, running services via docker-compose, and controlled with 'infrastructure-as-code' via the 'infrastructure' repository.
-
Right on.
I noticed a couple disconnects today. I assume that was to the above work ? They won't be a daily occurrence ?
-
@beelee I think we need to be clear on the 'disconnects'
Are we talking just the lobby disconnecting and reconnecting, or disconnecting from an actual game?
There was one global, everything restarted event today, that will not be terribly common.
Otherwise the lobby disconnects should be overall harmless - and it's the lobby being rebooted to pick up updates.
If the games disconnected, like players are in a game playing on a bot, and are booted (apart from the one global-everything-restarted even), that would be a new issue that should be identified as such.
Otherwise, we might very well have just updated the lobby more times in the last 2 days than we have in the last 10 years! We were running lobby versions way behind the game version most of the time.
-
Yea i just noticed guys saying disconnected and having to restart the game and get the save. Last 2-3 hours I guess
-
@beelee thanks, hard to know what specifically it means. Whether D/C from lobby, bot (in-game), and if bot - whether that was a bot issue or something related to the lobby reboot.
-
Yea Idk either. I was gonna ask but saw you were in there and I hesitate to interject myself in to things that I know little about.
I guess if it becomes problematic, we'll hear about it
-
@beelee word, I'm not paying a ton of attention, just mostly keeping an eye on how long the disconnects are and that the lobby comes back
It'll be something for us to keep an eye on. I expect though that lobby updates will be a bit "bursty", a couple features added here and there in a short time span and then nothing for a while.
-
well it seemed they were of the short 10 second or less variety or less. I guess next time it happens I'll ask if they have to restart, I assume they do.
If you were in a battle and it happened multiple times a day, i could see where it would be annoying
Anyway, we'll keep an eye on it. I don't really play in the lobby, although that may change a little. I just like to see how many people there are and if anyone having trouble with something.
-
If you were in a battle and it happened multiple times a day, i could see where it would be annoying
Why? My suspicion is because the "lobby reconnecting" dialog box is "always on top" - but I'm curious if that is actually the reason, or something else.
Overall, in-game should have no impact at all when the lobby restarts, zero. Except of course the 'disconnect' window showing up. I think a better version of that would be a 'disconnect' window that was part of the lobby window, rather than a modal pop-up.
The automatic reconnect to lobby should otherwise work. No need to restart or reconnect to lobby.
Seems there's an open question whether lobby dropping out is impacting in-flight games in any way. It should not be.. It could be only if the bots were triggering some sort of error, but generally buts can run disconnected from the lobby just fine.
-
Yea that's basically my question. If people have to restart the game or if it's just a few second blip and the game continues as normal.
Idk the answer. It just seemed some people were reacting that they had to restart.
Maybe they are jumping the gun and don't know to stand pat a few seconds.Last comment I saw half hour ago or so was someone saying "tomorrow should be better" in reference to the disconnects
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login