Scraping Event - Brief Forum Outage - 2026-Sep-26

LaFayette

A write up is here: https://github.com/triplea-game/forums/discussions/5

In short, a scraper started sending us 70x traffic and the forum fell over.

In response, made a number of changes that will help us going forward:

most categories are now hidden to non-registered users. "News" and the "Bug Report" categories are the only ones publicly visible now. This gives less surface area for a scanner, so they hopefully move on more quickly.

- applied a lot of infrastructure tunings to reduce the CPU cost of serving a forums website request

increased connection limits on the NGINX (the web server in front of forums)
added a missing database index that caused guest page views to be very expensive for the CPU
added site wide rate limits and per-IP rate limits. The site wide rate limits will more help the forums degrade gracefully rather than fully crash.
moved assets to be served directly from web cache instead of by the NodeBB forumn software. Requests for assets are much easier on the server now and no longer involve any database writes.
turned off the feature to count for the number of non-registered user page views. Simply counting the number of views requires an update to database. Every scraping request added to page counts, and so turning the feature off removes these database writes.

Overall, the updates make each web request a lot cheaper for the server to serve, which means higher capacity. The scraper event stopped at about the same time when we started responding to it. With these updates, we might mostly survive another scraping event (but if the request rate is high enough, we probably wouldn't).

beelee

@LaFayette

Is "scraping" essentially a massive "bot" attack ? although attack might not be the right description.

Just wondering if what triplea experienced was/is similar to what Djensen experiences at the other site periodically. It basically started after a nodeBB update, which was done here not all that long ago also.

Anyway, he put up some defenses as well, in case you are interested in what he did or he may be interested in what you did.

https://www.axisandallies.org/forums/topic/42468/bot-attacks?page=1

Edit
I just saw the other thread, so basically was a massive ai bot screwing everything up

We need our own bot Army to counterattack

Cernel

Once there is the confidence to reopen the forum to non-registered, I definitely suggest keeping the sections "Ladders", "Tournaments", "Play By Forum" and "Play By Email" hidden to guests. Ladders is practically dead (and not sure why it is still there unless it is for use by private ladders too, like the one hosted by @eyeswideshut), Tournaments is very contingent (and currently vacant of organizers as far as I know) and PBF/PBEM are really a thing you don't need to see unless you are part of it, usually.

(I'm curious why the hell would anyone do this. What have they to gain?)

LaFayette

+1 for the counter attack.

A scrape essentially means every URL was crawled on the site and everything downloaded. This bot came in several waves, and yesterday they turned up the scrape rate by 10x. We disconnected the forum briefly to allow updates to be sent to the server. The bot went quiet when that happened, but came back 10 minutes later at a lower rate with a changed user-agent and started scraping again .

I can't help but think of the algorithm that was implemented, this persistent attempt, try again but a bit later, except under the radar. I kinda suspect that 10x rate was meant to be as fast as we could serve traffic without completely taking down the site, yet it did. Very likely the full thing was fully automated, someone grabbing data for whatever reason.

LaFayette

@Cernel Forums is currently "open" to non-registered users, but limited in terms of categories. I agree that we should do a cleanup of categories.

LaFayette

@Cernel said:

(I'm curious why the hell would anyone do this. What have they to gain?)

Mass scraping of the internet is how LLMs were trained. There's a number of stories of how they use scrapers that do not declare themselves and overload websites (they're trying to scrape the whole internet, which means you need to do that quickly in order to get it done in a reasonable time frame). Though, anyone's guess is really as a good as mine. The scraper had access to a pool of thousands of IP addresses, this implies it's reasonably well set up entity and not a hobbyist.

beelee

@LaFayette

Just curious, can the LLM access the chat also ?

Edit
hmm ... reading up on them a bit, it seems as if it's just ai doing this on it's own, as opposed to some person directly attacking triplea, although someone had to initially send it on it's way, maybe

Idk. If so, it's already learning to come back with a smaller attack as Dan stated above.

AI ... uggh ... just gotta get used to it, because it's not going away. Try and make it work for you and defend as best you can against it.

Gonna make cell phone introduction look like kids play lol

xxXEddieXxx

@beelee

Scraping/Crawling is a normal process every platform that is selling information on the internet is doing it.

Some famous actors would be Google or other search engines. But also smaller companys do it for different reasons ( some company doing ads checking all the websites it can serve them for example).

There are technical and non technical ways in reducing its impact.
Cloudflare offers solutions that block malicous bots and scraping.
We could set a scraping policy (via robot.txt or other means) and set rules for bots if and in what capacity they can access the website.

so normaly this is not an issue but the actor (or with bad luck multiple) just didnt give a shit.

LaFayette

@xxXEddieXxx Correct, with a key distinction though, Google uses well identified crawlers. This, it pretended be Chrome on Linux (per the user agent header), and then changed that header to be Chrome on Windows afterwards. Not necessarily that much of a distinction in one way, in another it is deceptive (and plus wasn't well rate limited)...