Scraping Event - Brief Forum Outage - 2026-Sep-26
-
Once there is the confidence to reopen the forum to non-registered, I definitely suggest keeping the sections "Ladders", "Tournaments", "Play By Forum" and "Play By Email" hidden to guests. Ladders is practically dead (and not sure why it is still there unless it is for use by private ladders too, like the one hosted by @eyeswideshut), Tournaments is very contingent (and currently vacant of organizers as far as I know) and PBF/PBEM are really a thing you don't need to see unless you are part of it, usually.
(I'm curious why the hell would anyone do this. What have they to gain?)
-
+1 for the counter attack.
A scrape essentially means every URL was crawled on the site and everything downloaded. This bot came in several waves, and yesterday they turned up the scrape rate by 10x. We disconnected the forum briefly to allow updates to be sent to the server. The bot went quiet when that happened, but came back 10 minutes later at a lower rate with a changed user-agent and started scraping again .
I can't help but think of the algorithm that was implemented, this persistent attempt, try again but a bit later, except under the radar. I kinda suspect that 10x rate was meant to be as fast as we could serve traffic without completely taking down the site, yet it did. Very likely the full thing was fully automated, someone grabbing data for whatever reason.
-
@Cernel Forums is currently "open" to non-registered users, but limited in terms of categories. I agree that we should do a cleanup of categories.
-
(I'm curious why the hell would anyone do this. What have they to gain?)
Mass scraping of the internet is how LLMs were trained. There's a number of stories of how they use scrapers that do not declare themselves and overload websites (they're trying to scrape the whole internet, which means you need to do that quickly in order to get it done in a reasonable time frame). Though, anyone's guess is really as a good as mine. The scraper had access to a pool of thousands of IP addresses, this implies it's reasonably well set up entity and not a hobbyist.
-
Just curious, can the LLM access the chat also ?
Edit
hmm ... reading up on them a bit, it seems as if it's just ai doing this on it's own, as opposed to some person directly attacking triplea, although someone had to initially send it on it's way, maybe
Idk. If so, it's already learning to come back with a smaller attack as Dan stated above.
AI ... uggh ... just gotta get used to it, because it's not going away. Try and make it work for you and defend as best you can against it.
Gonna make cell phone introduction look like kids play lol
-
Scraping/Crawling is a normal process every platform that is selling information on the internet is doing it.
Some famous actors would be Google or other search engines. But also smaller companys do it for different reasons ( some company doing ads checking all the websites it can serve them for example).
There are technical and non technical ways in reducing its impact.
Cloudflare offers solutions that block malicous bots and scraping.
We could set a scraping policy (via robot.txt or other means) and set rules for bots if and in what capacity they can access the website.so normaly this is not an issue but the actor (or with bad luck multiple) just didnt give a shit.
-
@xxXEddieXxx Correct, with a key distinction though, Google uses well identified crawlers. This, it pretended be Chrome on Linux (per the user agent header), and then changed that header to be Chrome on Windows afterwards. Not necessarily that much of a distinction in one way, in another it is deceptive (and plus wasn't well rate limited)...
-
@LaFayette are we using cloudflare? pretty sure all their stuff is free for open source projects. i can tell you personally, from running a site with a bunch of traffic that it can def block a lot of the bad stuff with essentially no effort
"You should never have told me horses sleep standing up, it gave me a mental block." - Mister Ed
-
@ubernaut We're not.
A) a number of the tunings we applied were needed anyway. They were recommended by the NodeBB people in a "scaling guide" and should have those in.
B) Going through cloudflare at this juncture is extra effort. One time largely, but we have something that works.
C) Likely would not have helped too much actually. The scraper used a bogus user agent and did not declare itself as a scraper. They were sending us about 2 requests per IP per minute. Individually, it was a tiny amount of traffic per IP.
(C) is a bit nuanced. Out of the box there is little for cloudflare to do on this one. There's some toggles I'm aware of though where you can have it help you defend against excess load. In that case, there could be help. But again, there's some tunings we should have had applied already.
-
@LaFayette gotcha yeah they are mostly for high bandwidth type issues
"You should never have told me horses sleep standing up, it gave me a mental block." - Mister Ed
-
@ubernaut I feel like setting up cloudflare probably is on the todo list though..
Free Plan $0 / month Includes Bot Fight Mode (detects and challenges simple bots) and the ability to completely turn on the Block AI Scrapers toggle.
Looks like the setup is not that bad too.
-
@ubernaut cloudflare is comprehensive, IIRC you can toggle it so any new IP needs to do a captcha style challenge first. If we get attacked for real, then setting up cloudflare would likely be the only thing to restore the forum. OTOH, our data is what most people really want, I doubt very many really want to simply DDOS TripleA
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login