I dont believe it is that easy to tell if an IP is coming from a data center, unfortunately. To some extent you can tell its location with the geolocation database, but that is not authoritative and can be wrong or even contain intentional falsehoods. You can also see what is adjacent to it and make guesses that way, but that is often inconclusive due to common tunneling protocols like MPLS.
I believe the only remotely accurate way to tell if a connection is a bot is to use modules that detect bot behavior and redirect what it detects. Some users have even made infinitely recursive “traps” for bots that do not respect their robots.text file, which make their web page seem infinite so it scrapes forever over these fake pages that are less work to host than they are for the bot to parse through.
It is a very tough problem with few easy solutions. It will likely result in an arms race much like ads and ad blockers, hackers and firewalls, etc. It has so far, anyway.
I dont believe it is that easy to tell if an IP is coming from a data center
The bad scraping bots are proxying their traffic via resiential IPs, so it doesn’t even matter. Presumably this is through the free TV apps that you see mentioned on Lemmy occasionally, or through compromised devices.
I mean, adding the IP filter means you are getting a number of bots filtered out by a very simple decision making (originating IP address/segment) even before you have to apply any further processing, so it’s still an advantage even if that filter by itself only removes a non-majority portion of bot traffic. It means you are left with more / better processing capacity to filter the rest.
I dont believe it is that easy to tell if an IP is coming from a data center, unfortunately. To some extent you can tell its location with the geolocation database, but that is not authoritative and can be wrong or even contain intentional falsehoods. You can also see what is adjacent to it and make guesses that way, but that is often inconclusive due to common tunneling protocols like MPLS.
I believe the only remotely accurate way to tell if a connection is a bot is to use modules that detect bot behavior and redirect what it detects. Some users have even made infinitely recursive “traps” for bots that do not respect their robots.text file, which make their web page seem infinite so it scrapes forever over these fake pages that are less work to host than they are for the bot to parse through.
It is a very tough problem with few easy solutions. It will likely result in an arms race much like ads and ad blockers, hackers and firewalls, etc. It has so far, anyway.
The bad scraping bots are proxying their traffic via resiential IPs, so it doesn’t even matter. Presumably this is through the free TV apps that you see mentioned on Lemmy occasionally, or through compromised devices.
I mean, adding the IP filter means you are getting a number of bots filtered out by a very simple decision making (originating IP address/segment) even before you have to apply any further processing, so it’s still an advantage even if that filter by itself only removes a non-majority portion of bot traffic. It means you are left with more / better processing capacity to filter the rest.