Block Scans for Hidden Content
What It Does
Block scans for hidden content stops two kinds of probing on the site.
- A request for a PHP file that is not on disk is blocked (HTTP 403). The file name is kept with the block.
- Every other 404 is counted per IP address, for one minute. Past the limit, further missing pages are refused (HTTP 429) for a few minutes. The IP address is not added to the banned list, and pages that exist keep loading.
The first 404 that crosses the limit is logged. Later 404s during the penalty are refused without a new log. The response includes a Retry-After header, and the visitor sees “Too many pages not found. Please retry later.”
These requests are not counted:
- an IP address on the allowed list;
- a verified bot, such as Google or Bing, when Yes, do not count verified bots is on (it is on by default);
- a request sent by a SecuPress scan;
- common files browsers and crawlers ask for anyway: favicon, touch icons,
robots.txt, sitemaps,manifest, and similar well-known paths.
A visitor and a robot do not share the same limit. A robot is a user agent that looks like a bot or a crawler, or a request with no user agent. The lists can be extended with the secupress.plugins.ban_404.ignored_patterns and secupress.visitor.is_robot filters. A missing PHP file can be let through with secupress.plugins.ban_404.bypass.
Why It Matters
Scanners try many addresses to find a forgotten file or a hidden page. A missing PHP file is a common way in. A long run of 404s is the same search, even when the target is not a PHP file.
Banning the IP address would also block the rest of the site, including a person who followed a dead link. Refusing only the next 404s stops the scan and leaves real pages available.
How to Use
Open the Firewall & GeoIP module, Bad Behaviors section.
- Enable Yes, protect my site from these scans.
- Leave Yes, do not count verified bots on, unless those bots should count toward the limit too.
In Expert Mode, three fields appear under the same protection:
- How many tries before a block for visitors?: 5 to 300, 30 by default, per minute.
- How many tries before a block for robots?: 5 to 300, 60 by default, per minute.
- How long should the block last?: 1 to 60 minutes, 5 by default, 5 to 15 recommended.
These fields set the 404 block described above. They do not ban the IP address.