In shortPart of every site’s traffic is not human but automated software. Some of it is useful, like Googlebot indexing your site, and some is harmful: it scrapes content and prices, probes for flaws, guesses passwords and drains your resources. robots.txt does not stop bad bots because they ignore it, so protection belongs on the server or in Cloudflare, taking care not to block search engines.
Good bots and bad bots #
✓Useful botsGooglebot and Bingbot for search engines, and monitoring tools you choose.
!Content scrapersCopy your articles, images and prices to other sites.
!Vulnerability scannersTry thousands of URLs looking for old plugins and exposed files.
!Login and spam botsGuess passwords and fill forms. Read protecting contact forms.
Signs bots are straining your site #
- Traffic rises in your stats with no increase in sales or enquiries.
- The site slows down or hits your plan’s resource limits at odd hours.
- Visitor logs in cPanel show many requests from a few addresses or to pages that do not exist.
How to protect your site #
- Do not rely on robots.txt alone: Only good bots respect it. Bad bots ignore it and may even read it to find what you want hidden.
- Block abusive addresses: Use the cPanel IP blocker to block addresses that keep hammering your site.
- Block by User-Agent: Some bots announce their name, so you can block them with an .htaccess rule like the example below.
- Turn on Cloudflare protection: Bot Fight Mode and firewall rules block known bots before they reach your server, and you can switch on Under Attack mode under pressure.
- Rate limit requests: Especially for search and login pages, which are the heaviest for your site.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (MJ12bot|DotBot|PetalBot) [NC]
RewriteRule .* - [F,L]
</IfModule>
Do not block search engines: the real Googlebot comes from Google’s addresses. Some bad bots fake the Googlebot name, so never block that name itself. Before blocking any bot, make sure your SEO tools do not need it.
How Hyyat Host protects your site #
- Imunify360 detects suspicious automated traffic and challenges or blocks it, while real visitors pass.
- The CSF firewall limits large numbers of connections from one address.
- You can connect your site to Cloudflare for an extra blocking layer in front of the server.
Learn more about real-time IP reputation protection.
Frequently asked questions #
Does blocking bots hurt SEO?
Only if you block search engine crawlers. Blocking bad bots lightens the load on your site and does not hurt rankings.
Can I stop content scraping completely?
Not entirely, but blocking and rate limits make it much harder.
How do I see which bots visit my site?
From the visitor logs or Awstats in cPanel, or from Cloudflare analytics if your site is connected to it.
