mrgtb wrote: ↑
Wed Jul 22, 2020 4:18 pm
That's a good point. if you're going to block guests from seeing something, then you really should block bots also with them.
That depends, if you aren't really trying to protect the content it doesn't matter. I had one forum for example that was behind login and I didn't want indexed either, I was allowing in just Googles Adbots because I was serving ads in it. I had special group called ad bots for the ad bot. I wasn't really concerned if someone spoofed it because they only needed to register to see it.
phpBB can use an IP to identify bots which will avoid spoofing issue. If you were going to allow them to index content that must not be seen by guests you can set up special group called Bots by IP and add those bots to it where you have added the IP. You can allow access for ones identified by IP and deny for other bots in the regular bots groups where you have not added an IP indetifier.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”
Attributed - Thomas Edison