SpecE30
Welcome, Guest
Please Login or Register.    Lost Password?
SpecE30 General Topics Technical Site Issues
spece30.com content not indexed in search engines? (1 viewing) (1) Guest
Go to bottom Post Reply Favoured: 0
TOPIC: spece30.com content not indexed in search engines?
#27099
laz ()
User Offline Click here to see the profile of this user
spece30.com content not indexed in search engines? 1 Month, 3 Weeks ago  
It looks like http://spece30.com/robots.txt disallows /component/, which blocks the forums from being indexed by search engines. Is this intentional?

It'd be nice to use Google to search the forums in addition to the site's forum search function. What sent me down this path is the forum search has no topic de-duping, making searches that return a lot of results hard to manage.
 
Report to moderator   Logged Logged  
  The administrator has disabled public write access.
#27149
mwmills ()
User Offline Click here to see the profile of this user
Re:spece30.com content not indexed in search engines? 1 Month, 3 Weeks ago  
Yes, this is intentional.
 
Report to moderator   Logged Logged  
 
Mike Mills
  The administrator has disabled public write access.
#27151
TheRedBaron ()
User Offline Click here to see the profile of this user
Re:spece30.com content not indexed in search engin 1 Month, 3 Weeks ago  
What's the rationale behind that out of curiosity?
 
Report to moderator   Logged Logged  
 
Craig Geiger
#82
Racer X is my Hero
  The administrator has disabled public write access.
#27155
mwmills ()
User Offline Click here to see the profile of this user
Re:spece30.com content not indexed in search engin 1 Month, 3 Weeks ago  
It was discovered late last year that Search engines were placing a large (excessive) load on the SQL server that significantly impacted performance. This policy was adopted at that point and performance as well as load have been stabilized.
 
Report to moderator   Logged Logged  
 
Mike Mills
  The administrator has disabled public write access.
#27170
laz ()
User Offline Click here to see the profile of this user
Re:spece30.com content not indexed in search engines? 1 Month, 3 Weeks ago  
Most bots respect "Crawl-Delay: 60" in robots.txt to specify the delay in seconds between requesting pages. The google bot does, I just looked
 
Report to moderator   Logged Logged  
  The administrator has disabled public write access.
#27171
mwmills ()
User Offline Click here to see the profile of this user
Re:spece30.com content not indexed in search engines? 1 Month, 3 Weeks ago  
You are quite right, however, others do not. Thanks for your concerns but this is not going to be changed.
 
Report to moderator   Logged Logged  
 
Mike Mills
  The administrator has disabled public write access.
Go to top Post Reply
Powered by FireBoardget the latest posts directly to your desktop
 
Joomla Template by Joomlashack
Joomla Templates by JoomlaShack Joomla Templates by Compass Design