James
Mon Nov 03 07:10:20 CST 2003
John Savidge wrote:
> Yes, true, but you really use the robots.txt file to exclude
> particular files and folders be excluded from search engine indexing.
> I.e your images folders,
/images might be OK to exclude, but you might get traffic from Google
Images... why exclude this?
> database files, menu html files etc.
Not sure I understand you here...
1. Due to the way Google's PageRank algorithm works, a site will be
effectively penalised if files such as "menu html files" are excluded from
robots.txt. Google knows the pattern of links on each site - why do you
want to block commonly-used menus from the index?
2. What do you mean by restricting database files?
> Use of
> this file will actually improve the search engines ability to index
> your site more effectively as it is not indexing pages which are not
> relevant to your businesses details.
I would (politely) dispute this idea - IMHO most small businesses and SBS
administrators don't have a complete enough understanding of search engines
and their (ever-changing) algorithms to correctly identify what
files/folders can be safely excluded using robots.txt. Better to leave it
all available and suffer a little more bandwidth usage than to end up at the
bottom of the search engine due to a badly chosen robots.txt...
> In combination with meta tags this can really help get your website
> up the search engine ranking for the keywords you want to be found
> under.
Google doesn't may any attention to the <meta name="keywords"> tag
(
http://www.webmasterworld.com/forum3/17726.htm), neither do Altavista or
AllTheWeb [FAST]
(
http://www.searchenginewatch.com/webmasters/article.php/2167891). You can
get to the top spot in all these without any meta keywords at all.
> Take it from me I have achived this.
Good for you - all I can say is try surfing over to
news:alt.internet.search-engines or www.webmasterworld.com and I'm sure
you'll find even more up-to-date advice! :-)
James