Hello. server problems nebylo. check the setting robots.txt http://www.erahost.ru/faq.php Tell me how to configure robots.txt under specific cms
Hello. We offer You several configuration templates robots.txt for popularnych cms
wordpress
User-agent: * # apply to all search engines
Crawl-delay: * # set the delay between queries of a search robot Disallow: *? # disallow all dynamic pages
Disallow: *.php$ # disallow all pages with the extension ".php”
Disallow: /search page/ #disallow page of search results
Disallow: /tag/ # disallow all pages of tags (labels)
Disallow: /feed/ # disallow XML-feed latest articles
Disallow: /page/ # disallow unnecessary pages in WP that appear after you specify a static page as home (otherwise – is NOT specified)
Disallow: */attachment/ # disallow page attachments
Host: site.ru # specify the main mirror Yandex
Sitemap: http://site.ru/sitemap.xml # specify the address of the XML Sitemap
For joomla
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /index.php*
Disallow: /index*.php*
Allow: /images/
Allow: /index.php?option=com_xmap&sitemap=*&view=xml
Host: westoning
Sitemap: http://ваш_адрес_карты_сайта
Clean-param: searchword /
Crawl-delay: * |
