Hello. No problems with the server. for the work of the external services we can athecate Yong. please proverit settings robots.txt on the website and be sure to set the delay http://www.erahost.ru/faq.php Tell me how to configure robots.txt under specific cms Hello. We offer You several configuration templates robots.txt for popularnych cms
wordpress User-agent: * # apply to all search engines Crawl-delay: * # set the delay between queries of a search robot Disallow: *? # disallow all dynamic pages Disallow: *.php$ # disallow all pages with the extension ".php” Disallow: /search page/ #disallow page of search results Disallow: /tag/ # disallow all pages of tags (labels) Disallow: /feed/ # disallow XML-feed latest articles Disallow: /page/ # disallow unnecessary pages in WP that appear after you specify a static page as home (otherwise – is NOT specified) Disallow: */attachment/ # disallow page attachments Host: site.ru # specify the main mirror Yandex Sitemap: http://site.ru/sitemap.xml # specify the address of the XML Sitemap
For joomla User-agent: * Disallow: /administrator/ Disallow: /bin/ Disallow: /cache/ Disallow: /cli/ Disallow: /components/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /layouts/ Disallow: /libraries/ Disallow: /logs/ Disallow: /media/ Disallow: /modules/ Disallow: /plugins/ Disallow: /templates/ Disallow: /tmp/ Disallow: /index.php* Disallow: /index*.php* Allow: /images/ Allow: /index.php?option=com_xmap&sitemap=*&view=xml Host: westoning Sitemap: http://ваш_адрес_карты_сайта Clean-param: searchword / Crawl-delay: * the presence of Crawl-delay in robots.txt sure |