| Question: |
Good afternoon.Your fault is constantly losing connection search robot Yandex server.I have repeatedly contacted tech support Yandex, and they reply that these problems occur on the side of the host. Here is their response:\"Maxim, I see that now access disappeared again. Please email Your hosting provider to provide the site stable access for an indexing robot. Now the site is a robot with IP 178.154.189.15.\"Please take steps to ensure it did not recur. |
| Answer: |
Hello. 178.154.189.15 ip is not blocked. check the correctness of the configuration file robots.txt |
| Question: |
I never changed that before and it worked fine, and since March 15, the robot is stopped to index the site, and began to write, to hundred it failed to communicate with the web carcerem. Check andexpenditure and googlewebmaster shows that errors in the file robots.txt not detected.Technical support of Yandex answered me twice that this is a problem on the side of the hoster, so do something. |
| Answer: |
in the file ustanovleny of zaderzhki when crawling robot? make sure to set the delay of a few seconds. also disable indexing extra folder. is also required |
| Question: |
Until recently everything was fine in the file I have not changed anything, and all this time, it is perfectly indexed, in addition, the analysis of Yandex and Google shows that in of Nemorino there, so most likely it`s not him. |
| Answer: |
be sure to install a delay in the handling of the robot and exit from scanning unnecessary folders |
| Question: |
I don`t know how it can be done. I can send you this file so you looked at him and said, what`s wrong with it? |
| Answer: |
it should be the Directive User-agent: *
Crawl-delay: 10 and direcuy Disallow for folders you want to exclude from indexing. |
| Question: |
Tell me what we need to change?Now it looks like this:User-agent: *Disallow: /cgi-binDisallow: /wp-adminDisallow: /wp-includesDisallow: /wp-content/pluginsDisallow: /wp-content/cacheDisallow: /wp-content/themes Disallow: /trackbackDisallow: */trackbackDisallow: */*/trackbackDisallow: /tagAllow: /category/*/*Disallow: /feed/Disallow: */*/feed/*/Disallow: */feedDisallow: /*?*Disallow: /?s=Host: stroimsamydom.ruSitemap: http://stroimsamydom.ru/sitemap.xml |
| Answer: |
add Crawl-delay: 10 |
| Question: |
Where exactly, in what place? |
| Answer: |
the Above was given to a sample User-agent: *
Crawl-delay: 10 also add personally for Yandex User-agent: Yandex
Crawl-delay: 5 |
| Question: |
Okay, thanks, I`ll try |
| Answer: |
ok |
| Question: |
Hello again.Support Yandex said that the robots file? nothing to do with.Now I will send you what they wrote to me, understand please with this problem. |
| Question: |
Maxim, it`s not robots.txt - the robot cannot access the website TCP\\IP, here`s a snippet of the trace: 7 dtel-ix.as6663.net (193.25.180.192) 18.568 ms panas-xe-1-3-0.yndx.net (87.250.239.70) 18.244 ms 18.283 ms 8 dtel-ix.as6663.net (193.25.180.192) 18.279 ms 16.147 ms 18.388 ms 9 31.210.11.37 (31.210.11.37) 42.500 ms 42.476 ms 31.210.11.38 (31.210.11.38) 43.785 ms10 31.210.11.38 (31.210.11.38) 43.838 ms 43.902 ms*11 * * *12 * * *13 * * *14 * * *15 * * *16 * * *17 * * *Telnet to port 80 also bad:Trying 46.29.2.62... please Pass this information to Your hoster, please. |
| Answer: |
please Give the full trace results.
Inform your IP, you can read it on the page http://www.empire-host.ru/2ip.php |
| Question: |
Please see the full version of the trace:Maxim, the complete result trace IP 178.154.189.15 :1 178.154.189.254 (178.154.189.254) 0.169 ms 2.316 ms 1.971 ms 2 ugr-b-fw3-vlan1603.yndx.net (37.9.89.129) ms 0.123 0.159 ms ugr-b-fw10-vlan1603.yndx.net (37.9.89.130) 0.267 ms 3 ugr-b-c1-irb-802.yndx.net (130.193.45.222) 0.620 ms 1.743 ms 1.795 ms 4 std-p2-be13.yndx.net (87.250.239.87) 20.956 ms 20.949 ms m9-p2-eth-trunk5.yndx.net (87.250.239.52) 8.606 ms 5 m9-p1-be6.yndx.net (87.250.239.51) 17.249 ms ms ms 17.139 17.169 6 neun-ae2.yndx.net (87.250.239.23) 0.918 ms panas-xe-1-3-0.yndx.net (87.250.239.70) 18.182 ms panas-xe-0-0-1-984.yndx.net (213.180.213.119) 20.810 ms 7 dtel-ix.as6663.net (193.25.180.192) 16.002 ms panas-xe-0-0-1-984.yndx.net (213.180.213.119) 20.417 ms 20.447 ms 8 dtel-ix.as6663.net (193.25.180.192) 16.185 ms 16.185 ms 31.210.11.37 (31.210.11.37) 42.187 ms 9 31.210.11.38 (31.210.11.38) 51.127 ms 31.210.11.37 (31.210.11.37) 42.114 ms 31.210.11.38 (31.210.11.38) 41.648 ms10 31.210.11.38 (31.210.11.38) 50.791 ms 50.947 ms *11 * * *12 * * *13 * * *14 * * *15 * * *16 * * *17 * * *18 * * *19 * * *20 * * *21 * * *22 * * *23 * * *24 * * *25 * * *26 * * *27 * * *28 * * *29 * * *30 * * * |
| Answer: |
Inform your IP, you can read it on the page http://www.empire-host.ru/2ip.php |
| Question: |
Your IP address: 91.201.243.70 |
| Answer: |
You could check with them of great interest to our server with their bots? in less than 5 hours was 124849 requests 335 ip addresses of the Yandex. here are the TOP 50 ip supreame 3892 141.8.142.8
3819 178.154.189.6
3643 5.255.253.51
3527 5.255.253.94
5.255.253.62 3375
3008 141.8.142.55
2717 141.8.132.109
1924 141.8.142.85
1618 141.8.184.31
1590 130.193.51.2
1561 130.193.50.23
1549 130.193.50.16
1548 141.8.132.11
1350 130.193.50.3
1282 141.8.142.69
1274 130.193.51.75
1236 93.158.152.2
1202 141.8.132.56
1177 141.8.132.3
1159 178.154.189.33
1093 178.154.189.31
1036 141.8.142.60
1018 93.158.152.15
1009 141.8.132.90
1007 141.8.132.28
1006 141.8.142.13
1002 141.8.132.65
1001 141.8.183.22
1001 141.8.132.71
994 193.232.121.208
991 178.154.189.17
978 130.193.51.87
976 141.8.132.31
975 141.8.142.24
5.255.253.50 974
966 141.8.142.100
965 130.193.51.95
957 141.8.142.36
954 141.8.132.98
951 141.8.132.18
944 130.193.51.80
923 141.8.132.53
904 93.158.152.78
883 141.8.132.112
881 141.8.132.83
868 178.154.189.23
849 130.193.50.38
782 93.158.152.7
777 178.154.189.32
774 130.193.37.16 |
| Question: |
I have given them your question, yet received no response.What can you say about the solution to my problem?For some reason it arose, and when it will be fixed? |
| Answer: |
the problem depends on the activity of Yandex |
| Question: |
But before such problems did not exist, and until March 15, my site is normally indexed each time you update the search bases of Yandex.Besides, technical support of Yandex claims that this is a problem you still. |
| Answer: |
the problem is not on sa Yandex which is extremely aggressive began to scan the server |
| Question: |
And why did the server not responding to his robot?And this was not once, it`s been three database updates and with each update he sends me messages that the web server is not responding |
| Answer: |
potamu that the bot creates a high load |
| Question: |
Good afternoon.Are you still in touch? |
| Answer: |
Hello. What can I help You? |
| Question: |
You should probably be aware of our yesterday`s conversation with Your tech support.Technical support of Yandex offers You directly to contact them to resolve this issue, write to them please, what`s the problem, and solve this problem with joint efforts. |
| Question: |
Send them an email:Hello, Maxim!Not to play \"broken phone\", please ask Your hosting provider more details to describe the situation and send the logs of queries from robots, for example, at address [email protected] . Thank you! |
| Answer: |
Okay, we`ll contact them. |
| Question: |
Thanks, I will wait for the result. |
| Question: |
Tell me please, are there any results for my question? |
| Answer: |
no, we don`t attili. robots we do not block |
| Question: |
Good afternoon.Tell me, yesterday you contacted the technical support of Yandex on my question? |
| Answer: |
Yes, there was no answer. judging by the logs the bot You are visited |
| Question: |
They write me that you did not apply to them.\"Hello, Maxim!Until we received a letter from Your hosting company, some information on the matter there.\" |
| Answer: |
the request sent again. |
| Question: |
Decide already, please this question soon, as you can already pull... |
| Answer: |
what do you mean tanut? Yandex robots we have not blocked. we don`t see the problem of hosting. |
| Question: |
Why did the website two weeks ago ceased to be indexed and Yandex says that this is a problem on the hosting side? |
| Answer: |
we have no information about the input of Yandex to your website for such period, the logs daily. |
