If there are no errors Google will give you the green light, otherwise it will highlight the incorrect text string or strings so as to allow you to identify the error and resolve it before putting it online. The robots.txt file and WordPress Yes, I know, as soon as you read this little title you had a heart attack. Don’t worry, everything you have read up to now also applies to you who use WordPress , no more, no less. There are only a few details you need to take into account, namely: Before 2012, WordPress sites had to block access to the -admin and -includes folders with the robots.txt file.
CMS settings and the result will be this
This means you can’t edit Israel Telegram Number Data them directly you can only. Create a new one and replace it with your root directory . What not to do on robots.txt file? To make sure you transform the robots.txt file into an ally for the SEO optimization of your site, your first step must be to test it through the appropriate Google tester so as to check that the robots do not end up obscuring pages that you want to appear in the search engine results instead. Another fundamental piece of advice is to never block the CSS or JS folders : crawlers need to see your website as a real, flesh-and-blood user would see it; Therefore, if your pages require CSS and JavaScript to function, these cannot be hidden from bots . Clear?
Don’t dare – unless it is strictly necessary – give different directions to different search engine bots :
you could end up creating Kenya phone number database confusion. To this it must be added that it would be very difficult to keep the robots.txt file . Updated with these differentiated specifications. My advice for having a truly optimized robots.txt file is to use the user-agent. So as to have unique directives for all the different crawlers . If your aim is solely and exclusively to block specific pages from being indexed. I advise you not to go through the robots.txt file, but rather to use the “” in the header section . Of the individual pages or alternatively modify the files .