One of our (my) clients at work (my new job at OrangeSoda.com) had their website developed by this website development company called “ecreations.net”. Now I personally would think that a veteran website developer would know more about protecting website content and so forth than any other beginning webmaster or even search engine optimization specialists, but oh my!
They have been in business since 1997 and on their own website they have the most incompliant worst robots.txt file I have ever SEEN!!. (File saved on my server here.)
Ok so they put their entire meta tag header, which by the way is also the worst I have ever seen, and added it into the robots.txt file, they then proceed to add another list of keywords (as if the meta keywords tag duplicated twice wasn’t enough already enough) they add key phrases, one per line.
There is not one, not one line of code compliant with any robots.txt or search engine standard on this page, not one! This company could very well get you sensitive data indexed by search engines and you may not know until it is too late. So please be careful who you hire to do your website development. They may have been doing website dev work since 1997, but their ideas and knowledge level could also be just as old.