13 Eylül 2013 Cuma

Have You Considered Privacy Issues When Using Robots.txt & The Robots Meta Tag?

All E-Solutions Spotlights |





Understanding the difference between the robots.txt file and Robots  Tag is critical for search engine optimization and security. It can have a profound impact on the privacy of your website and customers as well. The first thing to know is what robots.txt files and Robots Tags are.


Robots.txt is a file you place in your website’s top level directory, the same folder in which a static homepage would go. Inside robots.txt, you can instruct search engines to not crawl content by disallowing file names or directories. There are two parts to a robots.txt directive, the user-agent and one or more disallow instructions.


The facebook sponsorlu reklam user-agent specifies one or all Web crawlers or spiders. When we think of Web crawlers we tend to think Google and Bing; however, a spider can come from anywhere, not just search engines, and there are many of them crawling the Internet.


The difference is the forward slash after Disallow:, signifying the root folder and everything in it, including sub-folders and files.


Robots.txt is versatile. You can disallow entire sub-folders or individual files. You can disallow specific search engine spiders like Googlebot and Bingbot. The search engines even extended robots.txt to include an Allow directive, file or folder name pattern matching, and XML sitemap locations.


What robots.txt does not do is to keep files out of the search engine indexes. The only thing it does is instruct search engine spiders not to crawl pages. Keep in mind that discovery and crawling are separate. Discovery occurs as search engines find links in documents. When search engines discover pages, they may or may not add them to their indexes.


See for yourself at  site:permanent.access.gpo.gov.


robotstxt-google-search


Using robots.txt to hide sensitive or private files is a security risk. Not only might search engines index disallowed files, it is like giving a treasure map to pirates. Take a look for yourself and see what you learn.


I used it to search for inurl:http://searchengineland.com/figz. As you can see, I found a few files I am probably not supposed to know about.


inurl-google-search


Don’t worry; if I had seen something risky or sensitive on Search Engine Land, I would never have shared this example. Can you say the same about your website or online application?


Because robots.txt does not exclude files from the search indexes, Google and Bing follow a protocol which does accomplish exactly that, the Robots tag.


Index or noindex instructs search engines whether or not to index a page. When you select index, they may or may not choose to include a webpage in the index. If you select noindex, the search engines will definitely not include it.


Follow or nofollow instructs Web crawlers whether or not to follow the links on a page. It is like adding an rel=”nofollow” tag to every link on a page. Nofollow evaporates PageRank, the raw search engine ranking authority passed from page to age via links. Even if you noindex a page, it is probably a bad idea to nofollow it. Let PageRank flow through to its final conclusion. Otherwise, you could be pouring perfectly good link juice down the drain.


A problem you will have with both robots.txt and the robots tag is that these instructions cannot enforce their directives. While Google and Bing will certainly respect your instructions, someone using Screaming Frog, Xenu, or their own custom site crawler can simply ignore disallow and noindex directives.


The only real security is to lock private content behind a login. If your business is in a competitive space, it will get crawled from time to time and there are few things you can do to stop or impede it.


One last note, I am not letting any cats out of the bag here. Pirates and hackers know all of this. They have known for years. Now you do, too.


Related Topics: All Things SEO Column | Channel: SEO


About The Author: Tom Schmitz is a longtime Internet marketing analyst and consultant specializing in inbound marketing, social media and SEO. He enjoys helping enterprise brands organize their Web presence and grow search engine and referral traffic. Tom began Internet marketing in 1996. You can read more of Toms musings at http://inboundbound.com. See more articles by Tom Schmitz


Follow @sengineland





Great article – I think it’s a very common misconception to think robots.txt will remove files from the index. Thank you for spreading the good word and helping keep everyone up on the less than admirable techniques that white hats might not be aware of.


Good, only a clarification. Most webmasters use both robots.txt and meta robots, thinking this is a double security, but the robots.txt block the spider that can not read the meta robots noindex instruction: it’s a possible own gol.


If people believe that using robots.txt to keep files secure they should choose a new profession. Anything that is private should be kept behind a login – it’s as simple as that.


I think, by-default it is allowed to access(crawl) all the web pages of your website and when you are using robots.txt then you are specifically disallowing pages of your website. Then why Google.com/robots.txt is using following line in robots.txt file ?


I like the Google thing (and I mean that sarcastically) that if you robots.txt a page, then they can’t read the meta tag. It’s one way of wanting to choose what does and does not go into the index, taking away control from website owners.


I get it. It’s their index and their search results. But I think website owners should ultimately pick. They probably know more about their specific group of users than a search engine does. They have better context (at least I hope they do).


I agree totally with you @tom clients are not aware about Google index, so they think because they changed their robots.txt every thing is solved. Google while refreshing its index doesn’t care about. The only way is to add noindex on page, and some times we should clear cache via GWT .. very time consuming. Hope standard will evolve to adapt to new comprehension we’ve about search engines.


What I have read and found is that the allow: is not needed. Stick to the disallow and forget about using allow: If you search the net for robots.txt file programs and research the proper usage and layout of one, you will find that the allow: is not actually needed when done properly.


up to I saw the bank draft four $4386, I be certain …that…my neighbour had been truly bringing in money part time on their apple laptop.. there sisters neighbour has done this 4 only about eight months and resently repaid the depts on there apartment and bourt a top of the range BMW M3. go to, jump15.comCHECK IT OUT


Robots meta tag is a good solution but if you’ve got a directory on your server with .pdf files or images or something else non-HTML that you do not want indexed or visible or don’t want anyone to know about you’re out of luck with robots meta. The only thing you can do in this case is password protect the directory.


See More Top News


See More News Briefs


See More Features


See More Columns


See More How To


Google May Be Updating Their Algorithm But They Won’t Confirm It


Bing Video Adds More Filters, Navigation & Pop Overlays


9 Link Building Lessons We Learned As Kids


10 Features Unique To Bing Ads That You Might Not Know About


Is PageRank Finally Dead? It Seems To Be, At Least In The Google Toolbar


Foursquare Now: Update Intros Location-Aware Alerts


6 SEO Challenges Every Business Owner & Marketer Must Contend With In 2014


Follow @sengineland


Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether youre just starting in search marketing or youre a seasoned expert.


Learn more about internet and search marketing with our free webinars, whitepapers and research reports at Digital Marketing Depot.


See More Topics


See more at Marketing Land


Hiç yorum yok:

Yorum Gönder