robots

What is ROBOTS meta tag ?

The ROBOTS meta tag is a special meta tag which is meant to be read by web crawlers such as Google, Bing, and others. The robots meta tags tells the web crawler how to handle the page that contains them. It is mostly used by web masters to request crawlers to ‘not add this page to the index’.

The most common way is to add these lines to the head section of your html:

<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">

This line above will tell the web crawler not to index a web page, but DO follow the links of this page.

Why would someone ask search engine to avoid indexing  ?

There could be multiple reasons that a site owner would ask a web crawler to skip indexing of a specific page. There could be many pages in a website that are generally lacking real value to search engines users.

Examples for such pages are :

  • Login page
  • Signup page
  • Members only pages
  • My shopping cart page (for e-commerce websites)
  • You got the idea.

For search engines, such pages do not hold real value, as the content in those pages is rarely to match any search query, or answer any question that user may ask the search engine.

After recent updates to google’s search ranking algorithm, the term “Quality Content” started to get new meanings by different experts in the field, causing many web masters to add the ROBOTS meta tag and ask web crawlers not to add certain pages of their website to the search index. The buzz was mostly around “Categories” pages and “Tags” pages because these pages usually contain parts of content of other pages, making webmaster concerned about these pages being marked as “Duplicate content”.

There is a common opinion among web masters that “Duplicate” content cannot be considered as “Quality” content. For example, anyone who will read this article, might think that categories and tags pages should not be indexed, as they are not considered “Quality” content. This is a good example of how misleading information can propagate deep into the minds of even the most experienced web masters. In his article, Mr. Harsh Agrawal is actually saying that the categories pages in his own blog are not considered ‘Quality’ content for search engine. Its hard to agree with such statement shoutmeoutloud.com is one of the most reputable blogs around, and its categories and tags pages are actually on the quality side. Having a page on your website that organizes your content in a different way, more targeted to the user, can be very useful for the readers. So the bottom line is that shoutmeoutloud.com is an awesome blog, including its categories and tags pages.

Before rushing into conclusions about what page is considered a “Quality” page and what isn’t, Web masters should really think about each page as a standalone unit and figure out if that page has any value to the users of the website or not. Duplicate content does not automatically mean that users cannot benefit from reading it. If the content is organized in particular logical way, around a specific topic, or tag, it can be very useful.

Is it natural to write to a robot ?

On the other side of the ROBOTS meta tag, stands the fact that adding such instructions to your website is in somehow similar to talking with a robot.Keep in mind that highly over optimized website may also experience some kind of penalty for being too optimized.

Conclusion:

Quality” content does not mean that a website should talk to a robot. Write content for users, the robots will follow.