What Should NOINDEX Do?
 By Navneet Kaushal
We have heard from Matt Cutts on many issues either through his blog or through the videos. The NOINDEX tag has been deliberated upon before by him, see: Handling Noindex Meta Tags and Google even made changes to how the tag was implemented, see: Google Accepts The "noindex" Directive In The robots.txt, Temporary or Permanent?.
Some time back Google came up with a video about removing your websites from its index, and later when it was discussed at one of the forums, many came to know for the first time, that the remove url feature came with an expiry of 90 days.
The latest post from Matt comes as a policy discussion about NOINDEX and how Google should treat the NOINDEX meta tag.
Based on a sample size of one page, different search engines have been found to handle the "NOINDEX" meta tag, sometimes differently:
• Google doesn't show the page in any way
• Ask doesn't show the page in any way
• MSN shows a url reference and Cached link, but no snippet. Clicking the cached link doesn't return anything.
• Yahoo! shows a url reference and Cached link, but no snippet. Clicking on the cached link returns the cached page.
However, the question now arises whether Google should completely drop a NOINDEX'ed page from its search results vs. show a reference to the page, or something in between?
Completely drop a NOINDEX'ed page
Google has implemented this behavior for past several years, and webmasters are used to it. The NOINDEX meta tag gives a way to completely remove all traces of a site from Google, however there's also the url removal tool. Sometimes it can so happen that if Google sees a link to a page 'A' but doesn't actually crawl the page, it won't know that page 'A' has a NOINDEX tag and then it might show the page as an uncrawled url. However, there's a remedy for that: currently, Google allows a NOINDEX directive in robots.txt and it will completely remove all matching site urls from Google. At the same time, this behavior could change based on the policy discussion about it.
Sometimes Webmasters hurt themselves by using NOINDEX, but if a site's traffic from Google is very low, the webmaster might want to diagnose the issue themselves. Furthermore, Google could add a NOINDEX check into the webmaster console to help webmasters self-diagnose if they've removed their own site with NOINDEX. Moreover, the NOINDEX meta tag serves a useful role that's different than robots.txt.
Show a link/reference to NOINDEX'ed pages
Google believes that its highest duty has to be to its users, not to an individual webmaster. So, when a user does a navigational query and Google doesn't return the right link because of a NOINDEX tag, it hurts the user experience. In case a webmaster wants to be out of Google's index completely, they can use Google's url removal tool. Although, the numbers are small, but Google has seen some sites accidentally remove themselves from the engine. An instance would be that if a webmaster adds a NOINDEX meta tag to finish a site and then forgets to remove the tag, the site will stay out of Google until the webmaster realizes what the problem is. Moreover, Google recently saw a slew of high-profile Korean sites not returned in Google because they all have a NOINDEX meta tag.
Eg.
the National Police Agency of Korea
the National Medical Center of Korea
Yonsei University
The contention is that because these high-profile sites aren't showing up in Google because of the NOINDEX meta tag, it's bad for users and hence for Google.
The Middle ground in between
A majority of webmasters who use NOINDEX do so deliberately and use the meta tag correctly, such as for parked domains that they don't want to show up in Google. Users are most disheartened when they search for a well-known site and can't find it. So what about the alternative that Google treats NOINDEX differently if the site was well-known. For example, if the site was in the Open Directory, then show a reference to the page even if the site used the NOINDEX meta tag, otherwise, don't show the site at all. In this way even though the majority of webmasters could remove their site from Google, but Google would still return higher-profile sites when users searched for them. |
0 comments:
Post a Comment