Tuesday, November 01, 2011<\/p>\n
\n As the web evolves, Google’s crawling and indexing capabilities also need to progress. We \n We generally advise to use \n We take precautions to avoid performing any task on a site that could result in executing an \n Let’s run through a few \n General advice for creating crawlable sites is found in our Your site shouldn’t look any different after such a change. If you see a blank page, a \n If you’d like to prevent content from being crawled or indexed for Google Web Search, traditional \n We’ll continue striving to increase the comprehensiveness of our index so searchers can find more
\n improved our indexing of Flash<\/a>, built
\n a more robust
\n infrastructure called Caffeine<\/a>,
\n and we even started
\n crawling forms<\/a> where it makes
\n sense. Now, especially with the growing popularity of JavaScript and, with it, AJAX, we’re
\n finding more web pages requiring POST<\/code> requests—either for the entire content of
\n the page or because the pages are missing information and\/or look completely broken without the
\n resources returned from POST<\/code>. For Google Search this is less than ideal, because when
\n we’re not properly discovering and indexing content, searchers may not have access to the most
\n comprehensive and relevant results.\n<\/p>\n
\n GET<\/code><\/a>
\n for fetching resources a page needs, and this is by far our preferred method of crawling. We’ve
\n started experiments to rewrite POST<\/code> requests to GET<\/code>, and while this
\n remains a valid strategy in some cases, often the contents returned by a web server for
\n GET<\/code> vs. POST<\/code> are completely different. Additionally, there are
\n legitimate reasons to use POST<\/code> (for example, you can attach more
\n data to a POST<\/code> request than a GET<\/code>). So, while GET<\/code> requests
\n remain far more common, to surface more content on the web, Googlebot may now perform
\n POST<\/code> requests when we believe it’s safe and appropriate.\n<\/p>\n
\n unintended user action. Our POST<\/code> requests are primarily for crawling resources that
\n a page requests automatically, mimicking what a typical user would see when they open the URL in
\n their browser. This will evolve over time as we find better heuristics, but that’s our current
\n approach.\n<\/p>\nPOST<\/code> request scenarios that demonstrate how we’re improving
\n our crawling and indexing to evolve with the web.\n<\/p>\nExamples of Googlebot’s
POST<\/code> requests<\/h2>\n\n
POST<\/code> XMLHttpRequest<\/code><\/i>: In this
\n step-by-step example, we improve both the indexing of a page and its Instant Preview by
\n following the automatic XMLHttpRequest<\/code> generated as the page renders.\n <\/li>\n\n
\n to render the page to better understand its content and\/or generate the Instant Preview.\n <\/li>\n
\n hot-fudge-info.html, using the POST<\/code> method.<\/p>\nPOST<\/code>, hot-fudge-info.html, along with its data payload,
\n is added to Googlebot’s crawl queue.\n <\/li>\nPOST<\/code> request to crawl hot-fudge-info.html.<\/li>\n
\n certain cases, we may also incorporate the contents of hot-fudge-info.html into
\n yummy-sundae.html.\n <\/li>\n
\n and we can properly display a snapshot of the page for Instant Previews.\n <\/li>\n<\/ol>\n<\/ul>\nImproving your site’s crawlability and indexability<\/h2>\n
\n Help Center<\/a>.
\n For webmasters who want to help Google crawl and index their content and\/or generate the Instant
\n Preview, here are a few simple reminders:\n<\/p>\n\n
GET<\/code> for fetching resources, unless there’s a specific reason to use
\n POST<\/code>.\n <\/li>\n
\n above, if hot-fudge-info.html is disallowed by
\n robots.txt<\/a>,
\n Googlebot won’t fetch it. More subtly, if the JavaScript code that issues the
\n XMLHttpRequest<\/code> is located in an external .js<\/code> file disallowed by
\n robots.txt, we won’t see the connection between yummy-sundae.html and hot-fudge-info.html, so
\n even if the latter is not disallowed itself, that may not help us much. We’ve seen even more
\n complicated chains of dependencies in the wild. To help Google better understand your site it’s
\n almost always better to allow Googlebot to crawl all resources.
\n You can test whether resources are blocked through
\n Webmaster Tools<\/a>
\n Labs<\/i> ><\/span>
\n Instant Previews<\/i><\/a>.\n <\/li>\n
\n Cloaking<\/a>
\n (sending different content to Googlebot than to users) is a violation of our
\n Webmaster Guidelines<\/a>
\n because, among other things, it may cause us to provide a searcher with an irrelevant result
\n —the content the user views in their browser may be a complete mismatch from what we
\n crawled and indexed. We’ve seen numerous POST<\/code> request examples where a webmaster
\n non-maliciously cloaked (which is still a violation), and their cloaking—on even the
\n smallest of changes—then caused JavaScript errors that prevented accurate indexing and
\n completely defeated their reason for cloaking in the first place. Summarizing, if you want your
\n site to be search-friendly, cloaking is an all-around sticky situation that’s best to avoid.
\n
\n To verify that you’re not accidentally cloaking, you can use
\n Instant Previews<\/a>
\n within Webmaster Tools, or try setting the User-Agent string in your browser to something like:<\/p>\n
\n JavaScript error, or if parts of the page are missing or different, that means that something’s
\n wrong.\n <\/li>\n
\n directly on the page and without requiring user-action to display. Most search engines are
\n text-based and generally work best with text-based content. We’re always improving our ability
\n to crawl and index content published in a variety of ways, but it remains a good practice to
\n use text for important information.\n <\/li>\n<\/ul>\nControlling your content<\/h2>\n
\n robots.txt rules<\/a>
\n remain the best method. To prevent the Instant Preview for your page(s), please see our
\n Instant Previews FAQ<\/a>
\n which describes the Google Web Preview<\/code> User-Agent and the nosnippet<\/code> meta<\/code> tag.\n<\/p>\nMoving forward<\/h2>\n
\n relevant information. And we expect our crawling and indexing capability to improve and evolve
\n over time, just like the web itself. Please let us know if you have questions or concerns.\n<\/p>\n