The topics in this section describe how you can control Google\u2019s ability to find and parse your content in order to show it in Search and other Google properties, as well as how to prevent Google from crawling specific content on your site.<\/p>\n\n\n\n
Here\u2019s a brief description of each page. To get an overview of crawling and indexing, read our How Search works guide.<\/p>\n\n\n\n
Google can index the content of most types of pages and files. The most common file types we index include:<\/p>\n\n\n\n
Text (.txt, .text, other file extensions), including source code in common programming languages: Basic source code (.bas)<\/p>\n\n\n\n
You can use the filetype: operator in Google Search to limit results to a specific file type. For example, filetype:rtf galway will search for RTF files with the term \u201cgalway\u201d in them.<\/p>\n\n\n\n
A site\u2019s URL structure should be as simple as possible. Consider organizing your content so that URLs are constructed logically and in a manner that is most intelligible to humans.<\/p>\n\n\n\n
Recommended : Use UTF-8 encoding as necessary. For example, the following example uses UTF-8 encoding for Arabic characters in the URL:<\/p>\n\n\n\n
If your site is multi-regional, consider using a URL structure that makes it easy to geotarget your site. For more examples of how you can structure your URLs, refer to using locale-specific URLs .<\/p>\n\n\n\n
Consider using hyphens to separate words in your URLs, as it helps users and search engines identify concepts in the URL more easily. We recommend that you use hyphens ( – ) instead of underscores ( _ ) in your URLs.<\/p>\n\n\n\n
Overly complex URLs, especially those containing multiple parameters, can cause problems for crawlers by creating unnecessarily high numbers of URLs that point to identical or similar content on your site. As a result, Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all the content on your site.<\/p>\n\n\n\n
Additive filtering of a set of items. Many sites provide different views of the same set of items or search results, often allowing the user to filter this set using defined criteria (for example: show me hotels on the beach). When filters can be combined in an additive manner (for example: hotels on the beach and with a fitness center), the number of URLs (views of data) in the sites explodes. Creating a large number of slightly different lists of hotels is redundant, because Googlebot needs to see only a small number of lists from which it can reach the page for each hotel. For example: Hotel properties at \u201cvalue rates\u201d:https:\/\/www.example.com\/hotel-search-results.jsp?Ne=292&N=461<\/p>\n\n\n\n
Dynamic generation of documents. This can result in small changes because of counters, timestamps, or advertisements.<\/p>\n\n\n\n
Problematic parameters in the URL. Session IDs, for example, can create massive amounts of duplication and a greater number of URLs.<\/p>\n\n\n\n
Sorting parameters. Some large shopping sites provide multiple ways to sort the same items, resulting in a much greater number of URLs. For example:https:\/\/www.example.com\/results?search_type=search_videos&search_query=tpb&search_sort=relevance&search_category=25<\/p>\n\n\n\n
Calendar issues. A dynamically generated calendar might generate links to future and previous dates with no restrictions on start or end dates. For example:https:\/\/www.example.com\/calendar.php?d=13&m=8&y=2011<\/p>\n\n\n\n
Broken relative links. Broken relative links can often cause infinite spaces. Frequently, this problem arises because of repeated path elements. For example:https:\/\/www.example.com\/index.shtml\/discuss\/category\/school\/061121\/html\/interview\/category\/health\/070223\/html\/category\/business\/070302\/html\/category\/community\/070413\/html\/FAQ.htm<\/p>\n\n\n\n
Consider using a robots.txt file to block Googlebot\u2019s access to problematic URLs. Typically, consider blocking dynamic URLs, such as URLs that generate search results, or URLs that can create infinite spaces, such as calendars. Using regular expressions in your robots.txt file can allow you to easily block large numbers of URLs.<\/p>\n\n\n\n
If your site has an infinite calendar, add a nofollow attribute to links to dynamically created future calendar pages.<\/p>\n\n\n","protected":false},"excerpt":{"rendered":"
The topics in this section describe how you can control Google’s ability to find and parse your content in order to show it in Search and other Google<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-30483","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/posts\/30483","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/comments?post=30483"}],"version-history":[{"count":1,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/posts\/30483\/revisions"}],"predecessor-version":[{"id":30792,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/posts\/30483\/revisions\/30792"}],"wp:attachment":[{"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/media?parent=30483"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/categories?post=30483"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/alienroad.com\/wp-json\/wp\/v2\/tags?post=30483"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}