Google parses the file with specific rules that differ from naive expectation. Getting them wrong usually fails silently.
Precedence
The most specific rule wins, measured by path length. If Disallow: /folder/ and Allow: /folder/page both match, the longer Allow wins. When rules are equally specific, the least restrictive applies.
Wildcards
*matches any sequence of characters.$anchors the end of the URL.Disallow: /*.pdf$blocks PDFs but not/file.pdf?download=1.
User-agent groups
A crawler obeys the single most specific group that names it, and ignores all others. A rule under User-agent: * is not merged with rules under User-agent: Googlebot, if a Googlebot group exists, the wildcard group is ignored completely for Googlebot.
Limits
Google reads up to 500 KiB; anything past that is ignored. Comments start with #. A 4xx response is treated as “no restrictions”; a persistent 5xx is treated as “disallow everything”.
Alien Road recommends: test every change in the Search Console robots.txt tester before deploying. The user-agent group rule catches people constantly, adding a Googlebot-specific line silently deactivates every wildcard rule you had, and the file still looks correct to a human reader.