Common errors

Yandex’s list of common errors is the most useful page in its recommendations, because each item describes a site that looks correct to its owner and reads wrongly to the robot.

The eight mistakes

• Navigation through scripts. The robot does not follow links built in JavaScript or Flash. Duplicate them with ordinary text links, or those sections are undiscoverable.
• <iframe>. Documents loaded into an iframe are not indexed, so content placed there does not count toward the containing page.
• Excessive automatic redirects. Avoid them except where a URL genuinely changed for technical reasons.
• Unstable URLs. Every page should live at a single permanent address — no session identifiers, and as few CGI parameters as possible.
• Cloaking, including the innocent version: regional site variants serving the robot something different from the user.
• Images instead of text. Yandex gives a specific and underappreciated reason: most external links point at the home page, so a home page that is an image with no text makes content analysis less reliable — the site’s strongest signal lands on a page the engine cannot read.
• Soft 404. Returning 200 with a stub page for missing URLs means the engine believes the page exists, keeps it in the database, and slows the indexing of useful pages.
• Faulty site engine. Script errors that produce different URLs for the same page depending on the route taken; and engine vulnerabilities that let attackers insert links to malicious sites.

The closing rule

“The simpler and clearer your site is, the better it is indexed.” That is Yandex’s own summary, and it is the correct test for any architecture decision — a clever navigation scheme that requires the engine to understand a framework is a decision to be worse understood.

The two most common in practice

Soft 404s and script-only navigation. Both are invisible in a browser, both are usually introduced by a framework default rather than a deliberate choice, and both directly reduce how much of the site gets indexed.