Problems Crawling

Problems Crawling and Indexing

  • Online forms: Search engines aren't good at completing online forms (such as a login), and thus any content contained behind them may remain hidden.
  • Duplicate pages: Websites using a CMS (Content Management System) often create duplicate versions of the same page; this is a major problem for search engines looking for completely original content.
  • Blocked in the code: Errors in a website's crawling directives (robots.txt) may lead to blocking search engines entirely.
  • Poor link structures: If a website's link structure isn't understandable to the search engines, they may not reach all of a website's content; or, if it is crawled, the minimally-exposed content may be deemed unimportant by the engine's index.
  • Non-text Content: Although the engines are getting better at reading non-HTML text, content in rich media format is still difficult for search engines to parse. This includes text in Flash files, images, photos, video, audio, and plug-in content
Previous
Next Post »