Getting a site’s homepage as well as its inner pages indexed by major search engines, such as Google, Bing, Blekko, Ask and Yahoo, is vital to your overall SEO (search engine optimization) efforts. If your web pages are not in search engine index, you can’t expect them to rank in organic search results. Here we’re going to discuss how to make sure every web page that matters to you is effectively crawled and subsequently indexed by search bots.
Make sure your website has a search engine-optimized design. It’s important to know that SEO starts with website design. While designing a site, ensure that it’s search engine-friendly, in the sense that you need to make proper use of Meta tags and Heading tags. Create a good internal linking structure, so that ever web page should be reachable from at least one static URL. Moreover, craft only search engine-friendly URLs, since search bots have difficulty crawling and indexing files that contain strange query strings.
Create quality and unique content, and optimize targeted keywords. Avoid duplicate content and prioritize important keywords using appropriate Heading tags, to enhance discoverability by search engine crawlers.
Create and upload robots.txt file. The robots text file is used to instruct search engine spiders which directories and web pages you want them to index and the ones you want them to exclude from their indexes. In case you don’t know how to create a robots text file, http://www.robotstxt.org/robotstxt.html has detailed instructions with example codes on how to get that done. A standard for robots exclusion can be found on http://www.robotstxt.org/orig.html. Use robots.txt file to invite search bots to index your web pages.
Create and upload an XML sitemap to your server. If you don’t know how to achieve that, the following site has detailed guidelines on sitemaps XML format: http://www.sitemaps.org/protocol.php.
Create an HTML sitemap file and upload to your server. An HTML site map is just an ordinary web page that contains links to all important web pages of a site. This file should include links to all the web pages that matter to you, which you want search engine crawlers to crawl and index, as well get your target audience to see them in one location.
Create accounts at Webmaster Tools. Create an account in Google Webmaster Tools, Yahoo Site Explorer and Bing Webmaster Tools. After doing that, officially submit your site, robots.txt and XML sitemap to each of them.
Point a couple of backlinks to every page you want indexed. This will help search bots to discover the web pages.
Use ping services: Ping services like Pingomatic.com help webmasters to inform search engines about website updates. Use them to notify search bots whenever you add new content or file to your website, since doing so facilitates quicker indexing.