In an attempt to index more content, more quickly, Google has begun letting Web-site operators point the search engine’s crawler at Web pages and tell it when content is updated.
Google Inc. started the beta program, called Google Sitemaps, Officials said it would help supplement its Web crawling technology in order to better index dynamic content and frequently refreshed pages.
Google also has released a tool called the Sitemaps Generator, which allows site operators to generate an XML-based sitemap. Google is offering the Python-based tool to open-source developers.
By letting Webmasters submit their sitemaps of URLs and other information, Google also is acknowledging that technology alone cannot find and catalog everything on the Web