Sitemaps

Site maps provide information about your website. Can be used by web crawlers to more efficiently search or fetch data. For example, a news fetcher can crawl several news websites using the sitemap to fetch news articles.

There is no standard place to put a sitemap so it is usually detailed in the robots.txt file found at the root of a website. i.e. http://www.washingtonpost.com/robots.txt