#website crawling

What Should a robots.txt File Include for a Small Site?

A small site usually needs a simple robots.txt file: identify crawlers, block only pages that should not be crawled, and point to your sitemap. Learn what to include and what to leave out.