Robots.txt

Robots.txt is a text file placed in the root directory of a website that instructs search engine crawlers which pages or sections of the site they are allowed or not allowed to crawl and index. It is one of the first files a search engine bot checks when visiting a website. Blocking certain pages through robots.txt can help focus search engine crawl budget on your most important content. However, incorrectly configured robots.txt files can accidentally block important pages from being indexed, which can seriously harm SEO performance.