Resources

Robots.txt - SEO Glossary

Robots.txt - SEO Glossary

A special file that is commonly used to exclude some or all robots from crawling certain files or directories on a website. This file should be placed in your website's root directory.