Back

robots.txt

Robots.txt - REP

Robots.txt is the file used by Robots Exclusion Protocol to instruct search engine robots what not to index.

It has the following syntax:

User-agent: {useragent}

Disallow: {directory or file}

It is incorrect syntax to attempt to explicitly allow directories or files, and this may cause the robots.txt file to fail, leaving a whole website unprotected.

A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z