Back

Robots Exclusion Protocol

REP - Robots Exclusion Protocol

REP is a system that allows you to instruct (complying) search engines which parts of your web server they should NOT index.

All the major search engines comply with REP, although if you update it, it may take a few days for the robots to change their indexing habits.

REP is useful for protecting private files (although conversely it also 'signposts' them to less-reputable software), and also for reducing unnecessary bandwidth usage.

A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z