blob: e3e1aed11a1a2e4c442c52094cb0bc0af66e3f96 (
plain) (
blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[
{ type: install
message: <<EOM
You can download a new version of the Robot Database from https://www.robotstxt.org/db/all.txt.
Download the database dump, and run the file robot_detection.py with the file as first argument.
$ fetch https://www.robotstxt.org/db/all.txt
$ python robot_detection.py all.txt
If the database has changed, it'll print out the new version of robot_useragents variable that you need to put into the source code.
EOM
}
]
|
