The function of this component is to gather raw directory listings from FTP servers for later processing.
The robot has been implemented as two Perl programs - robot , which parses the list of the FTP servers for each domain begin indexed and forks a second Perl script, ftp_list , to actually fetch the directory listing for the server.
If it was able to start an anonymous FTP session, ftp_list proceeds to check whether there is an ls-lR.gz file. If one is present, this is fetched, otherwise the ls -lR command is issued from the / in order to produce a recursive directory listing.