Support checking robots.txt #11
Description
IpAddressExpress should check robots.txt file on the webserver of the ipservice to make sure the configured ipservice does allow IpAddressExpress to make request to it.
If IpAddressExpress figures out that robots.txt contains:
User-agent: IpAddressExpress
Disallow: /
It should disable the ipservice to be used in the database.
robots.txt should be checked once every week for every ipservice and at startup of IpAddressExpress.