The Facebook Crawler crawls the HTML of an app or website that was shared on Facebook via copying and pasting the link or by a Facebook social plugin. The crawler gathers, caches, and displays information about the app or website such as its title, description, and thumbnail image.
Rangeheader of the crawler request or it should ignore the
The Facebook crawler user agent strings:
To get a current list of IP addresses the crawler uses, run the following command.
whois -h whois.radb.net -- '-i origin AS32934' | grep ^route
These IP addresses change often.
... route: 188.8.131.52/21 route: 184.108.40.206/21 route: 220.127.116.11/20 route: 18.104.22.168/20 route6: 2620:0:1c00::/40 route6: 2a03:2880::/32 route6: 2a03:2880:fffe::/48 route6: 2a03:2880:ffff::/48 route6: 2620:0:1cff::/48 ...
If your app or website content is not available at the time of crawling, you can force a crawl once it becomes available either by passing the URL through the Sharing Debugger tool or by using the Sharing API.
You can simulate a crawler request with the following code:
curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "$URL"