- Community
- Question
Hello,
Since is seems almost to contact FB concerning about from the Crawler (no reply from noc, abuse, and customer support is closing tickets without looking) we are started to get a bit annoyed about the issue we are facing.
Since a few days the
De User-Agent from the http request is : facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
We also checked that all IP addresses are originating from your datacenters as described on : https://developers.facebook.com/docs/sharing/webmasters/crawler/
Due to the crazy amount of request the load on our server increased from 0.5 to 12-15 average, and websites hosted on this server were almost unusable.
Because of that we needed to add a ratelimit on the user-agent facebookexternalhit/1.1
(+http://www.facebook.com/externalhit_uatext.php) but this has side effects, that when people are sharing a post, the crawler cannot always fetch the information from the webserver on our server, because it's being limited by the server.
To give some idea about the amount of requests :
root@servername:/home/httpd/vhosts/website.nl/logs# zgrep -c facebookexternalhit access_ssl_log.processed.5.gz 83161 root@servername:/home/httpd/vhosts/website.nl/logs# zgrep -c facebookexternalhit access_ssl_log.processed.4.gz 197321 root@servername:/home/httpd/vhosts/website.nl/logs# zgrep -c facebookexternalhit access_ssl_log.processed.3.gz 137907 root@servername:/home/httpd/vhosts/website.nl/logs# zgrep -c facebookexternalhit access_ssl_log.processed.2.gz 84284 root@servername:/home/httpd/vhosts/website.nl/logs# zgrep -c facebookexternalhit access_ssl_log.processed.1.gz 62772
This is from the last 5 days, where we started limiting about 3days ago, but when disable the ratelimit again, the load of the server is back to 15-20 in 2 minuts.
Please take this message seriously. i've tried almost anything to find a solution.
It does not help there is no point of contact for this API / Crawler,the information site is a great place to post contact info: https://developers.facebook.com/docs/sharing/webmasters/crawler/
I can share the info about the website, but i rather do that in DM .
Can you please help for this issue?
We are also experiencing issues with facebookexternalhit/1.1 starting May 25
We have fully blocked the user-agent after confirming all requests are coming from Facebook IP addresses. We received 40k requests starting May 25, 2024.
The user-agent "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" is apparently from Facebook's "Sharing Debugger" tool or from a "Social Plugins" rather than it's FacebookBot crawler.
And, furthermore, it is very suspicious to us that shortly after these thousands of requests began, we also started experiencing a card testing attack. However, the card testing has continued despite the block on facebookexternalhit/1.1 requests
Similar to above, my server suffered several tens of thousands of requests from bot facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Now I have to block this bot. I wonder if you have any solution or information about this facebookexternalhit/1.1. Why does it continuously send requests to the server?
I notice the same problem on several stores. The correlation I see is that queries are being generated to links like: ‘shop/[paramsfilter]’
Heads up that it seems that some of these requests are shifting over to the "meta-externalagent/1.1" crawler
I had to block Facebook access!! My site is down and even the site server is down
Same for my site, I have 4232 requests from facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) today till 4pm, getting worse up to a request every 3 seconds.
My hosting providers blocks access to the database for every user due to anormal amount of requests.
To get out of this situation, I banned FB host, give time penalty and send to 404. Just get away bots!
we are seeing this, for many clients on different servers, they start getting 1000s if not 10.000s of facebookexternalhit request and most of them are towards urls that don't exists. it does seems facebook is being abused somehow..
It just happened on one of our websites, 5k requests in less than 2 minutes.
User Agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
I'm facing this same problem as well.
We ended up blocking the useragent "facebookexternalhit" in cloudflare like others here, which unfortunately causes links on our site to not be able to be posted to FB with content previews.
I've filed a bugreport for this, hopefully they are going to look into this issue.
If you have any information, please leave a message here to let me know. Thank you.
Same me! I use Security WAF of cloudflare block User Agent "facebookexternalhit". currently everything seems good.
We also experience this and Facebook is doing NOTHING against it. Obviously it would be very easy: if a human being is sharing a link on Facebook, the useragent should be a different one compared to those made through an API. This way it would be so easy to distinguish these two cases.
As of today it started again after being quiet for almost a week. getting really tired of this.