3

I can navigate to a web site using a web browser. But trying a GET request with curl hangs. For example, the command curl -vL my_url gets stuck with the following log:

* Host $url:443 was resolved. * IPv6: *** * IPv4: *** * Trying ***:443... * Connected to $url (***) port 443 * ALPN: curl offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CAfile: /etc/ssl/cert.pem * CApath: none * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Unknown (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / [blank] / UNDEF * ALPN: server accepted http/1.1 * Server certificate: * subject: C=**; ST=***; L=***; O=***; CN=*** * start date: Nov 17 05:09:32 2023 GMT * expire date: Nov 17 14:59:59 2024 GMT * subjectAltName: host "***" matched cert's "***" * issuer: C=**; O=***; CN=*** * SSL certificate verify ok. * Certificate level 0: Public key type ? (2048/112 Bits/secBits), signed using sha256WithRSAEncryption * Certificate level 1: Public key type ? (2048/112 Bits/secBits), signed using sha256WithRSAEncryption * Certificate level 2: Public key type ? (2048/112 Bits/secBits), signed using sha256WithRSAEncryption * using HTTP/1.x > GET / HTTP/1.1 > Host: *** > User-Agent: curl/8.6.0 > Accept: */* > * old SSL session ID is stale, removing

What's the reason for this? I'm sorry, I can't reveal the website URL. I'm hoping that the above will reveal some information. I tried adding all the request headers from the browser, and the behavior doesn't change.

DNS lookup reveals the following information:

*** is an alias for ***.edgekey.net. ***.edgekey.net is an alias for ***.akamaiedge.net. ***.akamaiedge.net has address *** ***.akamaiedge.net has IPv6 address *** ***.akamaiedge.net has IPv6 address ***

EDIT: Based on further investigation, it's not just curl, all non-browser clients I tried (postman, curl, wget, openssl s_client to get SSL certs) times out. I have tried fetching the pages using the exact same request headers with curl/postman as the browser without success.

Improve this question
5
  • 1
    Does the website block requests from User-Agent: curl? Does it block any request that cannot handle javascript? Any logs or configurations from the server you are attempting to connect to? Commented Feb 22, 2024 at 11:54
  • 1
    User-agent, cookies, etc. Commented Feb 22, 2024 at 12:03
  • User-Agent doesn't help, I tried using postman with the same request headers, including User-Agent and it doesn't help. Cookies and Js, not sure. How can I test that? If it's based on cookies/Js, can the request be blocked at that level - as in request timeout at the network level? Commented Feb 22, 2024 at 12:59
  • FWIW: I'm having a similar problem. A work-around that worked briefly for me was the aria2 app, but then that was also blocked. Another work-around that I've got some hope for is lynx. Commented Feb 26, 2024 at 22:01
  • github.com/salesforce/ja3 after i saw this project long ago i assumed this was the most robust way to inhibit web scraping. i.e. fingerprint the bytes sent on the wire from the start. Akamai is certainly in that business. faking the user-agent in the case of TLS fingerprinting actually makes things worse (easier to filter). but, the server is still playing the obscurity game as there is likely a combination of client request that still works. something like Selenium; eg Cypress, Puppeteer, Playwright etc. seem to be an obvious workaround to me Commented Mar 2, 2024 at 7:35

3 Answers 3

Reset to default
1

as others have pointed out, check if the website has this in their robots.txt

User-Agent: curl Disallow: /

if it's there then you can't use curl with that particular website

if it's this then you can't do any crawling

User-agent: * Disallow: /

to find it just type /robots.txt after the url example.com/robots.txt

Improve this answer
6
  • Can you explain how to get a website's robots.txt file? Commented Feb 22, 2024 at 21:00
  • Does robots.txt file config causes even the openssl s_client to fail? Based on further investigation, it's not just curl, all non-browser clients I tried times out. Commented Feb 22, 2024 at 23:26
  • @Seamus fixed, thanks Commented Feb 23, 2024 at 3:32
  • you're welcome. and i wonder: do all webservers use the robots.txt scheme, or are there other schemes to control the robots (last question - i promise) Commented Feb 23, 2024 at 14:18
  • @Seamus Since it's a normal URL, all servers support it, but the question is whether the client respects it. The server doesn't do anything except provide robots.txt if the client requests it. To my knowledge, and I would gladly be further enlightened, enforcement of the refusal of robots.txt entries must be done via the web server's control file mechanisms. robots.txt is simply provided as a courtesy to the client to say, "I'm going to ignore any of these requests, so don't waste your time." Commented Feb 26, 2024 at 19:30
1

I managed to get the curl request working by adding the following headers. Note than all of them were needed, just the user-agent header wasn't enough:

> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0 > Accept-Language: en-US,en;q=0.5 > Accept-Encoding: gzip, deflate, br > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8

I obtained them by looking at the request headers in the network tab of the browser's developer console.

Accept header can also simply be Accept: */*.

Improve this answer
1
  • Asked and answered, well done! Commented Mar 4, 2024 at 2:52
0

I'm having the same problem with a website I track (that's what originally led me to your question). Oddly, I've been tracking this website for years (using curl) without issue - but now it's a "problem"! Maybe this is a trend?

Anyway - since reading your question, I've found a solution that works for me: lynx - the "command line" web browser. lynx has a ton of options, so you may need to invest some time in reading man lynx, etc. Here's what's working for me:

lynx -dump -accept_all_cookies 'https://wherever.com/viewtopic.php?t=abc123' > my-dump.txt
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.