What is Curl ?
[email protected]:~# curl –man
DESCRIPTION
curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.
curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more.
Curl is quite handy when it comes to scripting stuff. I use curl a lot in my daily work, it does everything you may want from authenticating to the server, obtaining cookies, using the existing cookies, file upload and many more.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Apache2 Debian Default Page: It works</title>
1. Pretty Curl
So we access Apache Default page via curl. We can make the output prettier with html2text.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10701 100 10701 0 0 1492k 0 --:--:-- --:--:-- --:--:-- 1741k
[Debian Logo] Apache2 Debian Default Page
It works!
2. Verbose Curl
With -v option so you see the HTTP requests and responses.
..SNIP...
(192.168.207.188) port 80 (#0)
> GET / HTTP/1.1
> Host: 192.168.207.188
> User-Agent: curl/7.61.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 10 Mar 2019 10:13:04 GMT
< Server: Apache/2.4.27 (Debian)
< Last-Modified: Mon, 18 Sep 2017 18:46:57 GMT
< ETag: "29cd-5597b2ac5aa40"
< Accept-Ranges: bytes
< Content-Length: 10701
< Vary: Accept-Encoding
< Content-Type: text/html
<
...SNIP...
[Debian Logo] Apache2 Debian Default Page
It works!
3. Silent Curl
We can also make the curl silent to not to have upload/total/spent/left/speed values in the output with -s.
[Debian Logo] Apache2 Debian Default Page
It works!
4. Authentication with curl
Using -u option with crendetials, you can login to the website.
Input recoding failed due to invalid input sequence. Unconverted part of text follows.
�|OK|
5. Obtaining Cookie with Curl
6. File Upload with Curl
Notice that we are using the existing cookie from the above request.
7. Follow Redirects with Curl
Let’s say that you access url which redirects to another location. Curl has -L option to follow the redirects.
8. Proxy with Curl
If you are troubleshooting a request and you would like to see the headers of it, you may want to use proxy or you may simply want to use a proxy to access a site. –proxy option is really useful.
9. Curl to https sites
-k, –insecure Allow insecure server connections when using SSL .
10. Post Request with Curl
I have already provided an example above in Obtaining Cookie with Curl section which is a post request login.
There are more options including data-binary and data-ascii
- -d, –data HTTP POST data
- –data-ascii HTTP POST ASCII data
- –data-binary HTTP POST binary data
- –data-raw HTTP POST data, ‘@’ allowed
- –data-urlencode HTTP POST data url encoded
- –delegation GSS-API delegation permission
- –digest Use HTTP Digest Authentication
Let me know please, if you need me to add any other useful tricks with curl.