This month we take a look at Curl, the open-source tool that lets you send and receive Web pages from the command line. Curl makes it easy to automate many security and administrative tasks, such as fetching a Web page for analysis or downloading a security patch from the Web at the command line.
Installing Curl
Curl is included in most UNIX distributions and provides binaries and source code for most other OSs. Even developers using the open-source PHP: Hypertext Preprocessor language (PHP) can take advantage of Curl, which offers a more secure method of accessing Web content directly from their PHP scripts.
When used in conjunction with Secure Sockets Layer (SSL) Web sites, Curl depends on the OpenSSL package. Curl is available in two versions: one with SSL and one without. Because you'll probably want to work with SSL-protected data, I recommend that you install the SSL version.
With the SSL version, you must download and install the OpenSSL package separately and before you use Curl. You can get OpenSSL for Windows binaries from the Gnu-Win32 SourceForge project at http://gnuwin32.sourceforge.net/packages/ openssl.htm. Check out this site—it also provides a lot of useful UNIX tools ported to Windows.
Download and install the OpenSSL package, then copy the two DLLs into your system32 directory:
copy "C:\Program
Files\GnuWin32\
bin\libeay32.dll"
%windir%\system32
copy "C:\Program
Files\GnuWin32\
bin\libssl32.dll"
%windir%\system32
Now install Curl. You can find the SSL-supported Curl binaries for Windows at http://curl.haxx.se/latest.cgi? curl=win32-ssl-sspi. The most recent version, curl-7.15.0-win32-ssl-sspi.zip, contains the curl.exe file and supporting documentation you'll need.
After you've installed Curl, test that it's working by typing
curl http://isc.sans.org/
infocon.txt
at a command line. If a color word (e.g., green) appears, Curl is working. In this very basic example, Curl fetches the Infocon status from the SANS Institute's Internet Storm Center Web site. Green means that the Internet is operating normally and no significant threat is known. If instead of green you see yellow, orange, or red, put down this article and visit http:// isc.sans.org to learn about the heightened Internet threat level. If you get an error, check your Curl installation.
Simply put, Curl fetches a Web page, then returns the page's HTML to the console. But Curl does more. Curl has built-in error checking. For example, typing
curl http://noserverhere
returns the error Curl: (6) Could not resolve host: noserverhere; Host not found. You can use error return codes in your scripts to test whether a Web page is accessible or whether a Web server is responding. For example, if you use Curl to fetch a Web page nightly—say, that day's Web site statistics—you could include in your script code that looks for error codes. Then, if Curl returns with a code Curl: (7) couldn't connect to host, you could send an alert or email notification right away.
Fetching Encrypted Data
One of the most important benefits of Curl is that it supports SSL. When Curl requests HTTPS pages, the pages are encrypted when traversing the network, then Curl displays unencrypted text. Curl also checks certificates—the certificate expiry date, whether the host name matches the host name in the certificate, and whether a root certificate is trusted—and warns you if a certificate isn't fully legitimate. You can specify a particular certificate file by using the --cacert file parameter. To disable certificate checking, use the -k parameter. (Alternatively, you can use the --insecure option.)
Not Just for the WWW
Curl provides more than simple Internet file transfers. You can use Curl to get a quick directory of an FTP site by typing
curl ftp://myftpsite
To see the site's subdirectory, type
curl ftp://myftpsite/subdir/
To download a file, simply specify the filename in the URL. The following example downloads a file called readme.txt directly from the command prompt and displays the file on your screen:
curl ftp://ftp.microsoft.com/
deskapps/games/readme.txt
It's often easier to script Curl for grabbing FTP files than to use an interactive FTP command.
By default, Curl displays output directly to the console, but you can redirect the output to a file by using the -o and -O parameters. (Of course, you'll want to redirect binary files to disk, unless you want to see binary code scroll across your screen.)
Specify -o when you want to get the page and store it in a local file. Specify -O to store the retrieved page in a local file and have Curl get the name of the remote document. (If the URL doesn't specify a filename, this action will fail.) If you use Curl to make a request to a Web site that has no filename and you want to save the output to a file, you can specify a filename on the command line, like this:
curl -o whoisOutputFile
http://www.arin.net/whois/
Prev. page
[1]
2
next page 
