Short: Htdump v0.9 - the http protocol dumper tool
Author: louise@louise.amiga.hu
Type: dev/gg
Architecture: m68k-amigaos
Uploaded: louise@louise.amiga.hu (LouiSe)
Url: http://arak.cs.hro.nl/~ren/linux/
Ported by LouiSe
more info and other AMIGA ports at:
http://louise.amiga.hu
-----------------------------------------
HTDUMP
Copyright 2000 by Ren Hoek (ren@arak.cs.hro.nl)
Purpose
-------
A program to use in a script to easily dump http content
in a file.
The program will accept links in the standard URL form
and will support tweaking options as specifying the referrer
address and supporting resuming of files.
Importance is put on being sure files get transferred correctly
without errors and being able to tweak headers to overcome
anti-leeching procedures on websites.
Offcourse it's a valuable tool too for web developers like me
who work a lot of CGI's..
Ok most of the info is moved into htdump itself..
Try a 'htdump --help | less' and read. Or try the rather
complex example right below. On the website there are
more examples available. Don't hesitate to email me
if you have any questions.
(Note.. I need to make this pretty.. and more mainstream
perhaps.. dunno.. let's see if anybody reads this first
*grin*)
Example
-------
Netscape has released version 4.72 and this time everybody
is allowed to download the 128-bit strong encryption
version, except the usual 'evil' people. To this
end Netscape has devised a method to protect the download
of their program. Since I'm Dutch, I can download this
program and don't have to worry about probable domain name
checks to see if I'm from an 'evil' country.
There are 2 reasons why I wanted to see how this
particulair situation worked, because
I can't use a download manager like Getright
and Netscape froze up on me after it downloaded 89% of
the new install version. *grr* And also because
of all the check and weird checks, it's a good
test of htdump
So let's see if we can make it download..
First, let's see the page where we can download it
(line below is 1 line, I cut is in bits so you can
read it on 80 char monitors)
https://wwwus.netscape.com/usdl-bin/pdms_dnstest.cgi?
PRODUCT=communicator4.72-win32-en-complete-128&
COMPONENTS=CLIENT&TEMPLATES=NSCP
If we click 'Agree' a download is automatically
started.
View the source and see:
<FORM METHOD="POST" ACTION="pdms_promise2.cgi">
<INPUT TYPE="SUBMIT" NAME="Accept" VALUE="I Accept">
<INPUT TYPE="BUTTON" NAME="Accept"
VALUE="I Do Not Accept" onClick="window.history.back();">
<INPUT TYPE="hidden" NAME="PRODUCT"
VALUE="communicator4.72-win32-en-complete-128">
<INPUT TYPE="hidden" NAME="COMPONENTS" VALUE="CLIENT">
</FORM>
If we check out the info for this page you will see
Form 1:
Action URL: https://wwwus.netscape.com/usdl-bin/pdms_promise2.cgi
Encoding: application/x-www-form-urlencoded (default)
Method: Post
Ok now we have a target and we know what we have to post.
Only the hidden things get send to the cgi if I'm not mistaken..
SUBMIT's and BUTTON's don't get send.. So make the following
command (cut up again, it's 1 line)
htdump -d -P
"PRODUCT=communicator4.72-win32-en-complete-128&COMPONENTS=CLIENT"
https://wwwus.netscape.com/usdl-bin/pdms_promise2.cgi
-u "Mozilla/4.05 [en] (Win95; U ;Nav)"
I had to fidget around to get it to work though.. it turned out
that Netscape also checks on User Agent.. Leave it out and
you get an internal server error back. :) So I just inserted
something nice :)
And we get this result:
-------------------------------------
Username: []
Password: []
B64_pass: []
Host : [wwwus.netscape.com]
Port : [443]
Path : [usdl-bin/pdms_promise2.cgi]
Debug : [1]
-------------------------------------
POST /usdl-bin/pdms_promise2.cgi HTTP/1.0
User-Agent: Mozilla/4.05 [en] (Win95; U ;Nav)
Content-type: application/x-www-form-urlencoded
Content-length: 64
PRODUCT=communicator4.72-win32-en-complete-128&COMPONENTS=CLIENT
-----------------------------------
SSL connection using RC4-MD5
header: [HTTP/1.1 302 Moved Temporarily]
header: [Server: Netscape-Enterprise/3.6]
header: [Date: Sun, 12 Mar 2000 00:35:14 GMT]
header: [Content-type: text/html]
header: [Set-cookie: NSCP-US-DOWNLOAD=1IhWuzjK5kMAAAAASLm0goK_Rm
UVRvydzyKhqQ== ; expires=Sunday, 12-Mar-00 04:35:15 GMT ;
domain=.netscape.com; secure]
header: [Refresh: 0; URL=http://home.netscape.com/download/thanks.html]
header: [Location: https:pdms_pick2.cgi/cc32d472.exe?product=
communicator4.72-win32-en-complete-128&ext=.exe]
header: []
-------------------------------------
Content-length: 0
Errorcode: 302
-------------------------------------
Ok as you can see we get a load of data.. what is interesting
is the Cookie and the redirection we get. So a new target
(Again this is 1 line, just cut up to make it readable)
htdump
"https://wwwus.netscape.com/usdl-bin/pdms_pick2.cgi/cc32d472.exe?
product=communicator4.72-win32-en-complete-128&ext=.exe"
-c "NSCP-US-DOWNLOAD=1IhWuzjK5kMAAAAASLm0goK_RmUVRvydzyKhqQ=="
-f cc32i472.exe &
This will make it download in the background. Come back a few
hours later and it will be there complete. I tried to use
Ranges (-R) but it still spit out the same. It's a dumb script
that doesn't understand resume unfortunatly.. But even if
my primary goal failed I learned of the 'Location' header
which I never seen before :) You learn something every day.
I think you should be able to use htdump efficiently now.
Use the debug option to see what is going on, and experiment.
Have fun
Ren Hoek (ren@arak.cs.hro.nl)