Command line and www

new2net

Hey guys! Hoping someone could help me out with a question. I used nc to connect to a web server on my LAN. The connection opens and I am able to use GET to display text files right on the command line. This is what I want. However I would also like to use GET to pull images off the web server. The problem is that when I say GET picture.png, a whole bunch of garbage is returned to the command line.

Is there a way to have GET pull the image and redirect it to something like Gimp (or any other application)?

Thank you.

paul78

Ahh... This would have made a good question of the day

Netcat probably isn't the best tool to do what you want. A more appropriate tool would be curl or wget.

But since you are asking about netcat...

The netcat app makes arbitrary connections tcp connections. It's basically like simply using telnet but without any terminal emulation. To retrieve an image from a web server, you would issue the correct command.

Sounds like you figured that part out. But the output that you are getting is being directed by netcat to your screen (I.e. stdout or maybe it's stderr - can't remember what nc does). So basically you have to redirect it to a file and then you can open that file with your app.

The problem with using nc is that you will also get the HTTP responses in the output - Ie the 200 OK response. So you will have to pipe the output to sed and filter out only the file that you want.

If Gimp or your image manipulation app will accept the image via a pipe - you can add that.

new2net

Ahh, thank you Paul78.

I looked into using the pipe prior to posting, but I think I am getting the syntax incorrect. For example, I have tried the following:

GET picture.png >> gimp

and

GET picture.png > gimp

My understanding of this is that it will ask for picture.png and then direct the output to gimp. However this does not work. Is there an issue with my syntax?

When you say pipe the output to a file... do you mean something like this:

GET picture.png > picturefile.png

From what I have looked up, the > operator will perform the redirection. Correct?

Thank you very much for the response. Please let me know if you have any other suggestions.

brownwrap

I think you are confusing redirection with a pipe. I think he is suggesting:

GET picture.png | gimp

paul78

yes - that's right I meant pipe which is |
To redirect to a file you would use > operator.

new2net

Hey guys,

I tried GET picture.png | gimp

The picture still doesn't open in Gimp. Instead a whole bunch of characters are returned to the command line.

Any other thing I may try....?

paul78

Hi - for some reason my post edit didn't take. How annoying -

Anyways - this is what I was trying to post before....

I am assuming that you are using a command like this:

echo -e "GET /myimage.png HTTP/1.0\r\n\r\n" | nc my.web.server 80

This will generate the output to stdout (or maybe it's stderr don't remember how netcat outputs).

So to get it to a file you have to redirect it:

echo -e "GET /myimage.png HTTP/1.0\r\n\r\n" | nc my.web.server 80 > myfile.txt

But if you examine myfile.txt - you will notice that it also outputs the HTTP responses like the 200 OK message.

So before you redirect it to a file or pipe it to gimp, you need to parse out the messages.

If you know which HTTP server is being used and you expect the raw HTTP response to be consistent - you can simply do something like this:

echo -e "GET /myimage.png HTTP/1.0\r\n\r\n" | nc my.web.server 80 | tail -n +12 > myfile.txt

The tail -n +12 will remove the first 12 lines before the output is redirected to the file. But the better technique is to use sed but my regex is pretty poor so maybe someone smarter can offer that tip

The best way is probably still to use curl or wget.

new2net

Thank you Paul! That is exactly what I needed!!

One thing though...the following command seems to make the contents of myfile.txt complete empty: tail -n +12 > myfile.txt... when I used the command without the tail option, it gives my the intended file, however it does contain the HTTP messages...

Is there a way to remove the first two lines of myfile.txt...? The first two lines contain HTTP 200 and Content Length.

paul78

It's empty because tail -n +12 will remove the first 12 lines of output. So if you want to just remove the first 2 lines. It's tail -n +2.

If you do a man tail to take a look at the manual page, you can find some of the options and meanings.

Also - assuming you have wget installed - do a man wget and you will probably find wget a more appropriate tool than netcat.

If you are new to Linux and UNIX based operating systems, getting to understand how a man page is setup could be useful. It can take a little while to get used to the stylistic and formatting guidelines used for man pages. But once you get used to it, you will find it using the manual pages invaluable - a man man command can give you a start.

hiddenknight821

First, I want to thank Paul78 for replying to the OP's question, which challeged me to think. Otherwise, I wouldn't be able to test the suggested solution as I was not sure what the OP was looking for. Indeed, the question should be posted in the existing Question of the Day thread, which would make this more fun.

Here's an example of what I did to test Paul78's suggestion. I tried to retrieve the picture of the logo from the Google Images homepage at images.google.com. Rather than using the tail command, I decided to put my sed knowledge to use since I really need more practice with it. For those of you that are not familiar with sed. It's a Stream Editor, which edit your file on the fly rather than editing it manually in vi.

echo -e "GET /intl/en_ALL/images/logos/images_logo_lg.gif" | nc images.google.com 80 | sed -n -e '1,/^^M$/!p' > images.gif

I must say this was my first encounter with nc command, and after looking into it, I figured I'll use it more often once I start building networking tools in Linux. Let me break down what the sed command does.

First, the -n option will supress any output that was not matched by the expression I used. Othewise, all output will be printed with the new edit that's being made. This isn't what we want since we do not want the first 11 lines here. the -e option is needed when using in-line command script rather than sed script from a file. Here's an excerpt of the nc output from the example:

HTTP/1.0 200 OK
Content-Type: image/gif
Last-Modified: Mon, 02 Apr 2012 02:13:37 GMT
Date: Sun, 21 Apr 2013 00:09:16 GMT
Expires: Sun, 21 Apr 2013 00:09:16 GMT
Cache-Control: private, max-age=31536000
X-Content-Type-Options: nosniff
Server: sffe
Content-Length: 8561
X-XSS-Protection: 1; mode=block

GIF89a^T^An^@÷ü^@^@s^K^C|^Nq^Z^Yl#%m-$s.(v/K~S<96>^E^_<9a>^F!<90>^V(©   $¸^K'µ^V,<8d>+7±)8Ã^M*Ë^M+Ô^N-Ç^U+Ì^S,Ô^T-×^V.Û^Z/Í^]5Ü^W3Ý^[2á^Z/á^Y1â^[2Ü$5Ð*6ä"2ê%3ê*6î,6ì29ò38ô;<½^^V´r4ËX^EÒ\^@Ùi^AÜt^DÎt^Uây^B÷B>Æz'<9f>;C×6Fì9Dð:B<91>KP°NQ¤qS<94>qp¯nmÑNTôIH÷SNéMSöYTøe\ß\fÓjlðpløtj^A<82>^M^B<89>^O^C<8d>^Q^T<84>^_^A<93>^Q^A<9d>^T^H<99>^T^A£^U^G«^X^P®^Z^L²^\^Q¶^^^X<98>*'<94>6^Y¥+^W»!#¬/&²1-Á3S<86>ZW¦`]Çi¿<83>9Ú<86>^Tè<87>^Dì<94>        ô<9a>^Eê<9a>^Qù¤^Gÿ¬^Fý*
ÿ²í£^Xõ©^Tý¶^Vÿ»^\Ô<97>3î¯/þ¼#ÿÁ)ÿÆ7´<86>O<9a><89>s±<8f>m´ }Ý¡cÿÌHüÎWõÏp(<<9d>
@  

As you can see, there are two blocks. The first block of 11 lines which includes the blank line is suppose to be truncated. Some of you that may have known how to use sed will be quick to use '1,/^$/!p' script but it's incorrect as the nc output has hidden metacharacters such as '^M' which represent return. If you output the nc to a file and open it in vi, then you'll see the hidden characters. So the correct sed script should be '1,/^^m$/!p', which means it will not print output from the first line to the first blank line.

The first '^' (caret) character is the literal character whileas the second caret preceded by 'M' isn't. You'd have to manually enter the metacharacter by holding CTRL while pressing 'V' then 'M' to get the '^M' metacharacter to appear. The '$' is, of course, literal. The first caret represents the beginning of line and the dollar sign represents the end of the line, which you can see when using ':set list' command in vi.

I hope everyone who's new to this find this extremely helpful with real examples that they can test.

ChooseLife

Awesome stuff, loved paul78's solution and even more so hiddenknight821's expansion of it.

In my case,

sed -n -e '1,/^[B]^M[/B]$/!p'

had to be replaced with its equivalent of

sed -n -e '1,/^[B]\r[/B]$/!p'

to work, I'm curious to know what is the reason for ^M representation of carriage return to work for hiddenknight821 but not for me.

Another variation of the filter it to look explicitly for the GIF file - its magic number (i.e. sequence of characters at the beginning identifying the file format) can be GIF87a or GIF89a, thus

sed -n -e '/^GIF8[79]a/,$p'

will achieve the same result while explicitly filtering in only the GIF file, and filtering out anything else (e.g. a 404 error message)

ChooseLife

paul78 wrote: »

Ahh... This would have made a good question of the day

As you wish

hiddenknight821

ChooseLife wrote: »

In my case,

sed -n -e '1,/^[B]^M[/B]$/!p'

had to be replaced with its equivalent of

sed -n -e '1,/^[B]\r[/B]$/!p'

to work, I'm curious to know what is the reason for ^M representation of carriage return to work for hiddenknight821 but not for me.

Interesting to see that \r worked for you. I can attest it didn't work on Mac OS X Snow Lion. In my previous post, I did the example using Ubuntu 12.04, and I just learned \r worked on Ubuntu as well. So, I guess \r is not an universal solution, but thanks for bringing that up as I just learned something new.

I'm not sure why ^M didn't work for you, but I hope you used the CTRL-V and CTRL-M key combination rather than entering the literal characters. Maybe different distribution or sed version has something to do with it. What were you using?

By the way, I must say your GIF regex was brilliant. I wasn't sure if it has a tag I can use as I was not familiar with it.

ChooseLife

hiddenknight821 wrote: »

Interesting to see that \r worked for you. I can attest it didn't work on Mac OS X Snow Lion. In my previous post, I did the example using Ubuntu 12.04, and I just learned \r worked on Ubuntu as well. So, I guess \r is not an universal solution, but thanks for bringing that up as I just learned something new.

I'm not sure why ^M didn't work for you, but I hope you used the CTRL-V and CTRL-M key combination rather than entering the literal characters.

While I'm not particularly interested in Mac OS's idiosyncrasies, it would be good to find a definitive answer on which one is a universal solution for *nices. I used literal chars and not CTRL-V and CTRL-M, and learning that that was a specific key combination makes me favor the \r approach - there is only one way to type that without the ambiguity created by combinations of keyboards, key mappings, terminal codes, and such.

hiddenknight821 wrote: »

By the way, I must say your GIF regex was brilliant. I wasn't sure if it has a tag I can use as I was not familiar with it.

Why thank you Neither was I, but a quick google search revealed the answer