Linux Bash Question on STDERR and STDOUT

JockVSJock

So I'm going thru the O'Reilly Book bash Cookbook and I though I had a good handle on redirects, stdout and stderr.

Example of redirecting stdout:

example > foo.txt 

Example of redirecting stderr and stdout, which overrights in the process:

example2  &> foo.txt 

In the book they give an older example of sending stderr and stdout to a file, and I don't understand why it is written the way it is.

Example from the book

example3 > foo.txt 2>&1 

The 1 is a file descriptor? I'm not sure what that means in Linux? I though 1 was stdout?

jibbajabba

Its a combination of both

1 = stdout
2 = stderr

So 2>&1 means stderr is being redirected where stdout is being redirected to.

apr911

jibbajabba is correct as to what 2>&1 does but I wanted to provide an example of when you might want to redirect stderr to stdout...

Curl puts a lot of information to both stderr and stdout. As a user running curl, you wouldn't necessarily know that because your terminal display stderr and stdout together but for programming or a oneliner, you need to be aware of that because pipes sends only the stdout of the 1st program to the stdin of the 2nd

cmd1 | cmd2
is saying to take the stdout from cmd1 and send it to cmd2 as stdin.

Going back to my Curl command, if I wanted to grep for certificate name and certificate expiration, which are both in the stderr of the curl command, I cant pipe it directly since stderr is discarded by pipe. If I redirect stderr to stdin first though, I can make it push the all the output of my curl command to grep in the same manner I would see it render in my shell which would then allow me to grep for certificate info.

curl -Ivk https://somesite 2>&1 | grep certificate

JockVSJock

apr911 wrote: »

jibbajabba is correct as to what 2>&1 does but I wanted to provide an example of when you might want to redirect stderr to stdout...

Curl puts a lot of information to both stderr and stdout. As a user running curl, you wouldn't necessarily know that because your terminal display stderr and stdout together but for programming or a oneliner, you need to be aware of that because pipes sends only the stdout of the 1st program to the stdin of the 2nd

cmd1 | cmd2
is saying to take the stdout from cmd1 and send it to cmd2 as stdin.

Going back to my Curl command, if I wanted to grep for certificate name and certificate expiration, which are both in the stderr of the curl command, I cant pipe it directly since stderr is discarded by pipe. If I redirect stderr to stdin first though, I can make it push the all the output of my curl command to grep in the same manner I would see it render in my shell which would then allow me to grep for certificate info.

curl -Ivk https://somesite 2>&1 | grep certificate

Ok, I'm familiar with curl command in Linux. I will have to lab this and post back my results.

Right now the lightbulb isn't going off in my head on this.

JockVSJock

I labbed this for about an hour this afternoon, correct me if I'm wrong or still confused here.

If I use this for output and error, 2>&1, it sends the stderr to some place, I think /dev/null, correct? For example:

curl -v# www.cnn.com 2>&1

I don't see a output file such as if I use the following below.

curl -v# www.cnn.com 2&>1

1 is the output file for stdout, however when I was doing this, I changed 1 to say outputfile and all of the output went there.

apr911

The first command doesn't create a file... It just concatenates/joins stderr and stdout together.

On your screen you probably wont see any difference because your terminal window probably already displays a concatenated version of stderr and stdout by default.

The benefit of 2>&1 really only comes into play when you are trying to manipulate information. Its fairly common to take stdout from 1 program and push it to a 2nd program as stdin.

This is done via pipes (|).
cat <file> | grep <something>

Is an example of using pipes, albeit a poor one since you could just grep directly from the file.

curl -Ivk <somesite> | grep <something>

Is another example of using pipes. The "issue" with pipes is that it only takes stdout (descriptor 1) and pushes it to stdin for the next program. By default pipes ignores/discards anything in stderr because normally, you wouldn't want to take errors from the 1st program as input into the 2nd...

Curl is a good example of when you WOULD want to take the "errors" of the 1st program as input into the 2nd. This is because, when visiting an HTTPS site, curl will output the certificate information but not as part of stdout, its part of stderr.

So try this:

curl -Ivk https://gmail.com > stdoutonly
curl -Ivk https://gmail.com 2> stderronly
curl -Ivk https://gmail.com 2>&1 > stdouterr

Now look at the 3 files. Compare and contrast how they are different. (For comparing the 3 outputs, curl kind of sucks for this comparison because while there is a lot of stderr and stdout, they happen almost independently... A command that writes to stderr and stdout at the same time would be much better but I dont know a good one off hand)

For the first file, you'll notice there is no certificate information included in the file... thus
curl -Ivk https://gmail.com | egrep "subject|expire"

Would not return the results you are looking for (assuming you are looking for the cert name and expiration)

The 2nd file does have the certificate information but there isn't an easy way to pipe stderr only to the next command.
The 3rd file contains the data contained in both file 1 and file 2 as taken from "stdout." Thus:

curl -Ivk https://gmail.com 2>&1 | egrep "subject|expire"

will return:

:~> curl -Ivk https://gmail.com 2>&1 | egrep "subject|expire"
* subject: C=US; ST=California; L=Mountain View; O=Google Inc; CN=mail.google.com
* expire date: 2014-05-26 00:00:00 GMT

apr911

To give you an idea of exactly what 2>&1 does, heres another example... Lets say I have a program named "myprogram" and it writes to stderr and stdout... My program prints a statement "This is line number #" where # is replaced by the line number. For the sake of ease, lets say odd line numbers go to stderr and even lines go to stdout...

When I run "myprogram" in my terminal, Im likely to get the following:

~>myprogram
This is line number 1
This is line number 2
This is line number 3
This is line number 4
and so on...

This is because your terminal likely already concatenates stderr and stdout together.

If I run myprogram and redirect only stdout to file, my file would be:

~>myprogram > stdoutonly ; cat stdoutonly
This is line number 2
This is line number 4
and so on...

If i run myprogram and redirect only stderr to file, my file would be:

~>myprogram 2> stderronly ; cat stderronly
This is line number 1
This is line number 3
and so on...

If I run my program and join stderr to stdout before redirecting to file, my file would be:

~>myprogram 2>&1 > outall ; cat outall
This is line number 1
This is line number 2
This is line number 3
This is line number 4
and so on...


This again become really important later when you go to use pipes (|) since the behavior of pipes only takes the stdout. Lets say myprogram only prints 10 lines (1 through 10) and I wanted to find all lines containing a "1".

I cant just pipe stdout because 1 is odd and is in stderr and even if I could easily pipe stderr only (again this is not actually easy), I cant because 10 is even and is in stdout... What I can do is join stderr and stdout and then pipe it to get just the output I want:

myprogram 2>&1 | grep 1
This is line number 1
This is line number 10