10.1R1.8 Release Issue

llllvllll · July 2013

I am new to Juniper can someone help me with this. I was trying to commit sync but got an Error :- Does not have enough space. So i googled the solution but i dont know much about Juniper's directories. If i follow the Procedure given below will it solve the issue? Can someone explain me the solution ?

(1) CD to /var/log directory and verify "pfed_event_trace.log" log increasing.

root@:BK:1% cd /var/log

root@:BK:1% ls -al | grep pfe
-rw-rw---- 1 root wheel 7365321 Mar 17 17:15 pfed_event_trace.log
-rw-rw---- 1 root wheel 0 Mar 17 16:34 pfed_trace.log

root@:BK:1% ls -al | grep pfe
-rw-rw---- 1 root wheel 7366969 Mar 17 17:15 pfed_event_trace.log
-rw-rw---- 1 root wheel 0 Mar 17 16:34 pfed_trace.log

(2) Move "pfed_event_trace.log" to "pfed_event_trace_old.log"

root@:BK:1% mv pfed_event_trace.log pfed_event_trace_old.log

root@:BK:1% ls -al | grep pfe
-rw-rw---- 1 root wheel 71 Mar 17 17:15 pfed_event_trace_old.log
-rw-rw---- 1 root wheel 0 Mar 17 16:34 pfed_trace.log

(3) Link "pfed_event_trace.log" to /dev/null

root@:BK:1% ln -s /dev/null pfed_event_trace.log

root@:BK:1% ls -al | grep pfe
lrwxr-xr-x 1 root wheel 9 Mar 19 20:13 pfed_event_trace.log -> /dev/null
-rw-rw---- 1 root wheel 71 Mar 17 17:15 pfed_event_trace_old.log
-rw-rw---- 1 root wheel 0 Mar 17 16:34 pfed_trace.log

(4) Identify PFED running process and record PID from your output (Process Identifier = 845 in this example)

root@:BK:1% ps -aux | grep pfed
root 845 0.0 0.2 10904 4080 ?? S 11:53PM 0:04.85 /usr/sbin/pfed -N
root 1070 0.0 0.0 2084 728 u0 R+ 3:49AM 0:00.00 grep pfed

(5) Kill the PFED process identified in previous step for your system output

root@:BK:1% kill 845 <USE the PID output from your setup>

(6) Verify that new PFED process was started - note that PFED PID should be new PID number.

root@:BK:1% ps -aux | grep pfed
root 1071 0.0 0.2 10904 4228 ?? S 3:49AM 0:02.01 /usr/sbin/pfed -N
root 1175 0.0 0.0 2084 656 u0 R+ 5:40AM 0:00.00 grep pfed

(7) Remove the "pfed_event_trace_old.log" created in previous step.

root@:BK:1% rm pfed_event_trace_old.log

root@:BK:1% ls -al | grep pfe
lrwxr-xr-x 1 root wheel 9 Mar 19 20:13 pfed_event_trace.log -> /dev/null
-rw-rw---- 1 root wheel 0 Mar 17 16:34 pfed_trace.log

P.S :- It's a live network if i follow the solution given above will it affect anything? Is it safe ?

Thanks in advance.

fiftyo · July 2013

Hi,

Wasn't too sure about the pfed process coming up instantly after killing the process, so did a minor lab;
root@:RE:0% ps aux | grep pfed | grep -v grep
root 14833 0.0 0.6 11224 6768 ?? S 11:31AM 0:00.18 /usr/sbin/pfed -N
root@:RE:0% kill 14833
root@:RE:0% ps aux | grep pfed | grep -v grep
root 14839 4.8 0.6 11224 6768 ?? S 11:32AM 0:00.18 /usr/sbin/pfed -N
root@:RE:0%

Seemingly the process came up again pretty much instantly, as can be seen from the log messages;
root> show log messages | last 5
Oct 10 11:32:11 init: pfe (PID 14833) exited with status=0 Normal Exit
Oct 10 11:32:11 init: pfe (PID 14839) started
Oct 10 11:32:11 pfed: LIBJSNMP_SA_IPC_REG_ROWS: ns_subagent_register_mibs: registering 2 rows
Oct 10 11:32:11 pfed: PFED unable to listen for stats

As this file is a log file, it's safe to remove, however, it could be a good idea to scp it over to your management machine.

llllvllll · July 2013

So what exactly is happening ? We deleted pfed_event_trace.log file ? what does pfed_event_trace.log -> /dev/null means ?

fiftyo · July 2013

You make a symlink to /dev/null, basically blackholing the output of the file, preventing it from growing in filesize. As for example from my workstation;
[archy links]$ ln -s /dev/null 1
[archy links]$ echo 1 > 1
[archy links]$ cat 1
[archy links]$
[archy links]$ for i in {1..9999}; do echo $i >> 1; done #redirecting 9999 lines to the file
[archy links]$ cat 1
[archy links]$
But as you can see the file is still empty because the system tossed all the output to the bit bucket. You can read some more at wikipedia about /dev/null and symlinks.
So to answer your question, you delete the original file, then you prevent the new file from growing by discarding all its data.

llllvllll · July 2013

Thank You very much. Can you tell me where can i find more about juniper directories(in details) and their function like mounting etc ? if you could provide me a link that would be wonderful.

fiftyo · July 2013

Sorry for the late reply.

Well, junos is based on freebsd, so you are probably better off learning linux/unix directory tree, as it will transfer over. There are however, some good KB articles. As for example https://www.juniper.net/techpubs/software/junos/junos93/swconfig-cli/viewing-files-and-directories-on-a-JUNOS-device.html goes through some directories. This one goes through processes Baseline Operations Guide
Anyway, I'd say you're better off reading the JNCIA-junos pdf's along with some day one guides(all free on juniper's website). And depending on your background the juniper enterprise switching/routing books are good references.

networker050184 · July 2013

Why not upgrade to a code with this bug fixed?

ccnxjr · July 2013

there are no magic commands that will solve the problem, you can't issue "request fix-my-storage-issue-issue" and "commit sync" to get it working.
*facepalm*
if this is a production system and your tinkering with it without knowing how it works then you should be fired (or at the very least be authorized to call JCare)
sorry if this sounds harsh, but , yeah
if you are a newb (like some of us) then maybe you have a non-production device you should be able to test on

firstly there are a lot of details to be fleshed out before anyone can validate whether you have the "right" solution to your problem, but here goes anyway
a "show system storage" should give you an overview of the mount points, available and used space.
you've googled and presented very specific solutions for problem set without defining what your specific goals are

if your space error is because your issuing a "commit synchronize" i'm assuming that your attempting to synchronize a new configuration across a group of switches that are part of virtual chassis

if that is so, then configurations are stored in "/config " , you should see used/available stats in the output of "show system storage"
if you are working on a device that is part of a virtual chassis setup then you should check each of the other devices that are part of the virtual chassis for available space

if any of those assumptions are true then don't touch anything that has to do with PFED unless you absolutely have to!

additionally if this is a EX-Series switch, you should really consider upgrading the OS to Junos 11.4R5.7 (or Junos 12.2R2.5 if it's an EX4550)
review the JTac recommended software versions for your specific switch, IIRC 10.1Rx.x is at end of life and no longer supported

llllvllll · July 2013

Thanks everyone for your feedback

@ccnxjr :- Well i used to handle cisco devices for the company now they want me to handle juniper aswell. So i am trying to learn it while working.

@fiftyo :- Thankyou for the links.

@networker :- I am still learning juniper, i think i should wait a little to understand how junos OS works before implementing anything.

networker050184 · July 2013

Learning would probably be a lot easier on code without bugs like this. Not to mention much more stable for your company which is the real important thing here.

10.1R1.8 Release Issue

Comments