Linux question of the day

W Stewart

Clevernamehere wrote: »

I'm thinking something to do with fdisk not being able to create partitions over 2TB?

close but why can't they?

W Stewart

onesaint wrote: »

# df -i
would give you node info (I just used it the other day when a server went RO for no reason!).


I've got (maybe an easy) one for you all:

Let's say I accidentally did an rm -rf on ../httpd/access_log but I still need the file.

Is it recoverable? If so, how?

Okay so I had to **** and do a little reading. I guess I won't spoil it but it's interesting none the less. I actually came across something similar to this at work. Just the opposite.

W Stewart

Nobody has answered it yet so I'm just going to say yes it's possible to recover the file as long as apache hasn't been restarted. I tried deleteing a log file on a customers server that was taking up a lot of space and even though the file was gone the disk space wasn't returned to the server. It turned out it was because apache still had the file open in memory and it needed to be restarted. What I just read a few days ago gave me an explanation as to why that happens and it's because the file is still open in the proc filesystem.

This guide can give you an idea of how it can be accomplished with lsof.

Linux.com :: Bring back deleted files with lsof

Anybody know why fdisk can't see partitions larger than 2TB?

ChooseLife

W Stewart wrote: »

This guide can give you an idea of how it can be accomplished with lsof.

Linux.com :: Bring back deleted files with lsof

Really nice trick and very useful, thanks a lot for sharing. For some time I've been wanting to ask a question on /proc fd's; yours has much usefulness to it.

ChooseLife

W Stewart wrote: »

I've got a 3TB harware RAID 1 volume. I can view the drive with

parted -l

but not with

fdisk -l

. Why is that?

Because 2Tb+ partitions require GPT, which is supported by parted, but not supported by fdisk.

onesaint

W Stewart wrote: »

Nobody has answered it yet so I'm just going to say yes it's possible to recover the file as long as apache hasn't been restarted. I tried deleteing a log file on a customers server that was taking up a lot of space and even though the file was gone the disk space wasn't returned to the server. It turned out it was because apache still had the file open in memory and it needed to be restarted. What I just read a few days ago gave me an explanation as to why that happens and it's because the file is still open in the proc filesystem.

This guide can give you an idea of how it can be accomplished with lsof.

Linux.com :: Bring back deleted files with lsof

Sorry for the late reply, works been insane recently.

That was it, W Stewart (great link too). We had a big conversation at work about trick interview questions and the lsof / recover dead files was one of them.

Really, all the ls* commands are invaluable.

W Stewart

ChooseLife wrote: »

Because 2Tb+ partitions require GPT, which is supported by parted, but not supported by fdisk.

Correct.

UnixGuy

New Question: This one is not tricky but very practical.

There's a Linux server with the filesystem "/u01", and suddenly there's a requirement to increase the size of this filesystem by, say, 1GB. So the SAN administrator maps a 1GB LUN to the server, how do you increase the size of "/u01" ?

This is the filesystem:

[root@centos ~]# df -h /u01
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/vg_data-u01          2.0G   35M  1.9G   2% /u01
[root@centos ~]# 

log32

You need to extend the Logical Volume and then match the filesystem to that size.
you will need to extend the volume group this LV is part of with vgextend.
then:
- unmount the filesystem of the LV
- for the logical volume: lvextend -L+1GB /dev/mapper/vg_data-u01
- run resize2fson the path
- remount

UnixGuy

for @log32 and everyone:

Is there a way that we can do this online (without unmounting?) Is this activity destructive (do we lose the data?) Is fsck necessary and why?

ChooseLife

UnixGuy wrote: »

Is there a way that we can do this online (without unmounting?)

Good question! I did not think online resizing was possible, so had to do some search...

UnixGuy

ChooseLife wrote: »

Good question! I did not think online resizing was possible, so had to do some search...

I don't know the answer to this myself, I did some searching and tried to do it with LVM and ext4 but It didn't work. With the ZFS filesystem in Solaris it can only be done online (there is no umounting) and it's a breeze.


Anyway, I wrote a step by step blog post about it using CentOS, LVM for an ext4 filesystem:
Linux How-To: Expand an LVM volume | Sahara Geeks

(PS: I wrote it quickly so don't mind some typos)

ChooseLife

Apparently, resize2fs and ext2online can do filesystem resizing online (vgextend/lvextend for LVM changes)

Here's one article I found on it:

How to increase the root partion using LVM without unmounting

UnixGuy

@ChooseLife: I will try it! I don't trust LVM a lot, I don't think I'll do it online in a production system though

ChooseLife

UnixGuy wrote: »

I don't think I'll do it online in a production system though

Exactly, that's why I've never done it before...

ccnxjr

I've tried re-sizing two online logical volumes, successfully and unsuccessfully

Arguably the failure occurred because I had files open on the volume that was unsuccessful .
Oddly enough I didn't realize that someone else wrote a "how to" about the exact same scenario i was trying to do.
Decrease my /home partition and increase my /root partition.
However I was only looking at generically "re-sizing partitions" without paying attention to the finer details, like, Which partitions, specifically...

This was the article i found after my mistake:

Resizing /home & /root Partitions - CentOS 6.2

I may re-attempt after my bruised ego has healed...

Apparently the risky business is decreasing a partition size.
Increasing partition sizes (by consensus) can occur while mounted and online.
I'll have to test this out while writing to a partition while increasing it's size.

W Stewart

I agree, shrinking a partition is always the most risky. I've found that when shrinking a partition offline, you can manage to undo some of the big mistakes like choosing the wrong starting sector or shrinking the partition smaller than the file system so long as you can put the partition back the way it was originally.

ChooseLife

Time to revive this thread... Let's get it up and running....

Question: What is the difference between "UP" and "RUNNING" flags of a network interface in Linux? (As shown in the output of ifconfig, netstat, etc)

onesaint

ChooseLife wrote: »

Time to revive this thread... Let's get it up and running....

Question: What is the difference between "UP" and "RUNNING" flags of a network interface in Linux? (As shown in the output of ifconfig, netstat, etc)

"Running" means the interface is available to be configured, while "up" means the interface is configured and activated. However, on my Mac, they are synonymous.

I had to verify my answer with this page. It's just one of those things you don't think about much, but can make a difference troubleshooting.

ChooseLife

onesaint wrote: »

"Running" means the interface is available to be configured

What would make an interface to be not available for configuring?

onesaint wrote: »

while "up" means the interface is configured and activated.

What is the definition of "activated" here?

onesaint

ChooseLife wrote: »

What would make an interface to be not available for configuring?

To clarify, an interface is always configurable. "Running" would be when the interface has the needed requisites to enable the interface to be put in an "UP" state, e.g. cabled and connected to a switch, but not .

ChooseLife wrote: »

What is the definition of "activated" here?

By activated, I mean the interface has been enabled with a command such as ifup.

ChooseLife

Ok, that I will take as the right answer

P.S. How about I tag you to post the next challenge?

onesaint

Ok, I think I've got one for you

Nagios warns me that a server's memory is >90%. I ssh in and see this:

# free -m
             total       used       free     shared    buffers     cached
Mem:          1999       1900         99          0         98       1315
-/+ buffers/cache:        486       1512
Swap:         4095          0       4095

From baselining I know the server should only be using about 400-500MB of ram.

Two questions; What has happened to all my memory and can I free any of it up? If so, how?

ETA:Ok, that's 3 questions.:)

UnixGuy

@onesaint:

Interesting question. I just want to know, what do you mean by "from baseline" ? is that the average memory usage that you observed over a certain period of time?

First I'd check the overall performance to see if this high usage of memory is affecting anything, so I would run "top" , and maybe "sar", and see what's the CPU's doing. So I can see if there's any particular user/process that's eating up resources.

ChooseLife

Onesaint, great question pertaining to the real world problems! Also really awesome because I learned of the new kernel tunable, really cool!

ChooseLife

UnixGuy wrote: »

So I can see if there's any particular user/process that's eating up resources.

In this case top/sar will show that there is no process taking large amount of memory. Overall system performance is good.

UnixGuy

@ChooseLife: If the performance of the system is ok, then I don't think that there is a need to take any action. I think the kernel uses the memory sometimes to make things faster. I don't know how to free up the memory though (apart from a reboot).

In my previous work place we had a monitoring solution(Microsoft SCOM) where it used to report high memory usage for some Linux servers and we ignored it.


What's the correct answer?

ChooseLife

UnixGuy wrote: »

I think the kernel uses the memory sometimes to make things faster.

Yes, this is it. "cached - 1315" is indicating that 1315Mb have been used by the kernel as a disk read cache. "Real" RAM usage is 486Mb according to the output above, so the system is actually running in accordance with the baseline, just using some RAM for caching. A really good description can be found here: Help! Linux ate my RAM!

UnixGuy wrote: »

I don't know how to free up the memory though (apart from a reboot)

Same here, I always thought this cache could not be directly purged, only indirectly by starting processes, which in turn take memory away from the cache.
But onesaint's question sent me on the search quest - and it turns out there is a tunable kernel parameter /proc/sys/vm/drop_caches that does exactly that: Drop Caches - linux-mm.org Wiki

P.S. onesaint, would that constitute a right answer?

fiftyo

ChooseLife wrote: »

In this case top/sar will show that there is no process taking large amount of memory. Overall system performance is good.

I believe this question comes down to message queues/shared memory segments which you can view via ipcs -a then there would probably some segment eating a lot of memory in the queue, which you could destroy via ipcrm -M <key value>.
Sample outputs
ubuntu:~$ ipcs

---

Shared Memory Segments

---

key shmid owner perms bytes nattch status
0xcbc384f8 3506184 user 600 64528 1

Identifying the key id in this case, we can see what process this is via
ubuntu:~$ ipcs -m -i 0xcbc384f8

Shared memory Segment shmid=0
uid=1000 gid=1000 cuid=1000 cgid=1000
mode=01600 access_perms=0600
bytes=393216 lpid=3834 cpid=3721 nattch=2
att_time=Sun Aug 4 09:36:38 2013
det_time=Sun Aug 4 09:36:38 2013
change_time=Sun Aug 4 09:36:08 2013
Then do;
ubuntu:~$ ps aux | grep 3834
user 3834 11.3 3.8 1685128 636080 ? Sl 09:36 16:08 /usr/lib/firefox/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -greomni /usr/lib/firefox/omni.ja -appomni /usr/lib/firefox/browser/omni.ja -appdir /usr/lib/firefox/browser 3721 true plugin
To locate the process, then kill it if its junk. However sometimes the processes are stuck in the queue, not showing up with PID, in that case you have to ipcrm -M <keyid>, in this case it would be 0xcbc384f8. As for example this can happen when you restart daemons, and processes for whatever reason gets stuck in the queue.
Not too sure if that was the answer you were looking for but worth a try

onesaint

UnixGuy wrote: »

@onesaint:

Interesting question. I just want to know, what do you mean by "from baseline" ? is that the average memory usage that you observed over a certain period of time?

That's exactly right. I clear out the cached ram and monitor the usage for a few days to get a baseline.