Linux question of the day

W Stewart

Forgot to mention. crash is due to a memory leak so I'm not too sure that increasing swap space is a solution. I believe cache memory also frees itself when an application needs it. In this particular situation, the server is a dedicated server and we're mostly hands off when it comes to personal data, so what can we do so that we don't have to run across the street and reboot the box every time it runs out of memory? I'll give you a hint, the solution is already built into the kernel.

W Stewart

Okay another hint. Lets say I just wanted the system to reboot itself when it ran out of memory rather than hanging so that I wouldn't have to run across the street to reboot it myself. What's the best way to accomplish this?

onesaint

Forgot to reply! You're talking about the OOM killer I suspect. Had to look this one up and found a great article on this from Oracle here.

Trashman

LinuxInAlaska wrote: »

I agree with Marco71 on why it is a bad idea and what can go wrong. I would take a different route to perform the delete and just do

rm -rf TrashCan/*

If you fat finger the name TrashCan, it would simply come back with a "No such file or directory" error.

Don't delete my TrashCan

W Stewart

onesaint wrote: »

Forgot to reply! You're talking about the OOM killer I suspect. Had to look this one up and found a great article on this from Oracle here.

That's it. I did have to look it up as well but I figured there had to be a sysctl setting to alter the way the system behaved when it ran out of memory but I was already aware of being able to set the reboot time on a kernel panic. I went on a week long vacation but I still check the shift reports and it doesn't look like anybody has ad to manually reboot the box yet.

MentholMoose

You're logged in as root and cannot delete or modify a file even though you seem to have the proper permissions. Why is this happening?

[root@linux ~]# ls -lZ asdf.txt
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 asdf.txt
[root@linux ~]# getfacl asdf.txt
# file: asdf.txt
# owner: root
# group: root
user::rw-
group::r--
other::r--

[root@linux ~]# rm -fv asdf.txt
rm: cannot remove `asdf.txt': Operation not permitted
[root@linux ~]# echo asdf > asdf.txt
bash: asdf.txt: Permission denied

Expect

chattr -/+ i ?

W Stewart

^ gonna have to go with Expect. Here's a more basic version of the same question. You're logged in as a normal user. You can read and modify a file but you can't delete it. Why is that?

Expect

Sticky bit ?

W Stewart

^^^^Yup

onesaint

all of the special mode stuff should show up in the mode attributes when running ls. I'm thinking this directory perms or an SELinux type error.

W Stewart

Which question is that an answer to?


Edit: you would need to run lsattr to see if the +i flag was set with the chattr command.

MentholMoose

Expect wrote: »

chattr -/+ i ?

Yes, that was fast.

UnixGuy

Interesting, so how often do you see chattr used in production? How useful is it? Have you seen in production? what was it used for?

Expect

UnixGuy wrote: »

Interesting, so how often do you see chattr used in production? How useful is it? Have you seen in production? what was it used for?

Everytime I write an upgrader script to a customer, I always keep old configuration files for a revert option, which means I move them to another directory and just chattr the files so accidental deletion won't happen. that's how I use chattr.

MentholMoose

What command will let you view the current iptables rules? (generally distribution-agnostic)
What is the default command-line "helper" application to create iptables rules on recent Fedora versions (and likely RHEL 7)? On recent Ubuntu versions? BONUS: Write an example command for both to open port 443/tcp.

W Stewart

UnixGuy wrote: »

Interesting, so how often do you see chattr used in production? How useful is it? Have you seen in production? what was it used for?

I've only seen it used at my previous place of employment to make sure customers didn't start arbitrarily messing with files. People seemed to treat it like more of a hacky work around and not the proper way of doing things though so it was only used so often. It's helpful when your offices dhcp server keeps overwriting your resolv.conf

W Stewart

MentholMoose wrote: »

What command will let you view the current iptables rules? (generally distribution-agnostic)
What is the default command-line "helper" application to create iptables rules on recent Fedora versions (and likely RHEL 7)? On recent Ubuntu versions? BONUS: Write an example command for both to open port 443/tcp.

1. iptables-save
2. system-config-firewall{,-tui} for redhat. I don't really use ubuntu so I'm not sure what the interface is called and I'd rather not **** and look it up.
3.iptables -A INPUT -m state --state NEW,ESTABLISHED -p tcp --dport 443 -j ACCEPT
It should be the same for both but redhat may add an additional RH-Firewall-INPUT-1 chain or something like that.



Edit: Here's a question. Who can tell me what I did with system-config-firewall{,-tui}?

Expect

it's a terminal user interface for managing iptables, basically a simple "gui" from the CLI, if that was your question(?)

MentholMoose

W Stewart wrote: »

1. iptables-save

Yes, that's exactly what I had in mind. I'd also accept "iptables --list" since the question was kind of vague.

2. I'm looking for the new CLI tool in recent Fedora versions which RHEL 7 will (likely) also use. system-config-firewall-tui does not work anymore (by default).

BONUS. I'm looking for the command using the tools from 2.

The Technomancer

W Stewart wrote: »

That's it. I did have to look it up as well but I figured there had to be a sysctl setting to alter the way the system behaved when it ran out of memory but I was already aware of being able to set the reboot time on a kernel panic. I went on a week long vacation but I still check the shift reports and it doesn't look like anybody has ad to manually reboot the box yet.

A more elegant way of managing this, especially when the memory leak is a non-resolvable issue without a new software release, is to monitor the RAM usage by the application in question (either with a cronjob or an external monitoring source like Nagios) and throw a SIGHUP at the process if it crosses that threshold. Zero downtime, no 5 minute+ long reboot, etc.

Example script for that to be run as a cronjob checking every minute...which assumes that the process mentioned here is the only one that would be causing the system to go OOM, and for example purposes, is the lone java app on a server:

#!/bin/bash

# This variable is the amount of free memory that you want to trigger the HUP at
threshold=50000

# This variable is the name of your process(es) you want to HUP
process="java"

# this variable is the free memory on the system as determined by vmstat
memfree=$(vmstat -s | grep "free memory" | awk {'print $1'}) 

# Logic loop

if [[ $memfree -lte $threshold ]]; then
  sudo pkill -HUP $process  #leave off sudo if running out of root's crontab
fi

Run that every minute out of your system user's (or roots, leave off sudo) crontab. Add a notification to the if loop if you want to track how often this occurs.

onesaint

W Stewart wrote: »

Which question is that an answer to?


Edit: you would need to run lsattr to see if the +i flag was set with the chattr command.

i was referring to the sticky bit, but, my answer was off. Good call to you and expect on the chattr +i.

W Stewart

Expect wrote: »

it's a terminal user interface for managing iptables, basically a simple "gui" from the CLI, if that was your question(?)

Sorry probably not clear enough. Explain what this does and why.

ls /usr/bin/system-config-firewall{,-tui}

W Stewart

The Technomancer wrote: »

A more elegant way of managing this, especially when the memory leak is a non-resolvable issue without a new software release, is to monitor the RAM usage by the application in question (either with a cronjob or an external monitoring source like Nagios) and throw a SIGHUP at the process if it crosses that threshold. Zero downtime, no 5 minute+ long reboot, etc.

Example script for that to be run as a cronjob checking every minute...which assumes that the process mentioned here is the only one that would be causing the system to go OOM, and for example purposes, is the lone java app on a server:

#!/bin/bash

# This variable is the amount of free memory that you want to trigger the HUP at
threshold=50000

# This variable is the name of your process(es) you want to HUP
process="java"

# this variable is the free memory on the system as determined by vmstat
memfree=$(vmstat -s | grep "free memory" | awk {'print $1'}) 

# Logic loop

if [[ $memfree -lte $threshold ]]; then
  sudo pkill -HUP $process  #leave off sudo if running out of root's crontab
fi

Run that every minute out of your system user's (or roots, leave off sudo) crontab. Add a notification to the if loop if you want to track how often this occurs.

I was considering scripting something but it was a dedicated server and some of my coworkers would say that just changing the sysctl settings would be overstepping my boundaries but I figured it may as well have been a setting that we included by default before handing the server over to the customers. I would also hope that the reboots would eventually prompt the customer to do something about the memory leaks as well but our hands are really tied as far as what we can and can't touch on a customer's server.

The Technomancer

Yeah, if it's customer colocated kit, you don't do anything they aren't filing a ticket and paying a remote hands and eyes fee for, and you're only doing what they tell you to prevent liability.

W Stewart

^ You seem to know your stuff based on your other post so I'm sure you've got to have an answer for this one.

ls /usr/bin/system-config-firewall{,-tui}

What output do I get and why?

The Technomancer

You'll get the same output as if you ran:

ls /usr/bin/system-config-firewall /usr/bin/system-config-firewall-tui

Curly braces indicate a shell glob, and when a comma is used, it represents multiple terms to be iterated over. Since the glob started with a comma, that meant the first term was blank, and -tui is tacked onto the end (as that was the location of the glob) of the term during the second iteration.

W Stewart

Finally somebody got it. The technical term is brace expansion. A very useful feature of bash that I like to use often at work.

https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html

The Technomancer

Here's one:

What is the ctime of a file, and how do you change it with one command?

W Stewart

I'm pretty sure the touch command will change it but I'm not to sure exactly what it is. I believe it was the last time the file was changed but I may have that confused with mtime.