Linux script

DevilWAH

Any one good wiht linux scripting?

Basicaly I need a really simple test script (well it sounds simple)

I want a script that can be run at the command line, and can read the input from a named text file

it then takes this input and does some cleaver stuff to it so and out puts a result at the end on the screen

ie

#!/bin/bash
#
tr ' ' '
' < text.txt |sort | uniq -c | wc -l


So this counts the number of words in the file text.txt

however even with a 100mb file it only takes a few seconds to run, can any one suggest another line or some more script I could add that would mean the script took about 5min to run on a smaller text file.

I dont know scripting very well, but I am trying to create high CPU usage, from a small(ish) text file input.

Any ideas?

Aaron

Met44

Are you only interested in creating high CPU utilization?

#!/bin/sh
while : ; do
     a=1
done

If you really want to use a file and not have it go on indefinitely, try what you are doing within a loop up to some high number.

Chris:/*

From your post above I understand you want to count the words of a text file, what else do you want to do? Start with this for the word count.

wc -w text_file

If you want to do more than this (and I mis-understood what you wanted) please define what you want the script to actually do.

Now if you want only unique words from the text file:
cat text_file | tr " " "\n" | sort | uniq -c | wc -l

DevilWAH

this will be a testscript for a distributed computer network (think folding at home/boinc)

at its most basic you pass the master server a excatuable/script, and an list of inut files,

in this case the script is the excatuable and the text file is the input

so you end up with some thing like

input1.txt, input2.txt, input3.txt..... input(n).txt

you submit the job to the server which then farms out the work to clients,

so clint one recives the script + input1.txt
client 2 gets script +input2.txt and so on.

so all I am looking for is to build a simple script that I can use to test, I actuly hadn't thought about a DO loop... that seems a simple thing to do

like I say I want to be able produce high CPU load, so I can show how incressign the number of client machines improves the speed of the total task.

At our compan we do lots of mathmatical modeling, so in the end the excatuable would be the models that have been designed, and the input files are different start points/seeds. For example the model may by how a diease spreads around the country taking air currents and transport in to account, the input file would populate values like start of outbreak, direction of wind speed, time of outbreak, etc etc.

so you run the same model many 10,000's of times each time with a different start point or different variables. becasue each job independent of the others its easy to breack them up and spread them over mutiply PC's.

This script is jsut a basic model simulation so I can play around and see what happens. The fact I am using a text file as the input is, it is just easy to creat a script that can read the file, with out having to worry to much about the programing/scripting side.

DevilWAH

Chris:/* wrote: »

I do not recommend a do or while loop, you should use a for loop. Something like this.

echo "Enter Target File:"
echo "> \c"
read $target
for i in $target
do
cat i | tr " " "\n" | sort | uniq -c | wc -l
done

how would I script a for loop that jsut does

do while x = 1 to 50;
..
..
..
x=x+1
done?

all so what value will for I in $target give me ?

I am just trying to work out exactly what this is saying. in I had a text file with 50 words in it what would happen ??

Chris:/*

The for loop is looking for input and so you can load a text file into the script or multipule text files if you adjust the script slightly. The script then parse it with the command above. Now if you have a set number of words in the file you want to parse then this would be the syntax with a while loop

$count=1
while [$count -lt 50]
do

done

It seems much of the specific information you are looking for can be found here:
http://www.injunea.demon.co.uk/pages/page205.htm for simple scripting.

& here for command look-ups:
http://www.freebsd.org/cgi/man.cgi

I do hope this info is helping.

MentholMoose

You can generate CPU load by doing a math calculation at high precision with bc (should be included with any Linux distribution). A square root calculated to 9999 decimal places can take several seconds. For example, here's 1 through 10:

$ seq 1 10 > num.txt
$ time for i in `cat num.txt`;do echo "scale=9999;sqrt($i)"|bc>/dev/null;done

real    0m19.063s
user    0m18.940s
sys    0m0.026s

It maxed out a CPU core for about 19 seconds (this is a 2.1GHz Intel Core2 Duo). It is trivially easy to generate the input files using seq, just give it a range. Adjust the decimal places as necessary (e.g. with scale=99999, it takes much longer).

tiersten

Run it through a hash function like MD5 a few hundred times and the CPU usage will scale then with the size of the file.

DevilWAH

HI guys,

Yes Chris that has been very helpful running lops has given some nice results, I think combined with some hash and maths functions I will have a nice test to demonstrate this working to management.

DevilWAH

ok one last thing

why am i getting this error ?? (its ok I see now it needed a space) but still not sure how to input the numvarible in the file name. he he nope never mind managed to fix that to

./Generator: line 4: [: missing `]'

here is the script

#!/bin/bash

filee=1
while [ $filee -lt 50]
do

count=1
while [ $count -lt 50]
do

cat /dev/urandom|od -N2 -An -i >> nums.num


count=`expr $count + 1 `
done

filee=`expr $filee + 1`
done

all i want is to run two loops inside each other to produce 50 files each with 50 random numbers.

and also how do i number the out put file nums.num to nums($filee).num, ie nums1.num.. nums2.num???

thank you so much for all the help on this. I really should get a book for all this

Chris:/*

I again apologize for any mistakes I am still only an associate level Linux scripter but I will help as much as possible.

To name the output file try this:
nums.{$file}.{$count}.num

If that does not work add a $ to the front side of the curly brace for each variable.

Count is a reserved command in Linux and the interpreter believes you are trying to use the count command rename the variable this should fix your problem. An alternative solution would be to declare the variable with the $ in front of it. This is a good habit to be in anyway, it lets future script writers know what you were trying to do.

Cheers.

Chris:/*

Since I believe you are going to start parsing data at some point you should look for some of the Oreily Mastering Regular Expression Books.

Good Linux Scripting books:
Amazon.com: Wicked Cool Shell Scripts (0689145701272): Dave Taylor: Books
Amazon.com: Classic Shell Scripting (9780596005955): Arnold Robbins, Nelson H.F. Beebe: Books
Amazon.com: Learning the bash Shell: Unix Shell Programming (In a Nutshell (O'Reilly)) (9780596009656): Cameron Newham: Books

DevilWAH

cheers for all the help. yer might get a book. to be honest thought i will only need to do a little bit. its another thing i would love to get in to but trouble is time! studying for cisco at the moment so this is a side project.

but once again cheers for all the help. been great

I now have a nice script that generate 20 files each with 50 random numbers

then each of these are run through a second script that generates the square root to 9999 places outputs this to a file and attaches at the end, the hash of the original input file

so i think I have managed to get all your suggestions in there some where

Chris:/*

Ha great to hear. As a tip for those how read your script in the future make sure to put your name at the top of it. Traditionally you include your name, date, script purpose and contact info in it so if someone has a question in the future they can hit you up.

Again great to hear you got what you wanted out of it, and best of luck with your Cisco material. That is a 2011 goal for me.

DevilWAH

ok slightly different track, but I thought I would post it here any way.

for testing only, how do i set up iptables in linux,

all I want is to say, allow any thing to and from subnet 192.168.1.0 255.255.255.0

I can tighten this up later but I want it completly open as I am trying out some stuff I know little about ports and protocols that will be used, and currently it wont connect, so I want to open it up first so i kow this is not an issue, and then lock down on the live system once it is working.

Forsaken_GA

Well, the first thing to check is whether or not you actually have any rules in place

iptables -L -n

If that shows something like this:

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Then you don't have any rules in place, and your connectivity problem is more likely the result of a missing route.

If you just want a completely open system, iptables -F will clear all firewall rules

If you just want to add some rules without blanking your current set -

iptables -I OUTPUT --source 192.168.1.0/24 -j ACCEPT
iptables -I INPUT --destination 192.168.1.0/24 -j ACCEPT

iptables -L -n should look something like this (assuming no other rules)

Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 192.168.1.0/24

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 192.168.1.0/24 0.0.0.0/0

That will put them at the top of the INPUT and OUTPUT chains (if your chains are named differently, than use those chains instead) to ensure that they're processed first and no other rules will get in their way (-A instead of -I appends them to the end of the chain, -D removes the rule from the chain)

DevilWAH

err one thing how do you remove a chain you have acudently added....

cheers thats a great explnation of how it all works!! cheers

Forsaken_GA

DevilWAH wrote: »

err one thing how do you remove a chain you have acudently added....

cheers thats a great explnation of how it all works!! cheers

Issue the same command, just replace the -I or -A with -D