Food, Wine, Football....and PERL?

After taking a long (and much needed) week off......wait, who am I kidding? Even over the course of being gone for an entire week for Thanksgiving, between family, football, great food, and even better wine, I still found time to dig into PERL. In hopes of reaching my end-of-the-semester goal of creating a final script that can parse through Stephanie's (extremely) large log/parse files, and finally running a full-on analysis of our systems, I downloaded Putty on my home PC and got straight to work after a (massive) Thanksgiving dinner. Did I mention we had one Thursday AND Friday? Jealous, I know.

In taking a step back from my previous statement, the script is actually complete. However, it is most definitely a "brute force" script, as we say in programming. In English, I took a much longer route with much more coding in order to complete the script. But, as with all things, there exists a much more efficient way for the code to work through itself. The current script, as it stands, opens and closes each of the 3 files for every iteration (aka run through) of the program, thereby extending the run-time of the program exponentially. Though with my test files it is approximately only a 10 minute wait on the run-time, if I were to attempt to run any of Stephanie's logs it certainly would take a few hours, at least. In other words, there is still some progress to be made. My next step as of now it to modify the script to, instead, open each of the files once and simply iterate through the lines repeatedly to find matching time stamps, rather than opening and closing each file every time. I'll be crying tears of thanks, metaphorically, when this script is all said and done.

But, on that note, what are you thankful for? I had plenty of things to be for over the past week: Family, friends, LOTS of football, food, wine, and a somewhat of a break in general from life. But, it didn't occur to me until the end of it that having the opportunity to learn PERL was also something I should be thankful for. I've mentioned it before, but it has been a welcoming opportunity to learn a new language that I thought I might be otherwise incapable of without CREU. So, in closing, thank you CREU and happy holidays to everyone - Cheers!

Cannot touch `cupcake': No such file or directory

Yes, calibration script, no cupcakes for you. Instead, victory cupcakes for me because I made it work! Nevermore did I want to dance, cheer and celebrate seeing that line. It meant my wonderful script from last week (touch the log, and test the directory path headache) was finally working. I can actually start looking at a new problem within some other file. Any file. Maybe calib again, so long as I don’t have to go near that block of code for a while. Why? Coding and patience are fragile things.

If that wonderful ssh connected terminal, happens to have a poor internet connection, well, you’re out of luck to debug the script as you test without serious lag. Frustrating but there’s workaround hacks like write the code offline, connect, run the test, look at the bugs, disconnect, repeat. Even better, right as the solution dangles itself to you to solve all problems once and for all, said internet connection DIES.

It’s like playing a game when Windows wants to restart for an update. Windows keeps minimizing your game right as you’re battling the big boss bad guy to bug you “Oh hey do you want to restart now or later for these updates to take effect?” No amount of patience will save you from the rage quit. Write down the genius solution. Take a day off. Pray to the campus dorm internet idols for a better connection & ping rate, then go at that sucker like there is no tomorrow. Code in anger. It gets the job done. Especially with enough lies, like if you finally get it working you never have to touch that code again.

Error between the Keyboard & Brain

My main task this week was to expand the calibration script with a simple task I’d done
before with a slight twist. First create a file using touch to hold a log in the directory for
a certain test, set up an if case because the user might want the log in a separate directory
and ask the user which they want to use. We’d rerun a few tests and then finally start
looking at some data. I had a few other tasks on top of this, but obviously I did not get
to those because programming can be deceptive. Once you feel grounded in another
language it can lead to a blind trust in the code you write. Hubris edging you on the
whole time, Pssh we’ve been using unix for two years now. We’ve used touch a thousand
times on the command line. So easy.

Instantly then when things go wrong we must distrust the new language! Dude no way
did I mess up my unix commands. It’s all perl’s fault! I’ve spend hours on the script, read
online tutorials finding no one else had problems doing this, reread the man page on
touch. It’s all stupid perl!

Except when perl isn’t at fault. And I realized this hours later. When my cutesy
workaround hack began turning into a contender for an obfuscated perl contest entry.
After shooting off an email to my professor that made me feel crummy. I obviously
sucked at coding for not solving this problem on my own. It’s so simple. Then the
response came and she asked if maybe the directory path wasn’t correct in my test inputs.
That reply almost made me ready to hide under my desk because that hinted at hours of
backing up mentally and rewriting the whole block of code to my original solution.

I would have loved to let myself get mad at that jerk of a user. All that time wasted! Bad
input was at fault! My code isn’t suppose to catch the user’s errors! But I was that bad
test user and I let those errors make me distrust the 80% working code. A little tweak to
get it 100% working was all it would take.

But really this stumble reminded me I’ve only been coding seriously for barely two
years. I’m still very much a toddler in terms of coding experience. This setback won’t
discourage me. Until this giant mess, I didn’t notice the huge improvements from my
first Programming I class to now. The important thing I need to keep reminding myself
is I need to keep writing consistently correct code over the long haul. That’s all that
is required. Not rushing myself because I should be handling more and writing harder
scripts. The code’s going to show that rushing by not working.

PERL me once, shame on you. PERL me twice, shame on me. PERL me thrice.....now that's nice!

English is certainly not the most spoken language in the world, being heavily overshadowed by Mandarin Chinese. But, it comes in at fair 2nd, spanning almost 400 million people across the globe. PERL on the other hand, probably comes in at a rigorously debated last place, or pretty darn close to it. One thing we can all agree on though is that a programming language whose intention was to create a seamless, convenient programming environment, still requires plenty of translation from one programmer to the next. In other words, the majority of my work the past 2 weeks centered around adapting old scripts in PERL to our newly created scripts via Stephanie's scripts and documenting accordingly.....that, and spending endless hours conjuring the perfect Halloween costume, which inevitably ended in me showcasing my inner nerd.


As the 2nd commandment of Stephanie's Undergraduate Research states, thou shalt have nothing to show for it more often than not. Fortunately, for my piece in the project, coding and documentation updates would be the upside to this, being immediately (for the most part) available and indicative of progress. But, we don't clock in/out while we're coding (We pretty much remain clocked in), so the time spent analyzing and coding certainly isn't accurately reflected enough in a small script that taunted you for hours on end. This was expected going in, for we are not naive knaves. On a positive note, however, the past 2 weeks were much more productive than the first week of learning PERL, given that I had something to build off of. The end reward of it all was a final tri-fecta of scripts that can read an input file of data taken from our 2 host computers, discard the trans fats, and give us what we want, the way we want it. Watch out Burger King!


Moving beyond our current work, with this being the mid-point in the semester, it seems fitting to take a look at how far we have progressed in the past 2 months. In reviewing Ben's (Last year's SSU CREU recipient) information from last year, being able to continue from where he left off seemed like quite the daunting task. Vaguely recalling him coming into our class to speak with Dr. Rivoire about scripts that had been running for 4 days straight did not perpetuate feelings of confidence, to say the least. To put it in perspective though, reading the articles about CPU/GPU analysis for our first week of CREU work didn't necessarily accomplish that either. And, to think that we are now doing exactly what I was not sure how we would even begin to tackle in the first place is evidence enough of progress. It has been quite the experience proving to myself that I could take on tasks that were totally benign to my previous Linux experience, as well as learn a completely new language without it being an available course here in the CS department - Quite the leap from sitting in the lab letting Lady Gaga pwn our computers. I've taken the logical approaches and confidence gained from the past 2 months and have applied them to my other classes/daily life. Heck, just by the nature of me being on campus more to work on CREU has subsequently ushered me into spending more time on other school work as well. And with that, I must bid you all adieu, but not without leaving the throngs of onlookers a taste of our PERL magnificence. Au revoir!



Our CPU (1 of 3 items we are currently analyzing) parsing script and output:

#!/usr/bin/perl

package test;

#Opens cpu data profile, spits out an error if it does not open.
open(FILE, "
#Reads in each line of the file one at a time.
foreach $line ()
{
#Splits the line into variables that we can use or trash.
($time, $ampm, $cputype, $c, $d, $e, $f, $g, $usage) = split(' ',$line);
($hr, $min, $sec) = split (":", $time);

#Checks to see if it is 12 in the morning, subtracts 12 if so (To reflect 24 hr time stamp)
if ($ampm eq "AM" && $hr eq "12")
{
$hr = $hr - 12;
}

#Checks to see if it is after 12PM, adds 12 if so (To reflect 24 hr time stamp)
if ($ampm eq "PM" && $hr ne "12")
{
$hr = $hr + 12;
}

#Checks to see if it is the first CPU being read, in order to print out 1 time stamp, rather than multiple of the same time stamp.
if ($cputype eq "0")
{
print $hr;
print ":";
print $min;
print ":";
print $sec;
print ",";
#Usage is the 100% minus the idle.
printf "%.2f", (100 - $usage);
}

elsif ($cputype eq "1")
{
print ",";
printf "%.2f", (100 - $usage);
}

elsif ($cputype eq "2")
{
print ",";
printf "%.2f", (100 - $usage);
}
elsif ($cputype eq "3")
{
print ",";
printf "%.2f", (100 - $usage);
print "\n";
}
else
{
}
}
}

Original File:
06:47:02 PM CPU %user %nice %system %iowait %steal %idle
06:47:03 PM all 0.50 0.00 1.01 0.00 0.00 98.49
06:47:03 PM 0 0.00 0.00 1.98 0.00 0.00 98.02
06:47:03 PM 1 0.00 0.00 1.00 0.00 0.00 99.00
06:47:03 PM 2 0.99 0.00 0.99 0.00 0.00 98.02
06:47:03 PM 3 1.04 0.00 0.00 0.00 0.00 98.96

06:47:03 PM CPU %user %nice %system %iowait %steal %idle
06:47:04 PM all 0.00 0.00 0.56 0.00 0.00 99.44
06:47:04 PM 0 0.00 0.00 1.00 0.00 0.00 99.00
06:47:04 PM 1 0.00 0.00 0.98 0.00 0.00 99.02
06:47:04 PM 2 0.00 0.00 0.00 0.00 0.00 100.00
06:47:04 PM 3 0.00 0.00 0.00 0.00 0.00 100.00


New Output File (5 out of 1000's of lines)

18:47:03,1.98,1.00,1.98,1.04
18:47:04,1.00,0.98,0.00,0.00
18:47:05,1.00,1.00,0.00,0.00
18:47:06,1.00,0.99,0.00,0.00
18:47:07,1.00,1.00,0.00,0.00

** This will be saved as a .csv file which can be read into Microsoft Excel and made very aesthetically pleasing!