Fun with cli file transformation

I used the Wake County property tax page to get a list of all of the homes in our neighborhood. I pasted that into a spreadsheet. The Proper and Proper last columns are actually cells using the =PROPER() function. It takes the given cell and converts it from all upper case to proper case. The resulting list looks like this:

Address Street Last name Owners Proper Proper last E-mail
123 SCRABBLES DR JONES FRED Fred Jones neighbor1@email.com
456 SCRABBLES DR BONES WILMA Wilma Bones neighbor2@eahoo.com

I probably could’ve done this whole thing without using the spreadsheet or the =PROPER() function but it was easier considering the original data came from a table. I then exported that as a CSV file.

Address,Street,Last name,Owners,Proper,Proper last,E-mail,
123,SCRABBLES DR,JONES,FRED,Fred,Jones,neighbor1@email.com,
456,SCRABBLES DR,BONES,WILMA,Wilma,Bones,neighbor2@eahoo.com,

What I’m trying to do is get a list with the following format:

"First Lastname" <emailaddress@domain.com>, "First Lastname" <email2@domain.com>

I did it with the following command line. The trickiest was getting awk to print out the double quotes uses “\42”

cat neighbors.csv | grep -v ^Address | awk -F , '{print "\42"$5" "$6"\42 <"$7">,"}' | grep -v "<>" | tr '\n' ' ' > full-list.txt

I then copy and pasted the resulting file contents into the “Direct add members” feature. The one tidbit that I had to do was only copy and paste 10 addresses at once. I probably could’ve updated the above script to do that for me. But by this point I got 90% of what I needed so I just split the rest by hand.

Remote connection to Windows from Nexus 4

I setup Windows 7 running in a VirtualBox guest on my CentOS 7 Linux server (mostly for doing taxes and having a Windows test machine). I’ve configured it to start headless, that is without a monitor on the server. My main means of connection is using rdesktop from my Fedora Linux desktops.

Tonight, I decided to see how smart my smartphone really is. Could I connect remotely to a Windows 7 box using a 4.7″ Nexus 4 Android phone? The answer? YES! Using RD Client I connected to the Windows machine and was able to interact with it. Is this a usable solution on such a small screen? No way. But if you need to access something you might be able to get around well enough to get what you want.

Connecting to Windows 7 using RD client on a Nexus 4 phone

If this was going to be something you did regularly, I’d suggest at least a 7″ tablet, probably a good 10″ one would be best.

Any way, why did I do this? Because I can. Isn’t technology fun?

aliases for VirtualBox’ VBoxManage commands

VBoxManage allows you to interact with VirtualBox using the CLI. You can do just about anything with the command, from starting/stopping VMs. Getting detailed information, modifying or even handling the disks and other storage media. To see all you can do with it checkout this document:
VBoxManage

I typically deal with 3 commands, starting/stopping vms and seeing the list of vms.

VBoxManage list vms|runningvms|...
VBoxManage startvm vmname --type headless
VBoxManage controlvm vmname savestate|poweroff

To make it easier I created some aliases for the above commands in my .bashrc as bash functions:

  • showvms
  • startvm
  • stopvm

Here are the functions:

startvm ()
{
    # start a VM as headless, no need for a GUI
    __davms "startvm" "--type headless"
}

stopvm ()
{
    # allow stopping a VM by saving its state or simply turning it off
    action="savestate"
    if [ "$1" == "--poweroff" ]; then
       action="poweroff"
    fi
    __davms "controlvm" $action
}

showvms ()
{
    # show all of the VMs including those that are running
    echo "All VMs:"
    VBoxManage list vms
    echo ""
    echo "Runnings VMs:"
    VBoxManage list runningvms
}

All of the bash functions use a common helper function __davms. I’m sure there are much nicer ways to alter the list but I stuck with sed and gawk.

__davms ()
{
    cmd="VBoxManage $1"
    # some commands require parameters AFTER the vmname is specified
    cmd_suffix="$2"
    case "$1" in
        "controlvm")
            davms=$(VBoxManage list runningvms | sed 's/\"\ {/\"-{/' | grep ^\" | sed 's/\"//g')
            matches=$(echo $davms | gawk -F '}' '{print NF}')
            ;;
        "startvm")
            davms=$(VBoxManage list vms | sed 's/\"\ {/\"-{/' | grep ^\" | sed 's/\"//g')
            matches=$(echo $davms | gawk -F '}' '{print NF}')
            ;;
    esac


    case "$matches" in
        0)
           echo "No vms found"
           show=""
           ;;
        *)
           echo
           echo "Multiple matches found..."
           i=1
           for option in $davms
           do
              echo "$i: $option" | sed 's/-{/\ {/'
              i=`expr $i + 1`
           done
           echo "q: Quit"
           echo 
           read -p "? " ans
           if [ "q" == "$ans" ]; then
              show=""
           else
              show=$(echo $davms | gawk '{print $'$ans'}' | sed 's/-{/\ /' | gawk '{print $1}')
           fi
           ;;
    esac

    if [ "" != "$show" ]; then
       #$cmd $show $cmd_suffix
       echo $cmd $show $cmd_suffix
    fi

}

Here’s what you see when you run these commands:

$ showvms
All VMs:
"os2" {036ff7c1-81b1-47ee-8c63-af4eff9f8998}
"fedora21" {e43a72c8-ce14-40a4-872a-bc863c186569}
"centos7" {f05e6d7f-b054-4b81-88a0-87f58a9bba53}
"windows7" {d7abca3c-3052-4003-8832-76815fc4d69c}

Runnings VMs:
"windows7" {d7abca3c-3052-4003-8832-76815fc4d69c}
$ startvm

Multiple matches found...
1: os2 {036ff7c1-81b1-47ee-8c63-af4eff9f8998}
2: fedora21 {e43a72c8-ce14-40a4-872a-bc863c186569}
3: centos7 {f05e6d7f-b054-4b81-88a0-87f58a9bba53}
4: windows7 {d7abca3c-3052-4003-8832-76815fc4d69c}
q: Quit

? 
$ stopvm

Multiple matches found...
1: windows7 {d7abca3c-3052-4003-8832-76815fc4d69c}
q: Quit

? 

Setting resolution of external monitors on Linux

Connecting my Fedora laptop to an external monitor by default limits the resolution of the external monitor. I know the monitor can do 1920×1080 vs 1024×768. I got tired of looking up the xrandr commands so I wrote up a little script.

Usage: ./addmode x-resolution y-resolution or ./addmode 1920 1080

If there are multiple monitors it will list them out giving you the option to select the correct one.

WARNING: If only one is available it will use that one without prompting. Also the script defaults to 1366×768 unless you pass in an x,y resolution.

#!/bin/sh

xres=$1
yres=$2
fullmodeline=$(cvt 1366 768 | grep Modeline | awk '{$1=""; print $0}')
modeline=$(echo $fullmodeline | awk '{print $1}')
monitors=$(xrandr | grep " connected " | awk '{print $1}')
matches=$(echo $monitors | gawk '{print NF}')

case "$matches" in
    0)
       echo "No matches found"
       show=""
       ;;
    1)
       show=$monitors
       ;;
    *)
       echo
       echo "Multiple matches found..."
       i=1
       for option in $monitors
       do
          echo "$i: $option"
          i=`expr $i + 1`
       done
       echo "q: Quit"
       echo
       read -p "? " ans
       if [ "q" == "$ans" ]; then
          show=""
       else
          show=$(echo $monitors | gawk '{print $'$ans'}')
       fi
       ;;
esac

if [ "" != "$show" ]; then
   #xrandr --newmode $fullmodeline
   xrandr --addmode $show $modeline
fi

You can download the script from https://gist.github.com/jmrodri/cad187f6cb5d5ed832a3

Fedora 13 to Centos 7

A couple of weeks ago, I finally updated my home server from Fedora 13 to CentOS 7. Fedora had the media support but I can’t keep the machine updated fast enough. This time I decided to go with another Red Hat derivative so I chose CentOS. The goal is to have an OS that has a longer release cycle. Most of the services I’m running haven’t changed in a while (considering Fedora 13 was ancient).

This was NOT an in place “upgrade”. Going to CentOS was a fresh install. I backed up all 70G of the root partition to an external drive then told CentOS to reformat the existing partitions. My biggest worry was the 2TB RAID 5 array. During the installation I told the installer to mount the array on /vol but DO NOT FORMAT. I checked it three times to make sure I didn’t lose anything. I did NOT backup the RAID array so this was the most dangerous part of the install I did.

Once installed the task of restoring the data back to the server, users, configuration, install needed software to support the services I had running on the machine, etc. Because Fedora 13 was so old, I couldn’t just rsync /etc.

First thing I had to do was change the UID and GUID of all the backup files since the default UID changed from 500 to 1000. I created the users, then I had to change the access to the backed up files and to the RAID array.

# owner
chown -R --from=500 1000 $1
chown -R --from=501 1001 $1
chown -R --from=502 1002 $1
chown -R --from=504 1004 $1
chown -R --from=505 1005 $1
chown -R --from=506 1006 $1
chown -R --from=507 1007 $1

# groups
chown -R --from=:500 :1000 $1
chown -R --from=:501 :1001 $1
chown -R --from=:502 :1002 $1
chown -R --from=:504 :1004 $1
chown -R --from=:505 :1005 $1
chown -R --from=:506 :1006 $1
chown -R --from=:507 :1007 $1

chown -R --from=481:466 992:989 $1

The firewall switched from iptables to FirewallD. So instead of restoring my /etc/sysconfig/iptables file I decided to figure out how to use FirewallD. I wrote a script for each of the iptables ports:

#...
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-port=PORT#/tcp
#...
firewall-cmd --reload

Most of the services were easy to get back running: http, vnc, and plexmediaserver. Some of the others were a bit tricky to get back up and running.

Streambaby had an error with the newer version of ffmpeg. It would fail with:

Exception in thread "main" java.lang.NoClassDefFoundError: net/sf/ffmpeg_java/v53/AVFormatLibrary …

Thankfully, this post had the suggested solution of simply adding the following to the streambaby.ini

com.unwiredappeal.tivo.vm.ffjava.FFmpegJavaVideoModule=false
ffmpegexe.transcode.sameqargs=-ab 192k

Other Streambaby & multimedia resources:

https://code.google.com/p/streambaby/wiki/StreamBabyIni
http://info.vortexbox.org/tiki-index.php?page=InstallStreamBaby
http://wiki.centos.org/TipsAndTricks/MultimediaOnCentOS7

Next up was to revive the NFS shares.

The server also acts as a TimeMachine drive to backup our MacBook Pro. The old server used netatalk 2.x so the configuration files changed here too.

  • config files moved from /etc/atalk to /etc/netatalk
  • afpd.conf became afp.conf
  • afp conf is completely different

I basically followed the following sources:
http://netatalk.sourceforge.net/wiki/index.php/Netatalk_3.1.7_SRPM_for_Fedora_and_CentOS
http://netatalk.sourceforge.net/3.1/htmldocs/configuration.html#idp139639181431264

The last service was to setup VirtualBox to run headless for my Windows 7 guest. See VirtualBox installation it wasn’t that difficult.

That’s was pretty much the only things I hit during the “upgrade”. It took a couple days to get everything up and running. I did it one service at a time.

One lesson I learned? Don’t wait so long to update your servers :)

Snow removal

Lot’s of folks complain that the south doesn’t know how to deal with snow. We don’t know how to drive in it. We don’t know how to work in it. We don’t have snow trucks, well not many and most are in the mountains. But we do know how to get rid of the snow, we let the Sun take care of it :)

This morning at 9 am both the front and backyard had plenty of snow covering it.

IMG_20150225_091149

IMG_20150225_091127

By 12:30pm, it was pretty much all gone.

IMG_20150225_123250

IMG_20150225_123227

I think we’re ready for the snow now :) Bring it!

14471503-1424897959-400x300