Fun with cli file transformation

I used the Wake County property tax page to get a list of all of the homes in our neighborhood. I pasted that into a spreadsheet. The Proper and Proper last columns are actually cells using the =PROPER() function. It takes the given cell and converts it from all upper case to proper case. The resulting list looks like this:

Address Street Last name Owners Proper Proper last E-mail
123 SCRABBLES DR JONES FRED Fred Jones neighbor1@email.com
456 SCRABBLES DR BONES WILMA Wilma Bones neighbor2@eahoo.com

I probably could’ve done this whole thing without using the spreadsheet or the =PROPER() function but it was easier considering the original data came from a table. I then exported that as a CSV file.

Address,Street,Last name,Owners,Proper,Proper last,E-mail,
123,SCRABBLES DR,JONES,FRED,Fred,Jones,neighbor1@email.com,
456,SCRABBLES DR,BONES,WILMA,Wilma,Bones,neighbor2@eahoo.com,

What I’m trying to do is get a list with the following format:

"First Lastname" <emailaddress@domain.com>, "First Lastname" <email2@domain.com>

I did it with the following command line. The trickiest was getting awk to print out the double quotes uses “\42”

cat neighbors.csv | grep -v ^Address | awk -F , '{print "\42"$5" "$6"\42 <"$7">,"}' | grep -v "<>" | tr '\n' ' ' > full-list.txt

I then copy and pasted the resulting file contents into the “Direct add members” feature. The one tidbit that I had to do was only copy and paste 10 addresses at once. I probably could’ve updated the above script to do that for me. But by this point I got 90% of what I needed so I just split the rest by hand.

Remote connection to Windows from Nexus 4

I setup Windows 7 running in a VirtualBox guest on my CentOS 7 Linux server (mostly for doing taxes and having a Windows test machine). I’ve configured it to start headless, that is without a monitor on the server. My main means of connection is using rdesktop from my Fedora Linux desktops.

Tonight, I decided to see how smart my smartphone really is. Could I connect remotely to a Windows 7 box using a 4.7″ Nexus 4 Android phone? The answer? YES! Using RD Client I connected to the Windows machine and was able to interact with it. Is this a usable solution on such a small screen? No way. But if you need to access something you might be able to get around well enough to get what you want.

Connecting to Windows 7 using RD client on a Nexus 4 phone

If this was going to be something you did regularly, I’d suggest at least a 7″ tablet, probably a good 10″ one would be best.

Any way, why did I do this? Because I can. Isn’t technology fun?

aliases for VirtualBox’ VBoxManage commands

VBoxManage allows you to interact with VirtualBox using the CLI. You can do just about anything with the command, from starting/stopping VMs. Getting detailed information, modifying or even handling the disks and other storage media. To see all you can do with it checkout this document:
VBoxManage

I typically deal with 3 commands, starting/stopping vms and seeing the list of vms.

VBoxManage list vms|runningvms|...
VBoxManage startvm vmname --type headless
VBoxManage controlvm vmname savestate|poweroff

To make it easier I created some aliases for the above commands in my .bashrc as bash functions:

  • showvms
  • startvm
  • stopvm

Here are the functions:

startvm ()
{
    # start a VM as headless, no need for a GUI
    __davms "startvm" "--type headless"
}

stopvm ()
{
    # allow stopping a VM by saving its state or simply turning it off
    action="savestate"
    if [ "$1" == "--poweroff" ]; then
       action="poweroff"
    fi
    __davms "controlvm" $action
}

showvms ()
{
    # show all of the VMs including those that are running
    echo "All VMs:"
    VBoxManage list vms
    echo ""
    echo "Runnings VMs:"
    VBoxManage list runningvms
}

All of the bash functions use a common helper function __davms. I’m sure there are much nicer ways to alter the list but I stuck with sed and gawk.

__davms ()
{
    cmd="VBoxManage $1"
    # some commands require parameters AFTER the vmname is specified
    cmd_suffix="$2"
    case "$1" in
        "controlvm")
            davms=$(VBoxManage list runningvms | sed 's/\"\ {/\"-{/' | grep ^\" | sed 's/\"//g')
            matches=$(echo $davms | gawk -F '}' '{print NF}')
            ;;
        "startvm")
            davms=$(VBoxManage list vms | sed 's/\"\ {/\"-{/' | grep ^\" | sed 's/\"//g')
            matches=$(echo $davms | gawk -F '}' '{print NF}')
            ;;
    esac


    case "$matches" in
        0)
           echo "No vms found"
           show=""
           ;;
        *)
           echo
           echo "Multiple matches found..."
           i=1
           for option in $davms
           do
              echo "$i: $option" | sed 's/-{/\ {/'
              i=`expr $i + 1`
           done
           echo "q: Quit"
           echo 
           read -p "? " ans
           if [ "q" == "$ans" ]; then
              show=""
           else
              show=$(echo $davms | gawk '{print $'$ans'}' | sed 's/-{/\ /' | gawk '{print $1}')
           fi
           ;;
    esac

    if [ "" != "$show" ]; then
       #$cmd $show $cmd_suffix
       echo $cmd $show $cmd_suffix
    fi

}

Here’s what you see when you run these commands:

$ showvms
All VMs:
"os2" {036ff7c1-81b1-47ee-8c63-af4eff9f8998}
"fedora21" {e43a72c8-ce14-40a4-872a-bc863c186569}
"centos7" {f05e6d7f-b054-4b81-88a0-87f58a9bba53}
"windows7" {d7abca3c-3052-4003-8832-76815fc4d69c}

Runnings VMs:
"windows7" {d7abca3c-3052-4003-8832-76815fc4d69c}
$ startvm

Multiple matches found...
1: os2 {036ff7c1-81b1-47ee-8c63-af4eff9f8998}
2: fedora21 {e43a72c8-ce14-40a4-872a-bc863c186569}
3: centos7 {f05e6d7f-b054-4b81-88a0-87f58a9bba53}
4: windows7 {d7abca3c-3052-4003-8832-76815fc4d69c}
q: Quit

? 
$ stopvm

Multiple matches found...
1: windows7 {d7abca3c-3052-4003-8832-76815fc4d69c}
q: Quit

? 

Setting resolution of external monitors on Linux

Connecting my Fedora laptop to an external monitor by default limits the resolution of the external monitor. I know the monitor can do 1920×1080 vs 1024×768. I got tired of looking up the xrandr commands so I wrote up a little script.

Usage: ./addmode x-resolution y-resolution or ./addmode 1920 1080

If there are multiple monitors it will list them out giving you the option to select the correct one.

WARNING: If only one is available it will use that one without prompting. Also the script defaults to 1366×768 unless you pass in an x,y resolution.

#!/bin/sh

xres=$1
yres=$2
fullmodeline=$(cvt 1366 768 | grep Modeline | awk '{$1=""; print $0}')
modeline=$(echo $fullmodeline | awk '{print $1}')
monitors=$(xrandr | grep " connected " | awk '{print $1}')
matches=$(echo $monitors | gawk '{print NF}')

case "$matches" in
    0)
       echo "No matches found"
       show=""
       ;;
    1)
       show=$monitors
       ;;
    *)
       echo
       echo "Multiple matches found..."
       i=1
       for option in $monitors
       do
          echo "$i: $option"
          i=`expr $i + 1`
       done
       echo "q: Quit"
       echo
       read -p "? " ans
       if [ "q" == "$ans" ]; then
          show=""
       else
          show=$(echo $monitors | gawk '{print $'$ans'}')
       fi
       ;;
esac

if [ "" != "$show" ]; then
   #xrandr --newmode $fullmodeline
   xrandr --addmode $show $modeline
fi

You can download the script from https://gist.github.com/jmrodri/cad187f6cb5d5ed832a3

Fedora 13 to Centos 7

A couple of weeks ago, I finally updated my home server from Fedora 13 to CentOS 7. Fedora had the media support but I can’t keep the machine updated fast enough. This time I decided to go with another Red Hat derivative so I chose CentOS. The goal is to have an OS that has a longer release cycle. Most of the services I’m running haven’t changed in a while (considering Fedora 13 was ancient).

This was NOT an in place “upgrade”. Going to CentOS was a fresh install. I backed up all 70G of the root partition to an external drive then told CentOS to reformat the existing partitions. My biggest worry was the 2TB RAID 5 array. During the installation I told the installer to mount the array on /vol but DO NOT FORMAT. I checked it three times to make sure I didn’t lose anything. I did NOT backup the RAID array so this was the most dangerous part of the install I did.

Once installed the task of restoring the data back to the server, users, configuration, install needed software to support the services I had running on the machine, etc. Because Fedora 13 was so old, I couldn’t just rsync /etc.

First thing I had to do was change the UID and GUID of all the backup files since the default UID changed from 500 to 1000. I created the users, then I had to change the access to the backed up files and to the RAID array.

# owner
chown -R --from=500 1000 $1
chown -R --from=501 1001 $1
chown -R --from=502 1002 $1
chown -R --from=504 1004 $1
chown -R --from=505 1005 $1
chown -R --from=506 1006 $1
chown -R --from=507 1007 $1

# groups
chown -R --from=:500 :1000 $1
chown -R --from=:501 :1001 $1
chown -R --from=:502 :1002 $1
chown -R --from=:504 :1004 $1
chown -R --from=:505 :1005 $1
chown -R --from=:506 :1006 $1
chown -R --from=:507 :1007 $1

chown -R --from=481:466 992:989 $1

The firewall switched from iptables to FirewallD. So instead of restoring my /etc/sysconfig/iptables file I decided to figure out how to use FirewallD. I wrote a script for each of the iptables ports:

#...
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-port=PORT#/tcp
#...
firewall-cmd --reload

Most of the services were easy to get back running: http, vnc, and plexmediaserver. Some of the others were a bit tricky to get back up and running.

Streambaby had an error with the newer version of ffmpeg. It would fail with:

Exception in thread "main" java.lang.NoClassDefFoundError: net/sf/ffmpeg_java/v53/AVFormatLibrary …

Thankfully, this post had the suggested solution of simply adding the following to the streambaby.ini

com.unwiredappeal.tivo.vm.ffjava.FFmpegJavaVideoModule=false
ffmpegexe.transcode.sameqargs=-ab 192k

Other Streambaby & multimedia resources:

https://code.google.com/p/streambaby/wiki/StreamBabyIni
http://info.vortexbox.org/tiki-index.php?page=InstallStreamBaby
http://wiki.centos.org/TipsAndTricks/MultimediaOnCentOS7

Next up was to revive the NFS shares.

The server also acts as a TimeMachine drive to backup our MacBook Pro. The old server used netatalk 2.x so the configuration files changed here too.

  • config files moved from /etc/atalk to /etc/netatalk
  • afpd.conf became afp.conf
  • afp conf is completely different

I basically followed the following sources:
http://netatalk.sourceforge.net/wiki/index.php/Netatalk_3.1.7_SRPM_for_Fedora_and_CentOS
http://netatalk.sourceforge.net/3.1/htmldocs/configuration.html#idp139639181431264

The last service was to setup VirtualBox to run headless for my Windows 7 guest. See VirtualBox installation it wasn’t that difficult.

That’s was pretty much the only things I hit during the “upgrade”. It took a couple days to get everything up and running. I did it one service at a time.

One lesson I learned? Don’t wait so long to update your servers 🙂