vij update

Vij is a bash function that makes editing specific files easier to find in a deeply nested directory structure. At the time I was working in Java which is notorious for having a deep directory structure. In 2008, I updated vij to be more powerful by offering a list of files matching the name, giving you the option to make a choice.

Our current project is in Golang and I’ve found a deficiency in vij. We have a vendor directory which contains our dependencies and sometimes vij will find files in the vendor directory which I don’t want to edit. It would be nice to exclude a directory from the find command used by vij.

I always want to ignore the vendor directory. But I also want to allow arbitrary names to also be filtered out. The existing vij implementation results in the following crazy output.

$ vij util.go

Multiple matches found...
1: ./pkg/broker/util.go
2: ./pkg/fusortest/util.go
3: ./vendor/k8s.io/kubernetes/vendor/golang.org/x/crypto/ssh/terminal/util.go
4: ./vendor/k8s.io/kubernetes/vendor/github.com/vmware/photon-controller-go-sdk/photon/util.go
5: ./vendor/k8s.io/kubernetes/vendor/github.com/spf13/viper/util.go
6: ./vendor/k8s.io/kubernetes/vendor/github.com/spf13/cobra/doc/util.go
...
16: ./vendor/k8s.io/kubernetes/vendor/github.com/kr/pty/util.go
...
108: ./vendor/github.com/coreos/etcd/client/util.go
q: Quit

? 

I just want to see the util.go files in my code tree and ignore all of the ones in vendor. I changed the dafiles variable to have two more options. One is always ignore vendor from the normal use case, but also allow the user to pass in a string to filter on.

    if [ "$3" == "" ]; then
        dafiles=$(find . -type f -name "$2" -not -path "*/vendor/*")
    else 
        dafiles=$(find . -type f -name "$2" | grep -v $3)
    fi

Now when I run vij util.go, vij ignores the vendor directory.

$ vij util.go

Multiple matches found...
1: ./pkg/broker/util.go
2: ./pkg/fusortest/util.go
q: Quit

? 

Let’s say I want to find all files named User*.java but not the ones in the test directory. You can simply supply “test” as the second argument to vij.

BEFORE

$ vij User*.java

Multiple matches found...
1: ./src/main/java/org/candlepin/auth/permissions/UserUserPermission.java
2: ./src/main/java/org/candlepin/auth/permissions/UsernameConsumersPermission.java
3: ./src/main/java/org/candlepin/auth/UserAuth.java
4: ./src/main/java/org/candlepin/auth/UserPrincipal.java
5: ./src/main/java/org/candlepin/model/User.java
6: ./src/main/java/org/candlepin/model/UserCurator.java
7: ./src/main/java/org/candlepin/resource/UserResource.java
8: ./src/main/java/org/candlepin/service/UserServiceAdapter.java
9: ./src/test/java/org/candlepin/auth/UserPrincipalTest.java
10: ./src/test/java/org/candlepin/auth/permissions/UsernameConsumersPermissionTest.java
11: ./src/test/java/org/candlepin/model/UserTest.java
12: ./src/test/java/org/candlepin/resource/UserResourceTest.java
q: Quit

? 

AFTER

$ vij User*.java test

Multiple matches found...
1: ./src/main/java/org/candlepin/auth/permissions/UserUserPermission.java
2: ./src/main/java/org/candlepin/auth/permissions/UsernameConsumersPermission.java
3: ./src/main/java/org/candlepin/auth/UserAuth.java
4: ./src/main/java/org/candlepin/auth/UserPrincipal.java
5: ./src/main/java/org/candlepin/model/User.java
6: ./src/main/java/org/candlepin/model/UserCurator.java
7: ./src/main/java/org/candlepin/resource/UserResource.java
8: ./src/main/java/org/candlepin/service/UserServiceAdapter.java
q: Quit

? 

Here’s the full implementation of the updated vij:

# reused by other commands that require a filename
__dafiles ()
{
    cmd=$1
    if [ "$3" == "" ]; then
        dafiles=$(find . -type f -name "$2")
        #dafiles=$(find . -type f -name "$2" -not -path "*/vendor/*")
    else
        dafiles=$(find . -type f -name "$2" | grep -v $3)
    fi
    matches=$(echo $dafiles | gawk '{print NF}')

    case "$matches" in
        0)
           echo "No matches found"
           show=""
           ;;
        1)
           show=$dafiles
           ;;
        *)
           echo
           echo "Multiple matches found..."
           i=1
           for option in $dafiles
           do
              echo "$i: $option"
              i=`expr $i + 1`
           done
           echo "q: Quit"
           echo 
           read -p "? " ans
           if [ "q" == "$ans" ]; then
              show=""
           else
              show=$(echo $dafiles | gawk '{print $'$ans'}')
           fi
           ;;
    esac

    if [ "" != "$show" ]; then
       $cmd $show
    fi
}

vij ()
{
    __dafiles "vim" $1 $2
}

Control your caps

On all of my computers I use gnome-tweak-tool to remap CAPSLOCK to CTRL. But there are times when I’m working in a virtual machine that doesn’t have that setup so I need to go back to old school methods.

If you’re on a box just run this from any terminal:

setxkbmap -option ctrl:nocaps

That avoids trying to kill a program with C which usually doesn’t work 🙂

Pretty Print directory of .json files

I had a bunch of compressed json files that I needed to pretty print to make them more readable. This little snippet will create a new pretty printed json file prefixed with pp:

ls *.json | xargs -I {} sh -c "cat {} | python -mjson.tool > pp{}"

Instead of having to look at files that look like this:

{ "attributes": [ { "name": "type", "value": "PKT" }, { "name": "arch", "value": "x86_64,x86" }, { "name": "name", "value": "Awesome OS" } ], "dependentProductIds": [], "href": "/products/00", "id": "00", "multiplier": 1, "name": "Awesome OS", "productContent": [ { "content": { "arches": null, "contentUrl": "/content/6/$releasever/$basearch/debug", "gpgUrl": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-awesome-os", "id": "FFFF", "label": "awesome-os-debug-rpms", "metadataExpire": 86400, "modifiedProductIds": [ "0A" ], "name": "Awesome OS (Debug RPMs)", "releaseVer": null, "requiredTags": "awesome-os-server", "type": "yum", "vendor": "Candlepin" }, "enabled": false } ] }

You get a bunch of files that look like this:

{
    "attributes": [
        {
            "name": "type",
            "value": "PKT"
        },
        {
            "name": "arch",
            "value": "x86_64,x86"
        },
        {
            "name": "name",
            "value": "Awesome OS"
        }
    ],
    "dependentProductIds": [],
    "href": "/products/00",
    "id": "00",
    "multiplier": 1,
    "name": "Awesome OS",
    "productContent": [
        {
            "content": {
                "arches": null,
                "contentUrl": "/content/6/$releasever/$basearch/debug",
                "gpgUrl": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-awesome-os",
                "id": "FFFF",
                "label": "awesome-os-debug-rpms",
                "metadataExpire": 86400,
                "modifiedProductIds": [
                    "0A"
                ],
                "name": "Awesome OS (Debug RPMs)",
                "releaseVer": null,
                "requiredTags": "awesome-os-server",
                "type": "yum",
                "vendor": "Candlepin"
            },
            "enabled": false
        }
    ]
}

This morning I came in to find that my root partition on my workstation was full, eaten up by the development VM (virtual machine) that had grown to 145GB. SIGH! This has happened in the past and usually I delete old VMs, move VMs to my home directory on /home etc.

Today I decided it was time to fix the problem. The root partition is 193G and /home is 280G. The home dir has about 210G free and after removing the development VM from root, it now has 150GB free. If I merge the two partitions I’ll have 360GB of free space to use for VMs etc. I will lose the nice ability to reinstall linux and not format /home partition as now it will be part of root but it avoids the day to day minutiae of managing free space between the two partitions.

I’m fortunate that I can free up the space on root enough to copy the contents of /home over to it. Otherwise the process would involve freeing up space on /home and reusing it on /root, blah blah blah.

First let’s copy /home to a new directory on root, this will create a /home-new/home but that’s ok.

rsync -avzX /home /home-new

Next, let’s unmount and remove /home.

umount /home
lvremove /dev/vg0/home

That freed up 280G, so we can now grow the root partition to use up that space.

lvextend -l +100%FREE /dev/vg0/root

So far so good, now we need to grow the filesystem of the root partition to use the new space as well. I’m using XFS, make sure you use the proper filesystem tool to resize the one you are using.

xfs_growfs /

Let’s restore home.

mv /home-new/home/* /home/

Update /etc/fstab not to mount /home as a different partition. Then reboot just to make sure things are good.

Fun with cli file transformation

I used the Wake County property tax page to get a list of all of the homes in our neighborhood. I pasted that into a spreadsheet. The Proper and Proper last columns are actually cells using the =PROPER() function. It takes the given cell and converts it from all upper case to proper case. The resulting list looks like this:

Address Street Last name Owners Proper Proper last E-mail
123 SCRABBLES DR JONES FRED Fred Jones neighbor1@email.com
456 SCRABBLES DR BONES WILMA Wilma Bones neighbor2@eahoo.com

I probably could’ve done this whole thing without using the spreadsheet or the =PROPER() function but it was easier considering the original data came from a table. I then exported that as a CSV file.

Address,Street,Last name,Owners,Proper,Proper last,E-mail,
123,SCRABBLES DR,JONES,FRED,Fred,Jones,neighbor1@email.com,
456,SCRABBLES DR,BONES,WILMA,Wilma,Bones,neighbor2@eahoo.com,

What I’m trying to do is get a list with the following format:

"First Lastname" <emailaddress@domain.com>, "First Lastname" <email2@domain.com>

I did it with the following command line. The trickiest was getting awk to print out the double quotes uses “\42”

cat neighbors.csv | grep -v ^Address | awk -F , '{print "\42"$5" "$6"\42 <"$7">,"}' | grep -v "<>" | tr '\n' ' ' > full-list.txt

I then copy and pasted the resulting file contents into the “Direct add members” feature. The one tidbit that I had to do was only copy and paste 10 addresses at once. I probably could’ve updated the above script to do that for me. But by this point I got 90% of what I needed so I just split the rest by hand.

Remote connection to Windows from Nexus 4

I setup Windows 7 running in a VirtualBox guest on my CentOS 7 Linux server (mostly for doing taxes and having a Windows test machine). I’ve configured it to start headless, that is without a monitor on the server. My main means of connection is using rdesktop from my Fedora Linux desktops.

Tonight, I decided to see how smart my smartphone really is. Could I connect remotely to a Windows 7 box using a 4.7″ Nexus 4 Android phone? The answer? YES! Using RD Client I connected to the Windows machine and was able to interact with it. Is this a usable solution on such a small screen? No way. But if you need to access something you might be able to get around well enough to get what you want.

Connecting to Windows 7 using RD client on a Nexus 4 phone

If this was going to be something you did regularly, I’d suggest at least a 7″ tablet, probably a good 10″ one would be best.

Any way, why did I do this? Because I can. Isn’t technology fun?

aliases for VirtualBox’ VBoxManage commands

VBoxManage allows you to interact with VirtualBox using the CLI. You can do just about anything with the command, from starting/stopping VMs. Getting detailed information, modifying or even handling the disks and other storage media. To see all you can do with it checkout this document:
VBoxManage

I typically deal with 3 commands, starting/stopping vms and seeing the list of vms.

VBoxManage list vms|runningvms|...
VBoxManage startvm vmname --type headless
VBoxManage controlvm vmname savestate|poweroff

To make it easier I created some aliases for the above commands in my .bashrc as bash functions:

  • showvms
  • startvm
  • stopvm

Here are the functions:

startvm ()
{
    # start a VM as headless, no need for a GUI
    __davms "startvm" "--type headless"
}

stopvm ()
{
    # allow stopping a VM by saving its state or simply turning it off
    action="savestate"
    if [ "$1" == "--poweroff" ]; then
       action="poweroff"
    fi
    __davms "controlvm" $action
}

showvms ()
{
    # show all of the VMs including those that are running
    echo "All VMs:"
    VBoxManage list vms
    echo ""
    echo "Runnings VMs:"
    VBoxManage list runningvms
}

All of the bash functions use a common helper function __davms. I’m sure there are much nicer ways to alter the list but I stuck with sed and gawk.

__davms ()
{
    cmd="VBoxManage $1"
    # some commands require parameters AFTER the vmname is specified
    cmd_suffix="$2"
    case "$1" in
        "controlvm")
            davms=$(VBoxManage list runningvms | sed 's/\"\ {/\"-{/' | grep ^\" | sed 's/\"//g')
            matches=$(echo $davms | gawk -F '}' '{print NF}')
            ;;
        "startvm")
            davms=$(VBoxManage list vms | sed 's/\"\ {/\"-{/' | grep ^\" | sed 's/\"//g')
            matches=$(echo $davms | gawk -F '}' '{print NF}')
            ;;
    esac


    case "$matches" in
        0)
           echo "No vms found"
           show=""
           ;;
        *)
           echo
           echo "Multiple matches found..."
           i=1
           for option in $davms
           do
              echo "$i: $option" | sed 's/-{/\ {/'
              i=`expr $i + 1`
           done
           echo "q: Quit"
           echo 
           read -p "? " ans
           if [ "q" == "$ans" ]; then
              show=""
           else
              show=$(echo $davms | gawk '{print $'$ans'}' | sed 's/-{/\ /' | gawk '{print $1}')
           fi
           ;;
    esac

    if [ "" != "$show" ]; then
       #$cmd $show $cmd_suffix
       echo $cmd $show $cmd_suffix
    fi

}

Here’s what you see when you run these commands:

$ showvms
All VMs:
"os2" {036ff7c1-81b1-47ee-8c63-af4eff9f8998}
"fedora21" {e43a72c8-ce14-40a4-872a-bc863c186569}
"centos7" {f05e6d7f-b054-4b81-88a0-87f58a9bba53}
"windows7" {d7abca3c-3052-4003-8832-76815fc4d69c}

Runnings VMs:
"windows7" {d7abca3c-3052-4003-8832-76815fc4d69c}
$ startvm

Multiple matches found...
1: os2 {036ff7c1-81b1-47ee-8c63-af4eff9f8998}
2: fedora21 {e43a72c8-ce14-40a4-872a-bc863c186569}
3: centos7 {f05e6d7f-b054-4b81-88a0-87f58a9bba53}
4: windows7 {d7abca3c-3052-4003-8832-76815fc4d69c}
q: Quit

? 
$ stopvm

Multiple matches found...
1: windows7 {d7abca3c-3052-4003-8832-76815fc4d69c}
q: Quit

? 

Setting resolution of external monitors on Linux

Connecting my Fedora laptop to an external monitor by default limits the resolution of the external monitor. I know the monitor can do 1920×1080 vs 1024×768. I got tired of looking up the xrandr commands so I wrote up a little script.

Usage: ./addmode x-resolution y-resolution or ./addmode 1920 1080

If there are multiple monitors it will list them out giving you the option to select the correct one.

WARNING: If only one is available it will use that one without prompting. Also the script defaults to 1366×768 unless you pass in an x,y resolution.

#!/bin/sh

xres=$1
yres=$2
fullmodeline=$(cvt 1366 768 | grep Modeline | awk '{$1=""; print $0}')
modeline=$(echo $fullmodeline | awk '{print $1}')
monitors=$(xrandr | grep " connected " | awk '{print $1}')
matches=$(echo $monitors | gawk '{print NF}')

case "$matches" in
    0)
       echo "No matches found"
       show=""
       ;;
    1)
       show=$monitors
       ;;
    *)
       echo
       echo "Multiple matches found..."
       i=1
       for option in $monitors
       do
          echo "$i: $option"
          i=`expr $i + 1`
       done
       echo "q: Quit"
       echo
       read -p "? " ans
       if [ "q" == "$ans" ]; then
          show=""
       else
          show=$(echo $monitors | gawk '{print $'$ans'}')
       fi
       ;;
esac

if [ "" != "$show" ]; then
   #xrandr --newmode $fullmodeline
   xrandr --addmode $show $modeline
fi

You can download the script from https://gist.github.com/jmrodri/cad187f6cb5d5ed832a3

Fedora 13 to Centos 7

A couple of weeks ago, I finally updated my home server from Fedora 13 to CentOS 7. Fedora had the media support but I can’t keep the machine updated fast enough. This time I decided to go with another Red Hat derivative so I chose CentOS. The goal is to have an OS that has a longer release cycle. Most of the services I’m running haven’t changed in a while (considering Fedora 13 was ancient).

This was NOT an in place “upgrade”. Going to CentOS was a fresh install. I backed up all 70G of the root partition to an external drive then told CentOS to reformat the existing partitions. My biggest worry was the 2TB RAID 5 array. During the installation I told the installer to mount the array on /vol but DO NOT FORMAT. I checked it three times to make sure I didn’t lose anything. I did NOT backup the RAID array so this was the most dangerous part of the install I did.

Once installed the task of restoring the data back to the server, users, configuration, install needed software to support the services I had running on the machine, etc. Because Fedora 13 was so old, I couldn’t just rsync /etc.

First thing I had to do was change the UID and GUID of all the backup files since the default UID changed from 500 to 1000. I created the users, then I had to change the access to the backed up files and to the RAID array.

# owner
chown -R --from=500 1000 $1
chown -R --from=501 1001 $1
chown -R --from=502 1002 $1
chown -R --from=504 1004 $1
chown -R --from=505 1005 $1
chown -R --from=506 1006 $1
chown -R --from=507 1007 $1

# groups
chown -R --from=:500 :1000 $1
chown -R --from=:501 :1001 $1
chown -R --from=:502 :1002 $1
chown -R --from=:504 :1004 $1
chown -R --from=:505 :1005 $1
chown -R --from=:506 :1006 $1
chown -R --from=:507 :1007 $1

chown -R --from=481:466 992:989 $1

The firewall switched from iptables to FirewallD. So instead of restoring my /etc/sysconfig/iptables file I decided to figure out how to use FirewallD. I wrote a script for each of the iptables ports:

#...
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-port=PORT#/tcp
#...
firewall-cmd --reload

Most of the services were easy to get back running: http, vnc, and plexmediaserver. Some of the others were a bit tricky to get back up and running.

Streambaby had an error with the newer version of ffmpeg. It would fail with:

Exception in thread "main" java.lang.NoClassDefFoundError: net/sf/ffmpeg_java/v53/AVFormatLibrary …

Thankfully, this post had the suggested solution of simply adding the following to the streambaby.ini

com.unwiredappeal.tivo.vm.ffjava.FFmpegJavaVideoModule=false
ffmpegexe.transcode.sameqargs=-ab 192k

Other Streambaby & multimedia resources:

https://code.google.com/p/streambaby/wiki/StreamBabyIni
http://info.vortexbox.org/tiki-index.php?page=InstallStreamBaby
http://wiki.centos.org/TipsAndTricks/MultimediaOnCentOS7

Next up was to revive the NFS shares.

The server also acts as a TimeMachine drive to backup our MacBook Pro. The old server used netatalk 2.x so the configuration files changed here too.

  • config files moved from /etc/atalk to /etc/netatalk
  • afpd.conf became afp.conf
  • afp conf is completely different

I basically followed the following sources:
http://netatalk.sourceforge.net/wiki/index.php/Netatalk_3.1.7_SRPM_for_Fedora_and_CentOS
http://netatalk.sourceforge.net/3.1/htmldocs/configuration.html#idp139639181431264

The last service was to setup VirtualBox to run headless for my Windows 7 guest. See VirtualBox installation it wasn’t that difficult.

That’s was pretty much the only things I hit during the “upgrade”. It took a couple days to get everything up and running. I did it one service at a time.

One lesson I learned? Don’t wait so long to update your servers 🙂

Deal with git am failures

Today I needed to take my model git branch which was based off of the develop branch a week ago, and rebase it off of the master. First I did the obvious, git rebase master in my model branch, that yielded tons of conflicts. The conflicts were a result of commits in the develop branch being squashed when they got merged to master to keep the history clean. Instead of fixing tons of conflicts, I simply chose to take my 3 commits and create patches from them:

git format-patch -3 HEAD

I created a new branch off of master and applied my patches using git am.

git checkout master
git checkout -b model_updates
git am ~/0001-add-rhev-params-remove-serialized-hash.patch 
git am ~/0002-start-v21-of-deployments.patch
git am ~/0003-rename-columns-and-include-missing-columns.patch

Patch 0001 and ultimately 0003 applied without problems. Patch 0002 on the other hand gave me fits, I expected it, but I refused to recode the change. git am ... just fails when it hits a conflict. Using the trusty Google, I found this blog: Deal with git am failures. I followed the instructions from that blog post:

git am --abort #abort the previously failed 0002 patch run
git apply ~/0002-start-v21-of-deployments.patch --reject # yield .rej files I could inspect
git add server/app/controllers/fusor/api/v2/deployments_controller.rb # ensure changes are fine
git add server/app/controllers/fusor/api/v21/ # new files from patch
git am --resolved # tells git we're good to go, proceed.

Done! Way easier than dealing with the conflicts from the rebase. Easier than trying to recode the patch as well.