Horrible configure script

A friend of mine was having trouble building the latest version of R on RHEL 6.8. The trouble was that the configure script could not verify the updated zlib. I gave him some advice, but nothing seemed to work. Typically I’ve solved this with a simple update of the LD_LIBRARY_PATH.

Last night, I provisioned a RHEL 6.8 VM, downloaded the latest version of R, and tried building it myself. As expected, the configure script reported that I did not have zlib 1.2.5 installed.

checking if zlib version >= 1.2.5... no
checking whether zlib support suffices... configure: error: zlib library and headers are required

This is simple, just download an updated zlib, build and install. I installed zlib 1.2.11 in /usr/local/lib64. Re-ran configure and go the same result as above. I set LD_LIBRARY_PATH=/usr/local/lib64:$LD_LIBRARY_PATH, configure still complained about not finding an updated zlib.

Updated zlib is installed, LD_LIBRARY_PATH is set. I ran sudo ldconfig -p | grep libz. It shows it can be found in /usr/local/lib64. So frustrating that everything seems to be in order and yet configure doesn’t see it.

My friend was using a blog post to help him out, when I looked at it the author didn’t see to have a problem with configure after updating zlib.

What in the world is the configure script doing to determine what version of zlib we are using?

if test "${have_zlib}" = yes; then
  { $as_echo "$as_me:${as_lineno-$LINENO}: checking if zlib version >= 1.2.5" >&5
$as_echo_n "checking if zlib version >= 1.2.5... " >&6; }
if ${r_cv_header_zlib_h+:} false; then :
  $as_echo_n "(cached) " >&6
else
  if test "$cross_compiling" = yes; then :
  r_cv_header_zlib_h=no
else

  cat confdefs.h - <conftest.$ac_ext
/* end confdefs.h.  */

#include 
#include 
#include 
int main() {
#ifdef ZLIB_VERSION
/* Work around Debian bug: it uses 1.2.3.4 even though there was no such
   version on the master site zlib.net */
  exit(strncmp(ZLIB_VERSION, "1.2.5", 5) < 0);
#else
  exit(1);
#endif
}

So it’s writing out a C program to verify the library. That’s when I saw it.

exit(strncmp(ZLIB_VERSION, "1.2.5", 5) < 0);

ZLIB_VERSION is “1.2.11”. So they are comparing the first 5 characters. So that means “1.2.1” vs “1.2.5” and as you can see “1.2.1” is LESS THAN “1.2.5”. Well that would be a problem and never work. So I tried fixing it by changing it to use 6 characters. Well, “1.2.11” in lexicographic order is before “1.2.5”. So that breaks too.

I know there has to be a better way, but since this isn’t my code I just wanted to it to build. I changed it to:

exit(strncmp(ZLIB_VERSION, "1.2.05", 6) < 0);

Why in the world would they use strncmp to compare versions? Aren’t there much better ways to determine libraries installed on a RHEL system? I was astounded this ever made it out to the wild.

vij update

Vij is a bash function that makes editing specific files easier to find in a deeply nested directory structure. At the time I was working in Java which is notorious for having a deep directory structure. In 2008, I updated vij to be more powerful by offering a list of files matching the name, giving you the option to make a choice.

Our current project is in Golang and I’ve found a deficiency in vij. We have a vendor directory which contains our dependencies and sometimes vij will find files in the vendor directory which I don’t want to edit. It would be nice to exclude a directory from the find command used by vij.

I always want to ignore the vendor directory. But I also want to allow arbitrary names to also be filtered out. The existing vij implementation results in the following crazy output.

$ vij util.go

Multiple matches found...
1: ./pkg/broker/util.go
2: ./pkg/fusortest/util.go
3: ./vendor/k8s.io/kubernetes/vendor/golang.org/x/crypto/ssh/terminal/util.go
4: ./vendor/k8s.io/kubernetes/vendor/github.com/vmware/photon-controller-go-sdk/photon/util.go
5: ./vendor/k8s.io/kubernetes/vendor/github.com/spf13/viper/util.go
6: ./vendor/k8s.io/kubernetes/vendor/github.com/spf13/cobra/doc/util.go
...
16: ./vendor/k8s.io/kubernetes/vendor/github.com/kr/pty/util.go
...
108: ./vendor/github.com/coreos/etcd/client/util.go
q: Quit

? 

I just want to see the util.go files in my code tree and ignore all of the ones in vendor. I changed the dafiles variable to have two more options. One is always ignore vendor from the normal use case, but also allow the user to pass in a string to filter on.

    if [ "$3" == "" ]; then
        dafiles=$(find . -type f -name "$2" -not -path "*/vendor/*")
    else 
        dafiles=$(find . -type f -name "$2" | grep -v $3)
    fi

Now when I run vij util.go, vij ignores the vendor directory.

$ vij util.go

Multiple matches found...
1: ./pkg/broker/util.go
2: ./pkg/fusortest/util.go
q: Quit

? 

Let’s say I want to find all files named User*.java but not the ones in the test directory. You can simply supply “test” as the second argument to vij.

BEFORE

$ vij User*.java

Multiple matches found...
1: ./src/main/java/org/candlepin/auth/permissions/UserUserPermission.java
2: ./src/main/java/org/candlepin/auth/permissions/UsernameConsumersPermission.java
3: ./src/main/java/org/candlepin/auth/UserAuth.java
4: ./src/main/java/org/candlepin/auth/UserPrincipal.java
5: ./src/main/java/org/candlepin/model/User.java
6: ./src/main/java/org/candlepin/model/UserCurator.java
7: ./src/main/java/org/candlepin/resource/UserResource.java
8: ./src/main/java/org/candlepin/service/UserServiceAdapter.java
9: ./src/test/java/org/candlepin/auth/UserPrincipalTest.java
10: ./src/test/java/org/candlepin/auth/permissions/UsernameConsumersPermissionTest.java
11: ./src/test/java/org/candlepin/model/UserTest.java
12: ./src/test/java/org/candlepin/resource/UserResourceTest.java
q: Quit

? 

AFTER

$ vij User*.java test

Multiple matches found...
1: ./src/main/java/org/candlepin/auth/permissions/UserUserPermission.java
2: ./src/main/java/org/candlepin/auth/permissions/UsernameConsumersPermission.java
3: ./src/main/java/org/candlepin/auth/UserAuth.java
4: ./src/main/java/org/candlepin/auth/UserPrincipal.java
5: ./src/main/java/org/candlepin/model/User.java
6: ./src/main/java/org/candlepin/model/UserCurator.java
7: ./src/main/java/org/candlepin/resource/UserResource.java
8: ./src/main/java/org/candlepin/service/UserServiceAdapter.java
q: Quit

? 

Here’s the full implementation of the updated vij:

# reused by other commands that require a filename
__dafiles ()
{
    cmd=$1
    if [ "$3" == "" ]; then
        dafiles=$(find . -type f -name "$2")
        #dafiles=$(find . -type f -name "$2" -not -path "*/vendor/*")
    else
        dafiles=$(find . -type f -name "$2" | grep -v $3)
    fi
    matches=$(echo $dafiles | gawk '{print NF}')

    case "$matches" in
        0)
           echo "No matches found"
           show=""
           ;;
        1)
           show=$dafiles
           ;;
        *)
           echo
           echo "Multiple matches found..."
           i=1
           for option in $dafiles
           do
              echo "$i: $option"
              i=`expr $i + 1`
           done
           echo "q: Quit"
           echo 
           read -p "? " ans
           if [ "q" == "$ans" ]; then
              show=""
           else
              show=$(echo $dafiles | gawk '{print $'$ans'}')
           fi
           ;;
    esac

    if [ "" != "$show" ]; then
       $cmd $show
    fi
}

vij ()
{
    __dafiles "vim" $1 $2
}

Charisma

Looking back through the past elections going back to the 1980 election of Reagan, I noticed something about the winners. All but one had charisma. Despite some candidates being more capable and experienced. The charismatic candidates were better able to rile up their voters to get them to come out and vote, coupled with particular pains felt by those particular voters.

Let’s take a look at the elections:

1980: Reagan vs Carter (incumbent)

We had an oil shortage, inflation was high, coupled with high unemployment. Trouble abroad with Iran (hostages), and some other factors. Reagan was a masterful orator and could communicate with the masses. He gains the nickname, “Great Communicator“. He is able to get everyone on his side, wins in a landslide 489-49 electoral votes. Charisma wins.

1984: Reagan (incumbent) vs Mondale 

The economy is doing better for some, I still heard rumblings of tough times from the lower middle class. AIDS epidemic is front and center in the news. The First Lady, Nancy, is out and about with say no to drugs campaign. Mondale is not a very exciting candidate and things weren’t “feeling” so bad for the incumbent president. But it is hard for Mondale to compete with the charisma of Ronald Reagan, being obliterated in the general election 525-13.  Charisma crushes it.

1988: Bush (HW) vs Dukakis

Ok this is the one election that I felt wasn’t really affected by charisma of the candidates. I mean Dukakis was short and didn’t have the “look” people wanted in a president. Bush is actually quite boring but he gets to ride on the coat tails of a hugely successful and charismatic Reagan. I used to say the Republicans screwed themselves when they passed the two-term limit amendment to prevent another FDR. I think Reagan would’ve easily won a third term. I guess they were protecting themselves from a possible third Clinton or Obama term in the future 🙂 I digress, Bush goes on to win 426-111. We have two pretty boring candidates, for this argument I will give Bush the charisma points. Charisma wins again.

1992: Clinton vs Bush (HW) (incumbent)

Ah my first election, I missed being able to vote in 1988 by a measly one year. So I was excited to begin the process of being able to vote. We’ve entered a new decade. Just coming off the Gulf War, some folks are upset that we went to war. Oil prices spiked in 1990 still fresh in minds. People are still talking about trickle down and supply side economics from the 80s not working for them. A charismatic southern democrat, Bill Clinton. He found a way to engage the young voters. He’s able to charm folks to a win over the incumbent President Bush. Clinton wins 370-168 even with a third party candidate in the mix. Charisma for the win.

1996: Clinton (incumbent) vs Dole

The economy is booming, the internet is opening to public, and gas prices were low, good ole $1 for a gallon. The Dow Jones was hovering in the 5000, by election time it had risen above 6000. Personally, just graduated college in 1995, got a good paying job as a software engineer. The administration wasn’t doing as well publicly though. Despite the scandals happening at the time, like the Lewinsky scandal, Clinton marches to a decisive 379-159 win over the experienced but less charismatic, Dole. Charisma 5 for 5 so far.

2000: Bush (Dubya) vs Gore

The new millennium. We survived the doom and gloom of the Y2K bug where the entire internet was going to just shutdown because of storing years as 2 digits instead of four, nothing happened 🙂 The Dow Jones is well into 10000.  Tech is moving a faster pace: Google, founded in 1998 and  Amazon, founded in 1994, are both thriving. This is the year I got married, relocated to North Carolina, got a new job, and bought our first house all in the span of 4 months. So the Clinton/Gore administration policies were serving me just fine. The rest of the country didn’t seem to agree with me, well they kind of did.  This would’ve been another case where the 22nd amendment hindered a possible third term, Clinton was still quite popular, with an approval rating of low to mid 60s. Despite his approval ratings, Gore was very boring. He was intelligent and a statesman but just bleh. In comes a cowboy, a president you could have a beer with, but has the rough and tough charisma people think we need. This was a devastating election because it was so hotly contested when CNN announces it for Gore only to retract it because of discrepancies in Florida. Ultimately, Bush wins the election 271-266. Cowboy charisma wins over boring statesman.

2004: Bush (incumbent) vs Kerry

Just 3 years after the worst attack on American soil, we are at war in Afghanistan and Iraq. The economy isn’t fairing very well, after the dot com bust. Gas prices are rising, finally hitting $2. Bush has a pretty high approval rating, in the 60s, coming off his first term and his handling of September 11th. Remember, Bush was the cowboy. We put up John Kerry to be the opposing candidate. Kerry was a skilled politician, but not charismatic at all. He added John Edwards to his ticket which helped a little, but not enough to overcome the cowboy charisma of Bush. Bush wins 286-251. Charisma 7, boring 0.

2008: Obama vs McCain

In 2008, we are all annoyed at the high gas prices rising well past $3/gallon. Ravaged New Orleans by Hurricane Katrina which was hugely mismanaged, and Bush was blamed for it. The economy is doing poorly still reeling from the dot com bust and in the midst of what folks termed, The Great Recession. A young charismatic African-American, Barack Obama, rises from the Democratic party to round up millennials and minority groups with an easy to remember slogan: Yes We Can. Contrast this to McCain which many liberals seemed to like except when he was running for president, he became a different person to appeal to his base. He wasn’t very charismatic, which he tried to offset with his running mate, Palin. These two could not be more opposite. Obama wins 365-173 to become the first African-American president.

2012: Obama (incumbent) vs Romney

Two years after the Patient Protection and Affordable Care Act passes, people are upset at being forced to get health care. The administration bailed out the auto industry, bailed out Wall Street, gas prices are still in the upper $3. But the economy has survived, the Dow Jones hit a low of 6627 in 2009. By 2012, it had rebounded back to 13000. Romney was a capable candidate, especially compared to Bush from 2000. He was a successful governor, he would’ve been a capable president. Romney was missing one thing, charisma. He was no match for Obama’s charismatic speeches. Obama wins the 2012 election, 332-206.

2016: Trump vs Clinton (Hillary)

We finally hit the present. Gas prices are back into the low $2 range. The Dow Jones has climbed above 18000. This was an election purely based on charisma. Donald Trump is able to rile up his base to get people voting. He did it in a different manner, a pretty nasty bully but behind all the nasty things he said, he did have a message that many folks heard. But the Democrats thought that an experienced candidate that could maneuver the intricacies of the presidencies would be enough to win. We missed one aspect, Hillary Clinton was a dud from a personality point of view. She always seemed fake, tried to hard to empathize and I believe that was truly her undoing. Trump wins 306-232. Charisma wins yet again.

I’m extremely disappointed in the outcome of this years election. I still believe that Hillary Clinton was a more qualified candidate. I think she would’ve been tougher than any man could ever hope to be. She would’ve been a hawk in the Oval Office. She just couldn’t convince the people of that, which is equally important.

I firmly believe it doesn’t matter how much you know and how capable a leader you could be, if you lack personality you will NOT win the presidency. You need to have the charisma to be able to rile up your voters in order to increase your turn out. That’s really all it takes now, in this age of social media.

Looking forward to 2018 mid-terms and 2020! I just pray to God that Trump doesn’t destroy everything before then.

Network performance

Recently I had some network issues at work with lower than normal speeds, narrowed it down to the D-Link switch. I rebooted it and things seemed to be back to normal. Then at home I was hitting some similar performance issues both wired and wireless.

First thing was to verify that the wall port was giving me the correct speeds, then through the switch. Bingo, speeds were dropping. Ensured I was using Cat6 cables, found one cable that just didn’t want to connect, threw it out. Then I used the IT Crowd mantra and tried turning it off and on again. Speeds were back to normal.

Rebooted the downstairs switch as well, used iperf to measure the wired performance, 935 Mbits/sec.

$ iperf -p 2222 -c 172.31.6.3
------------------------------------------------------------
Client connecting to 172.31.6.3, TCP port 2222
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 172.31.6.221 port 35464 connected with 172.31.6.3 port 2222
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes   935 Mbits/sec
$ iperf -p 2222 -c 172.31.6.3
------------------------------------------------------------
Client connecting to 172.31.6.3, TCP port 2222
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 172.31.6.221 port 35470 connected with 172.31.6.3 port 2222
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes   935 Mbits/sec

So wired performance fixed. Next up was the wireless. I was getting 8.3MBps/66.4 Mbps, second run 7.0MBps/56Mbps. Reading the dd-wrt docs it reads:

You must use HT40 channel width & WPA2 AES when encrypting your wireless connection, otherwise the router will fallback to 802.11a/g speeds since WPA2 AES is a mandatory requirement of 802.11n specification as WPA2 AES is the only approved encryption method in the 802.11n standard.

Security was already WPA2 AES, but the Channel Width was set to the dd-wrt default of Full (20MHz). I changed it to Wide HT40 (40 MHz). Rebooted the router, and gave the tests another go.

First I used dd and nc then iperf. The dd reported 13.7MBps/109.6Mbps.

$ dd if=/dev/zero bs=1024K count=512 | nc -v 172.31.6.3 2222
Ncat: Version 7.12 ( https://nmap.org/ncat )
Ncat: Connected to 172.31.6.3:2222.
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 39.1152 s, 13.7 MB/s
Ncat: 536870912 bytes sent, 0 bytes received in 39.32 seconds.

iperf results in similar results if only slightly slower.

$ iperf -p 2222 -c 172.31.6.3
------------------------------------------------------------
Client connecting to 172.31.6.3, TCP port 2222
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 172.31.6.211 port 52358 connected with 172.31.6.3 port 2222
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   110 MBytes  92.2 Mbits/sec
$ iperf -p 2222 -c 172.31.6.3
------------------------------------------------------------
Client connecting to 172.31.6.3, TCP port 2222
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 172.31.6.211 port 52360 connected with 172.31.6.3 port 2222
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   102 MBytes  85.0 Mbits/sec

Two runs resulted in 92.2Mbps and 85.0Mbps. The three runs average out to 95.6Mbps wireless.

While not bad wireless performance, I was hoping to get closer to the 300Mbps product rating.

Network Hardware:

Computers:

  • Lenovo T460S
  • homebuilt server, MSI 785GTM-E45 motherboard with gigabit port

More Go links

Cobra is both a library for creating powerful modern CLI applications as well as a program to generate applications and command files.

https://github.com/spf13/cobra

Viper is a complete configuration solution for go applications including 12 factor apps. It is designed to work within an application, and can handle all types of configuration needs and formats.

https://github.com/spf13/viper

pflag is a drop-in replacement for Go’s flag package, implementing POSIX/GNU-style –flags.

https://github.com/spf13/pflag

7 common mistakes in Go and when to avoid them
https://www.youtube.com/watch?v=29LLRKIL_TI

Go! Speed Racer Go!

I finally reached a point where I could start running the go version of sm-photo-tool. I finished the option validation for the list command. While I was testing it I noticed how much faster the Go version felt. Here are the python vs Go versions of the commands.

The python version took 80 milliseconds to run the validation of parameters and print out the error message. This is expected, you have to start the python VM (and if not precompiled, the code would need to be compiled). In this test, the code was precompiled.

$ time sm-photo-tool list invalid blah
ERROR: valid options are ['album', 'galleries']

real	0m0.080s
user	0m0.059s
sys	0m0.021s

Ok so how fast is the Go version? My guess was half the time, 40ms. I was way off. Try 6ms. SIX! That’s amazing to do the exact same amount of work.

$ time ./sm-photo-tool list invalid blah
ERROR: valid options are [album galleries]

real	0m0.006s
user	0m0.001s
sys	0m0.005s