Busy weekend

It was a busy weekend, Liz went away for the weekend while I watched the 3 kids. They were actually very well behaved and gave me little to no trouble. I was expecting quite a bit 😀 We went to their Friday playgroup, afterwards had the kid’s favorite: McDonald’s for dinner. On Saturday, we went to Border’s to get the kid’s some books, I went with the intention to buy Doctor Who Season 2, but didn’t want to pay the $68 for it. I ended up getting a Doctor Who book instead, yeah I’m as surprised as you that I ended up with a book.

In the evenings I spent the time catching up on my sci-fi shows: finished watching Doctor Who Season 4, saw all of Season 1 (I started with Season 2), and I started watching Firefly (thanks robin). I also managed to get 2 patches submitted and accepted to redstone-xmlrpc, and I submitted the package to jpackage for review. Another library I use all the time is jdbcLogDriver written by a former colleague of mine, I got commit access to it this weekend as well. Today I started to shake the dust off zmugfs. I still haven’t been able to add write access to the file system, and the read mode is extremely slow at startup. I need a better algorithm for downloading the photos and caching them.

zmugfs roadmap

I released zmugfs 0.1 on October 31st, 2007, but I haven’t done anything else with it since then. I’m overdue for finishing it.

Next up is to add the write mode capability to the file system which will actually allow you to upload your photos to smugmug.com. Here’s a quick list:

  • ability to create:
    • albums
    • categories
    • subcategories
  • upload photos
  • edit attributes of photos (not quite sure how this will work)

Stay tuned for more.

    SOAP is dirty!

    Contrary to popular belief, SOAP is dirty! When folks talk about web services they immediately think SOAP, which is unfortunate. When I think of a web service I think of system either a web site, a service running in your companies intranet, even a service running on the same machine, that I can send simple messages to and receive responses. The key is SIMPLE messages.

    Yes, that’s a rather loose definition. Mail servers probably fall into that definition, but to me that’s a web service. Well not very webby but a service nonetheless. So if SOAP is dirty what else can you use to create web services? Well there’s the venerable XML-RPC which is truly simple unlike SOAP. Most people don’t use XML-RPC because it doesn’t support “objects”, but then again those are overrated too. Seriously folks, you send a message you get back a structure. You really don’t need more than a hash or an array.

    Other reasons folks don’t like XML-RPC are that it lacks support for long data types (only supports integers), UTF-8 encoding (makes it hard to use for internationalization), and doesn’t have the concept of null. Those are all valid reasons, but a lot of times you don’t need that stuff in which case it’s still better to use XML-RPC rather than SOAP. The ease of development and ease of use from an api users point of view out weighs a lot of those things. XML-RPC is also trivial to understand. The specification is easy to digest: http://www.xmlrpc.com/spec. Compare that to the monstrosity of the SOAP spec. There still quite a few application that use XML-RPC as an API: smugmug.com, func, Red Hat Network, and flickr.

    If the limitations of XML-RPC truly are deal breakers for you, you’re probably wondering “I guess I’ll need to use SOAP!” Well, aside from needing soap to stay clean, you can actually be SOAP free in your web services and still use longs, nulls, etc. How? Use REST.

    There are two ways to implement REST services. There’s the purist way which uses the verbs of the HTTP protocol such as PUT, DELETE, POST, etc. The other more common way is to simply use the GET and POST. You supply your parameters on the URL including the method to be called and get back an XML response. A downside to REST is that the XML response is defined by the web service implementor, unlike XML-RPC or SOAP which have a defined structure. Nonetheless, REST has become very popular among web sites such as smugmug.com, amazon.com, and yahoo.com.

    REST and XML-RPC are not the only alternatives to SOAP, JSON is another. JSON is commonly done as a REST web service with the exception that the response is in JSON format. Most folks assume JSON is for use with JavaScript and web applications only, but that is not true. The thing I like most about JSON + REST is I get the ease of calling by a simple URL and get a well formatted easy to read response that supports nulls, UTF-8, and longs. You get none of the scum from SOAP, none of the limitations of XML-RPC, and a well understood format unlike the typical REST response.

    Ok the best way to “see” this is by looking at some code. Let’s look at XML-RPC first. Let’s assume we are calling the “smugmug.images.get” method at smugmug.com. Using an XML-RPC library for your language (java, python, perl).

    client = ServerProxy("http://smugmug.com/xmlrpc")
    session = client.loginWithPassword("uname", "pass")
    imgdata = client.smugmug.images.get(session, image_id)
    print imgdata['id']

    It’s that simple. The library did the parsing for me. imgdata will most likely be a hash. The library sent over something like this:

    <?xml version="1.0"?>
      <methodCall>
        <methodName>smugmug.images.get</methodName>
          <params>
            <param>
              <value><string>AXE0123</string>
            </param>
            <param>
              <value><i4>40</i4></value>
            </param>
          </params>
       </methodCall>

    The server responds with something like this:

    <?xml version="1.0"?>
    <methodResponse>
      <array>
        <data>
          <value><i4>1404</i4></value>
          <value><i4>1504</i4></value>
          <value><i4>1</i4></value>
        </data>
      </array>
    </methodResponse>

    Pretty easy to read isn’t it? But as you saw in the code you didn’t have to know how to read it.

    Let’s try the same thing with REST. There are no frameworks that handle REST directly as it’s just a simple HTTP GET or POST and an XML document response.
    url = "http://api.smugmug.com/services/api/rest/1.2.1/"
    call = url + "?method=smugmug.images.get&session=AXE0123&id=40"
    response = urllib.urlopen(call).read()
    # parse response XML into a dictionary
    imgdata = parse(response)

    In most cases you’ll probably have to write your own framework which isn’t really that hard, I did it. What gets sent out is a simple HTTP GET request to http://api.smugmug.com. What you get back is an XML document which you will need to parse.

    <?xml version="1.0" encoding="utf-8" ?>
    <rsp stat="ok">
     <method>smugmug.images.get</method>
     <Images>
      <Image id="17833"/>
      <Image id="17832"/>
     </Images>
    </rsp>

    Yes, it’s XML but so far it’s not too bad is it? You’re probably itching to see what JSON and SOAP can do huh? Well, heeeeere’s JSON! (that’s a Johnny Carson reference for you youngins).

    url = "http://api.smugmug.com/services/api/json/1.2.1/"
    call = url + "?method=smugmug.images.get&session=AXE0123&id=40"
    response = urllib.urlopen(call).read()
    # use builtin python library simplejson to read
    imgdata = simplejson.loads(response)

    The nice part about this is I don’t have to parse the response because libraries like simplejson do that for me. And I can easily make the calling code generic as I did in zmugjson.py. Again, the request is nothing more than a simple HTTP GET. The response is a nice JSON object.

    {
      "stat":"ok",
      "method":"smugmug.images.get",
      "Images":[
        {"id":222910058},
        {"id":222910121}
       ]
    }

    As you can see it is very easy to read and no nasty XML to deal with either. Best part is you could use this in an AJAX web ui with no need to create more than one API.
    There you have it, nice alternatives for creating web services without using SOAP.

    Oh you want to see the SOAP version of the above? hrm. smugmug was wise not to create a SOAP version of the API, but here is what it would probably look like. I’m warning you, you don’t want to see it. Ok here goes.

      import org.apache.axis.client.Call;
      import org.apache.axis.client.Service;
      import javax.xml.namespace.QName;
    
      public class TestClient {
        public static void main(String [] args) {
          try {
            String endpoint =
                "http://api.smugmug.com/services/api/soap/1.2.1/";
    
            Service service = new Service();
            Call call = (Call) service.createCall();
    
            call.setTargetEndpointAddress( new java.net.URL(endpoint) );
            call.setOperationName(new QName("http://soapinterop.org/", GetImagesFromSmugmug"));
    
            List<Integer> ret = (List<Integer>) call.invoke( new Object[] { "AXE0123", new Integer(40) } );
    
            System.out.println("Sent 'Hello!', got '" + ret.toString() + "'");
          } catch (Exception e) {
            System.err.println(e.toString());
          }
        }
      }

    Yes, I know it’s Java and not python, but that’s another problem with SOAP. The better libraries are written for Java not python, perl, etc.

    What would the SOAP request look like you ask? Probably like this:

    <SOAP-ENV:Envelope
      xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
      SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/>
       <SOAP-ENV:Body>
           <m:GetImageIds
             xmlns:m="Some-URI">
               <SessionId>AXE0123</SessionId>
               <id>40</id>
           </m:GetImageIds>
       </SOAP-ENV:Body>
    </SOAP-ENV:Envelope>

    Followed by a nasty response:

    <SOAP-ENV:Envelope
      xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
      SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/>
       <SOAP-ENV:Header>
           <t:Transaction
             xmlns:t="some-URI"
             xsi:type="xsd:int" mustUnderstand="1">
               5
           </t:Transaction>
       </SOAP-ENV:Header>
       <SOAP-ENV:Body>
           <m:GetImageIds
             xmlns:m="Some-URI">
               <Images>
                 <Id>1404</Id>
                 <Id>1504</Id>
                 <Id>1</Id>
               </Images>
           </m:GetImageIds>
       </SOAP-ENV:Body>
    </SOAP-ENV:Envelope>

    Seriously folks, it is truly possible to create web services and software as a service WITHOUT resorting to the evil that SOAP is. So the next time you plan on developing a web services api for your application consider XML-RPC, REST, and JSON.

    zmugfs 0.1 RELEASED!

    After two and half months of development, I would like to announce the first release of zmugfs (yes, the Halloween release) :). I was inspired to write this program when I got my wife, Elizabeth, to try out Fedora to import her digital pictures because “it just works in Fedora”. But uploading using the website was a bit lame, so I thought, “it would be cool if you could open up nautilus and see your pictures, then copy them to the folder to upload them”.

    That’s how zmugfs was born. What the heck is zmugfs? It’s a FUSE-based filesystem written entirely in python (my new favorite language) which connects to your account on smugmug.com using their JSON apis.

    DOWNLOAD zmugfs here!

    zmugfs requires zmugjson and fuse-python to work, actually requires a paid smugmug.com account to be useful.

    Installing on Fedora 7:

    • yum install fuse-python
    • download the f7 rpms of zmugfs and zmugjson
    • configure it with your username and password ($HOME/.zmugfs/zmugfsrc or /etc/zmugfs/zmugfs.conf)
    • /usr/bin/zmugfs <mount directory>
    • DONE!

    Installing on Fedora Core 6:

    • yum install fuse-python (from the extras repo)
    • download the rpms of zmugfs and zmugjson
    • configure it with your username and password ($HOME/.zmugfs/zmugfsrc or /etc/zmugfs/zmugfs.conf)
    • /usr/bin/zmugfs <mount directory>
    • DONE!

    Or if not a Fedora user, and you should be, try the tarballs.

    Here’s a sample config file:
    smugmug.username=zmuggy@somewhere.org
    smugmug.password=zmuggys_password
    # number of images to cache
    image.memory.cache=5
    # amount of image data to cache in megabytes
    # 50 ~ 25 images
    image.disk.cache=50

    disk cache for zmugfs

    Tonight I wrote the disk caching logic (well most of it). zmugfs will now look in the memory cache for the imgdata, if it doesn’t exist it looks in $HOME/.zmugfs/cache for the file. If it can’t find it there, it does the final attempt to read it from smugmug itself.

    I haven’t figured out how I’m going to calculate the size of the disk cache, that’s the last step to finishing that feature, otherwise it’ll use whatever disk space you have.

    That’s it for hacking tonight. Off to watch TV.

    zmugfs on Fedora 7

    I tested out zmugfs on Fedora 7 today and it completely blew up. The objecthook function being passed into simplejson.loads() behaves differently. On Fedora Core 6 (using python 2.4.4 and simplejson 1.3) my objecthook method is called once with the entire JSON response which I can parse and return a Tree object. On Fedora 7 (using python 2.5 and simplejson 1.7) it gets called 152 times with each call being a subset of the original response.

    I really would like it to run on Fedora Core 6, but it’s pretty dead since Fedora 7 has been out and Fedora 8 will be out soon enough.

    So will zeus abandon FC6 support for something new like F7 or F8? Or will zmugfs be updated to work on all Fedora’s? Stay tuned for more zmugfs on zeusville. Same zmug channel, same zmug time!

    <insert cheezy soap opera music here>

    zmugfs caching

    Tonight I finished a really lame in memory cache to store the image data. Basically it’s a dictionary keyed by the path (filename) and the values are another dictionary with image data and timestamp. There is a new configuration entry “image.memory.cache” where you list the number of images to cache in memory, defaults to 5 (about 10MB). When it hits the max size, it removes the oldest entry in the map.

    Next up is disk caching. I plan to cache some of the images on disk (in $HOME/.zmugfs/cache) I could probably also use /var/cache/zmugfs for a global install but for the first cut I’ll stick with the former. This adds yet another configuration entry “image.disk.cache” specified in megabytes. Yes, I know it’s lame that the value for the two caches are specified in different units. But the idea of dealing with file sizes in the memory caches versus number of entries was not appealing.

    After the disk caching, the only thing left is packaging and some setup scripts to make it easier to create your configuration file.

    getting close …

    We’re getting very close to releasing zmugfs 0.1, the read-only release. As you may recall on Monday I got zmugfs to show imgdata in nautilus.

    img data

    But I could only get it to work if I used the thumbnail sized imgdata. In the read() method I was logging into smugmug, reading the thumbnail imgdata, logging out of smugmug, getting the imgdata via http GET, and returning. Well, this is crazy because read gets called multiple times with different offsets. So I ended up implementing the open() and release() methods. In the open, I get the imgdata (original size) via the smugmug api and store the imgdata in an image cache (an in memory dictionary). In the read method, it now respects the offset and size attributes by reading from the image cache. Then in the release method, I delete the entry from the image cache. I know not much of a cache. 🙂 But it solved my problem. I’m putting the cache work off a bit, not sure how to approach that yet.

    Next thing I wanted to try all of the command line utils (at least the ones I’ve used before) to see how they react to zmugfs. I was pleasantly surprised with the result.

    command result comment
    basename SUCCESS  
    cat SUCCESS  
    chgrp FAILURE chgrp method not implemented
    chmod FAILURE chmod method not implemented
    chown FAILURE chown method not implemented
    cp from SUCCESS able to copy image from smugmug using zmugfs
    cp to FAILURE mknod not implemented
    file SUCCESS got appropriate file type
    ln -s SUCCESS softlink created
    ls SUCCESS listing appears correctly
    mkdir FAILURE mkdir method not implemented
    mv FAILURE copy part worked; unlink not implemented
    rm FAILURE unlink not implemented
    rmdir FAILURE rmdir not implemented
    touch FAILURE setattr not implemented

    From a readonly perspective, it is looking great. What’s left before a release you ask? Just a few things.

    • packaging
      • write spec file
      • figure out setup.py
    • design a logo
    • move code to zmugtools for branding purposes
    • work out image cache

    That’s it, once I finish those things zmugfs 0.1 will be released out into the wild! I’m really excited as this is my first ever open source project. I’ve submitted patches to a few things, but never anything this extensive.