It’s Git to be magic!

To help me manage the source code for Pony Express I use the Git distributed version control system (DVCS). A DVCS is a tool used to record and mange the changes (revisions) made to source code over time.   It also allows many developers to work on the same project concurrently; each developer works on a ‘copy’ of the source files and the DCVS manages the merging of changes between the copies.

Prior to developing Pony Express I had not used Git a great deal and never for a real project.  I wanted to learn more about how to use it, so decided I would learn about it ‘on the job’, as it were.   Up until a few days ago I was muddling along with it quite well, I knew enough to be able to use the more basic commands, git commit, git log, git diff, git status, git branch and so on, but I knew I was missing something that would make it so much easier…

It was then that I discovered the power of git rebase!  This git command is very powerful and can do several very useful things, two of which I will discuss here.  The first function I will cover is squashing multiple commits into one large commit, the second function is its main use, which is to merge two branches that have both diverged since one was forked from the other (or ‘Forward-port local commits to the updated upstream head’ as the man page puts it).  I found the man page pretty impenetrable, which is why I found out about this command only recently, and part of the reason for writing about git rebase here.

Squashing multiple commits into one.

My first use for git rebase was to squash multiple commits into one larger commit.  I frequently came across a situation where I would have uncommitted changes in one branch and would need to switch to another branch to do a quick bug fix or something.  The problem here, is that uncommitted changes are carried across into the other branch when you do a git checkout <branch>; which is not what I want.  I had been using git stash which temporarily ‘stashes’ uncommitted changes allowing me to change branches and carry out the bug fix.  I would then recover the stashed changes using git stash apply when I came back to the original branch.

A much better solution would be to commit the changes before moving to the other branch, however, I don’t want lots of small partial commits swamping the log.  The answer to this is to use git rebase in interactive mode to squash the many partial commits into a larger commit.  Only the single larger commit would then show up in the log.  So how does this work?

Here we have commits A, B and C on the master branch, commit D on the branch B1 which we are on (indicated by *).  Commit D represents the early work on a new feature, which we are continuing with.  However, we wish to move to branch B2 to fix a small bug or something.

So we do git commit -a to commit the latest changes as commit E, and git checkout B2 to switch branches.

We can now work on branch B2 and fix the bug, committing the fix as commit F.

We need to finish the work on branch B1, so we do a git checkout B1, complete our changes and commit the final work as commit G.

We now have 3 commits, D, E and G that we would like to squash into one commit with all the changes for the new feature.  To do this we call git rebase with the -i option to start it in interactive mode, we also need to tell git rebase the last commit we want to stay as is, for us this is Commit C.  Commit C can be referenced  as ‘HEAD~3’ (likewise, D is HEAD~2, E is HEAD~1 and G is HEAD) so the command git rebase -i HEAD~3 starts up an editor for us and lists the commits D, E and G together with their log messages like so:

pick D Begun implimentation of cool new feature.
pick E Halfway finished cool new feature.
pick G Finished cool new feature.

#Rebase C..G onto C
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

As you can see it is also possible to edit and reword commits, but I won’t go into that here. We want to squash commits D, E and G together so we need to replace the word ‘pick’ from commits E and G with ‘squash’, to get this:

pick D Begun implimentation of cool new feature.
squash E Halfway finished cool new feature.
squash G Finished cool new feature.

#Rebase C..G onto C
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

We then exit the editor and save the changes, git then starts the editor up again to allow us to modify the log message for the single large commit, which starts out as an amalgamation of the individual commit log messages, like this:

# This is a combination of 3 commits.
# The first commit's message is:
Begun implimentation of cool new feature.

# This is the 2nd commit message:
Halfway finished cool new feature.

# This is the 3rd commit message:
Finished cool new feature.

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# Currently on branch B1.
# Changes to be committed:
# (use "git reset HEAD ..." to unstage)
[snip..]

You modify this message to say whatever you want about the new big commit and then exit and save as before.

Then as quick as a flash, git merges all the changes made in those three commits into one commit and then attaches that to the HEAD of the branch B1 to give this:

Commits D, E and G have merged into new commit H.

As you can see, this makes it very easy to make many commits on a branch and then to parcel them up when necessary, into more meaningful larger commits.  This enables you to change branches with ease, and to work on many areas of code at the same time, without those changes creating utter chaos in your code.  I know this certainly aids my productivity and keeps my confusion levels down 🙂

Now that we have looked at one way to use git rebase, let us turn our attention to the main function of the command.

Forward-port local commits to the updated upstream head.

What this gobbledygook means is that git rebase can merge commits from one branch into another branch behind any commits you have made on the branch you are working with. This is made clearer with a diagram:

Here is the same commit tree as used earlier after some more work has gone on; commit F (a bug fix) was merged with the master branch and another commit, I (another developers cool new feature), has also been merged in.  You are working on branch B1 and have made commits H, J and K (your new killer feature) and you would like to have the changes in the master branch in your branch too.   Using git merge in this situation would cause the changes made in commits I and F to be added to the HEAD of branch B1, ie: after your commits in history. This then gets messy when it comes to pushing your changes into the master branch at a later date. A simple git rebase master is the magical incantation you want to use in this situation. This rebases the branch B1 so that it diverges from branch master after commit I, instead of commit C as it did before.

The way it does this is to ‘undo’ all the changes in commits H, J and K on branch B1 and saves them in a temporary area.  It then resets B1 to be the same as master and then applies commits H, J and K to the new HEAD.  Now this process isn’t always strait-forward; it is possible that there may be conflicts when commits H, J and K are applied.  If this is the case then you will need to resolve those conflicts manually in the usual manner and then run git rebase --continue to complete the rebase.

Using git rebase like this means that when the time comes to push your changes into the master branch, they should go in with the minimum of conflicts, as your branch has been kept up to date with master.

Well that is it for this introduction into the power and magic of git rebase.  I hope that it is of use to someone out there grappling with the intricacies of git.

Advertisements
Posted in Development, git | Leave a comment

So little time..

Wow, I can’t believe that it has been over six months since my last post! The trouble is, when I get free time I would rather hack on Pony Express, than write about hacking on Pony Express.. 🙂 However, I hope in the future to find some more time to write some more posts about Pony Express and Android App development. In the meantime, I hope to use my blog to highlight some other things I come across that I find useful or interesting. Watch this space for something interesting coming soon.. !

Posted in Android, Blogging, Development, Pony Express | Leave a comment

Pony Express Tutorial Part 2a: Are You Being Served?

If you have read part 1 of this tutorial series then you will know that I have been developing an Android app called Pony Express, and that I am using it to demonstrate various aspects of the Android API, and their integration into a complete app.  Pony Express is a pod-catcher and as such needs to be able to download podcasts.  Part 1 dealt with parsing an RSS feed in order to determine the URLs from which the podcast episodes can be downloaded.  This part will introduce the Android Service class, and use it to download and save episodes of the Linux Outlaws podcast to the SD Card of your Android device.

Podcasts down the pipe.

Downloading podcasts is not an instantaneous process; it takes some time.  In addition, it does not require any user input once started.  These two things make the use of an Android service ideal for this task.  Services are designed to run in the background until they either complete, or are explicitly stopped.  However, they do not by default run in a separate process, or in a separate thread from the main application.  This means that if you use them for CPU-intensive or blocking tasks (such as downloading), they need to be run in a separate thread to avoid making the user interface unresponsive.  Fortunately, Android has a sub-class of Service, IntentService, that sets up a worker thread by itself; thereby saving us the hassle of doing it.  So, how do we use the IntentService class to download the podcasts?

Service!

IntentServices are very simple to use.  We simply create a new class extending IntentService, and override the abstract method onHandleIntent() with the code we want to run in the worker thread.

package org.sixgun.ponyexpress.service;

import android.app.IntentService;
/**
*Service that handles the downloading and saving of podcasts.
*It queues multiple calls to startService from the calling Activity
*and does them one at a time in a worker thread.  When it has finished all
* its work it stops itself.
*/

public class Downloader extends IntentService {
    public Downloader() {
        //We need to call the Constructor of the superclass with a name for our class.
        //This is required but only useful for debugging
        super("Downloader");
    }
    @Override
    protected void onHandleIntent(Intent intent) {
    //Code to do the downloading goes here.
    }
}

The first thing our Downloader service is going to need is the URL from which to download the podcast.  In Part 1 of this series we created an RSS parser class that got the URLs for each episode from the RSS feed.  So, how do we get each of these to our Downloader?  To answer that we are going to have to look at how services are started.

‘First Serve.’

Services, including IntentServices, can be started in two ways.  The methods startService() and bindService() can both start a service, but how do we specify which Service to start?  Well, both of these methods require something called an Intent to be passed to them as a parameter.  An Intent can either be Explicit or Implicit: Explicit intents designate the target Service by name.  Implicit intents describe the Action required, or data type to handle and Android determines which is the best Service (or Activity) to start based on that.  Here, we will be using the simpler Explicit intent since we know that we want to start our Downloader service.

Intents can also hold data in a Bundle called ‘extras’ and it is in this ‘extras’ bundle that we will put the URL of the podcast episode to download.  So, in order to start our Downloader service from the main Pony Express app, we create a new intent designating the Downloader class (and the Context(this), don’t worry about Context here, we’ll cover it next time), add the episode URL to the Extras bundle, and then call startService(intent) with the intent we created.

Intent intent = new Intent(this,Downloader.class);
intent.putExtra("URL", "http://media.libsyn.com/media/linuxoutlaws/linuxoutlaws159.ogg");
startService(intent);

The Android system will then start an instance of Downloader for us and pass the Intent to onHandleIntent(Intent intent), which we can then access like this:


private static final String TAG = "PonyExpress Downloader";
private URL mUrl;
@Override
protected void onHandleIntent(Intent intent) {
		Bundle data = intent.getExtras();
		mUrl = getURL(data.getString("URL"));
                //Rest of the code to handle the download goes here.
}
/**
	 * Parse the url string to a URL type.
	 * @param _url string from the Intent.
	 * @return URL object.
*/
private URL getURL(String _url) {
	URL url;
	try {
		url = new URL(_url);
	} catch (MalformedURLException e) {
		Log.e(TAG, "Episode URL badly formed.", e);
		return null;
	}		return url;
}

Now that the Downloader has the URL we need to do one more thing before commencing the download.  We need to prepare the path where we will store the episode on the device’s SD Card.

‘The Path to Enlightenment…’

Files saved on the SD Card of an Android device can either be stored in an application specific path, or in a one of several ‘themed’ directories such as Music, Podcasts, Ringtones, Pictures etc..  Storage in an application specific directory means that the files can be made private to the application, and, in Android 2.2, they will be deleted when the application is deleted.  Files in the themed directories are available to many applications and are not deleted automatically.  For Pony Express I decided to use an application specific directory and so that is what I will demonstrate here.  The Android documentation specifies that this Pony Express directory should have the path:

/Android/data/<package_name>/files/

where <package_name> is org.sixgun.ponyexpress for Pony Express.  To create this path we first need to get a File instance that represents the root of the SD Card, like this.

File root = Environment.getExternalStorageDirectory();

Now, we create a new File instance representing the desired Podcast path and create it.

File path = new File(root, "/Android/data/org.sixgun.ponyexpress/files/");
path.mkdirs();

Next we need a filename with which to save the episode.  The URL of each episode has the filename at the end, so with some URL and String manipulation we can split the end off and use that as the filename like so.

final String filename_path = mUrl.getFile();
final String filename = filename_path.substring(filename_path.lastIndexOf('/'));

The call to getFile() with our example URL returns “linuxoutlaws/linuxoutlaws159.ogg”, so we need to split the string at the ‘/’ to give us just the filename.

To write to this file we have to create a FileOutputStream object.

try {
    FileOutputStream outFile = new FileOutputStream(new File(path,filename));
} catch (FileNotFoundException e) {
    Log.e(TAG, "Cannot open FileOutputStream for writing.",e);
}

We are now ready to open a connection to the server hosting the podcast and to download the episode.

Energize!

To download the file we open an InputStream with the URL and read the bytes from that stream into a buffer.  The buffer contents are then written to the FileOutputStream we created.

try {
    final InputStream inFile = mUrl.openStream();
} catch (IOException e) {
    throw new RuntimeException(e);
}
byte[] buffer = new byte[1024];
int size = 0;
try {
    while ((size = inFile.read(buffer)) > 0 ) {
        outFile.write(buffer,0, size);
    }
} catch (IOException e) {
    Log.e(TAG, "Error reading/writing to file.", e);
}

So there we have it.  The episode has now been downloaded and saved to the SD Card.  If the Downloader IntentService hasn’t been sent anymore intents to handle it will close its worker thread and finish.

That is it for Part 2a, we have touched on Services, and IntentServices in particular, introduced Intents, covered saving files to the SD Card, and downloading files from the Internet.  Of course, this Downloader is quite basic; in particular it doesn’t notify the user about what it is doing, and it doesn’t check that there is a working internet connection before trying to download the podcast.  These things and more I will cover in part 2b, where I will add a few bells and whistles to the Downloader and make it more resilient.

Posted in Android, Development, Pony Express | 4 Comments

Pony Express Tutorial Part 1: You gotta start somewhere..

So, where to start. Well, if you haven’t read my last post where I explained the rationale for this series of posts then go back and read it ;), as for me, well, for a change I thought I wouldn’t start with ‘Hello World!’  If you really want to, you can find it in the Android developer docs, so I won’t repeat it here.  Instead let’s start with the big picture and then break it down.  What does the app need to do, and what functions and assets does it need in order to do it?  The app I am developing, Pony Express, needs to do three things: download podcast episodes, play them and sent status updates to identi.ca.   In the various parts of this series we will look at each of these, but let’s get started with what’s required to download the podcasts.

Catching those ‘casts…

In order to download podcast episodes we need to do three things:

  • Find out where we can download the podcast episodes from.
  • Download the episodes.
  • Record somewhere that the episodes have been downloaded, so we don’t download them again next time.

So, this part of the series will focus on how we determine where we download the podcast episodes from.

‘They seek him here, they seek him there…’

Podcasts are usually published via an RSS feed from a website such as feedburner.  An RSS feed is written in XML, and consists of a number of elements.  Each element holds a piece of information, as can be seen in the example below.

<rss version="2.0">
    <item>
       <title>Linux Outlaws 155 - There is No Fabian Stable</title>
        <pubDate>Fri, 11 Jun 2010 14:13:46 +0200</pubDate>
        <enclosure url="http://media.libsyn.com/media/linuxoutlaws/linuxoutlaws155.ogg" fileSize="47403238" type="application/ogg">
    </item>
    <item>
        <title>.......</title>
        <pubDate>...... </pubDate>
        <enclosure .................>
    <\item>;
</rss>

So, this feed has rss, item, title, pubDate and enclosure elements .  The title, pubDate and enclosure elements are between the opening and closing tags of the item elements, which defines them as child elements of each item element.  The item elements are likewise child elements of the rss element.  So, an RSS feed for a podcast would usually contain many item elements, one for each episode; each with its own title, pubDate, and enclosure child elements.  To download each podcast we need to know the URL that it can be downloaded from, which in this example is given by the url ‘attribute’ of the enclosure element.  Now, this is all very nice, but how can we access this information with our app?

What we need to use is one of the the Android XML libraries parse() methods.  These methods take an XML source and a SAX (Simple API for XML) Content Handler.  The XML source can be a String object, the XML from a Reader, or the XML from an InputStream.  A SAX Content Handler instance defines which elements we are interested in, and what we want to do when we encounter them.  Leaving aside the SAX Content Handler for a moment, the XML we want to parse is an RSS feed served by a website.  We can use the Java net library to open a connection to the site, and to return an InputStream object, with a class like this:


package org.sixgun.ponyexpress.util;

import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;

public abstract class BaseFeedParser {
    final URL mFeedUrl;
    /** Constuctor - Takes the String url of the feed to be passed.
    * @param feedurl
    */
    protected BaseFeedParser(String feedUrl){
        try {
             mFeedUrl = new URL(feedUrl);
        } catch (MalformedURLException e) {
            throw new RuntimeException(e);
        }
    }
    /**
    * Opens a connection to mFeedUrl
    * @return an InputStream from the mFeedUrl
    */
    protected InputStream getInputStream() {
        try {
            return mFeedUrl.openConnection().getInputStream();
        } catch (IOException e) {
            return null;
        }
    }
}

Note that this is an abstract class, this is because we may want to use this functionality in different sub classes, each handling different XML streams.  In order to use this class to parse the RSS feed, we need to subclass it and create an EpisodeFeedParser class.

package org.sixgun.ponyexpress.util;

import java.util.ArrayList;
import java.util.List;
import java.io.InputStream;

import org.sixgun.ponyexpress.Episode;

public class EpisodeFeedParser extends BaseFeedParser {
    public List<Episode> parse() {
        final Episode episode = new Episode();
        final List<Episode> episodes = new ArrayList<Episode>();

        InputStream istream = getInputStream();
        //We will add the code to set up the parsing next.
        return episodes;
    }
}

The EpisodeFeedParser class has only one method of its own, parse(), which at the moment only calls the getInputStream() method of its super class, BaseFeedParser, and then returns an empty List of Episodes.  The Episode type is a regular Java object which I won’t define here, for the sake of simplicity.  It has fields for each episodes url, title and publication date etc.. as well as getters and setters for them.  We now need to add the code that will allow us to create a SAX Content Handler and allow us to parse the RSS feed.

Play that SAX!

To create a SAX Content Handler, you first need to describe the structure of the XML you are interested in, using the RootElement and Element classes of the Android SAX library.  The RootElement is a single element that all of the other elements that you want to access, are child elements of.  In the example RSS feed above, and in most RSS feeds, the rss element is the root element of all the other elements, so we can define it as our RootElement like so:


RootElement root = new RootElement("rss");

We can then declare our interest in each of the item and enclosure child elements using the requireChild() method of the RootElement class :


Element item = root.requireChild("item");
Element enclosure = item.requireChild("enclosure");

Next we need to set some ‘listeners’ for each of the elements we want to catch.  In Android, listeners are callback functions that are called when particular events occur.  Here, we can set listeners for the start or the end of each element and they will be called when the XML is parsed.  We use a StartElementListener when we want to read the attributes of an element, so this is the type we will use to read the podcast URL from the enclosure element.

enclosure.setStartElementListener(new StartElementListener() {
    @Override
    public void start(Attributes attributes) {
        String episodeUrl = attributes.getValue("","url");
        episode.setLink(url);
    }
});

In this code, the start(attributes) method is called when the start of the enclosure element is parsed.  The first parameter of the getValue method is the namespace URI, which we don’t need here so we pass in an empty string.  The second parameter is the attribute we want the value of, in this case “url”.

We use an EndTextElementListener when we want to read the text held by an element, such as the podcast title in the title element, or the publication date in the pubDate element.

item.getChild("title").setEndTextElementListener(new EndTextElementListener() {
    @Override
    public void end(String body) {
        episode.setTitle(body);
    }
});

Here the end(String body) method is called when the end of a title element is parsed, and the body of the title element is passed in to be stored in an episode instance.

The EndElementListener’s end() method is called when the end of a particular element is reached; we will use it with the item elements, to store the episode instance we have been using in an array (episodes), before moving onto the next episode/item.

item.setEndElementListener(new EndElementListener() {
    @Override
    public void end() {
        episodes.add(episode);
    }
});

We now have everything in place and are ready to create the SAX Content Handler and pass it and the RSS InputStream in to the XML.parse method like this.

Xml.parse(istream, Xml.Encoding.UTF-8, root.getContentHandler());

Now if we put all that code into the EpisodeFeed Parser class we started earlier, we get the finished class shown below.

package org.sixgun.ponyexpress.util;

import java.util.ArrayList;
import java.util.List;
import java.io.InputStream;

import org.sixgun.ponyexpress.Episode;

import android.sax.Element;
import android.sax.EndElementListener;
import android.sax.EndTextElementListener;
import android.sax.RootElement;
import android.sax.StartElementListener;
import android.util.Xml;

public class EpisodeFeedParser extends BaseFeedParser {
    public List<Episode> parse() {
        final Episode episode = new Episode();
        final List<Episode> episodes = new ArrayList<Episode>();

        InputStream istream = getInputStream();

        RootElement root = new RootElement("rss");
        Element item = root.requireChild("item");
        Element enclosure = item.requireChild("enclosure");

        enclosure.setStartElementListener(new StartElementListener() {
            @Override
            public void start(Attributes attributes) {
                String episodeUrl = attributes.getValue("","url");
                episode.setLink(url);
            }
        });

        item.getChild("title").setEndTextElementListener(new EndTextElementListener() {
            @Override
            public void end(String body) {
               episode.setTitle(body);
            }
        });

        item.setEndElementListener(new EndElementListener() {
            @Override
            public void end() {
                episodes.add(episode));
            }
        });

        try {
            Xml.parse(istream,XmlEncoding.UTF-8, root.getContentHandler());
        } catch (Exception e) {
            throw new RuntimeException(e);
        }

        return episodes;
}
}

We can now create an instance of EpisodeFeedParser from within Pony Express; the RSS feed we give it will be parsed and a List of Episodes, each with their titles and all important URL’s, stored in the appropriate fields, will be returned.

EpisodeFeedParser parser = new EpisodeFeedParser("http://feeds.feedburner.com/linuxoutlaws-ogg");
List<Episode> episodes = parser.parse();

So, that is it for part 1 of this series.  We have created the classes that will enable Pony Express to determine the URL’s from which to download the podcasts.  Next time, we will look at utilising Android Services to download the episodes to our device.

In the meantime, if you want to check out Pony Express further take a look at the code hosted on gitorious.

Posted in Android, Development, Pony Express | 6 Comments

The day I finally got a blog..

Thursday, the 8th of July, 2010, the day I finally caught up with the rest of the modern world and started writing a blog.  It is not that I am a technophobe, far from it, I just didn’t ‘get’ blogging and couldn’t see what I could possibly write about that anyone else would be interested in.  But recently, I have found myself searching the web for various bits of information and finding some of the answers in blogs.  This coincided with me starting to work on a new project and I thought it would be useful to give some of the things I’m learning back to the web, in the hope of helping others undertaking similar projects.

So what is this project that I have started?  Well I am a fan of the great Linux podcast Linux Outlaws.  The two presenters of the show, Dan Lynch and Fabian Scherschel came up with the idea of an Android app for Linux Outlaws that would allow the user to download and playback the show, and to comment on episodes via the identi.ca social network using integrated commenting.  The name of this app was to be Pony Express.  I had recently bought an Android phone, the HTC Legend, and was interested in trying my hand at some app development, so I thought this would be a good project to get involved in.

Now, that was a couple of months ago, and the project is progressing well.  During the development I found myself making extensive use of Google to solve problems and to learn more about the Android API.  Many of the hits returned by Google were for blogs featuring various tutorials and hints.   I found that many of the tutorials were quite limited, in that they detailed how to use one aspect of the API (for example: the media player) but did not really expand on how it might fit into a larger application.  So I thought it might be useful to others if I could create a series of posts about how I would go about Android app development, if, with the benefit of hindsight I was to write Pony Express again.  I aim to take an aspect of development and/or the Android API in each part of the series, and use the Pony Express code to demonstrate their use and function and how they fit into the larger application.  Part 1, “You Gotta Start Somewhere..” is here.

Posted in Android, Blogging, Development, Linux, Pony Express | Leave a comment