|« January 2005||March 2005 »|
I'm Ryan Lowe, a Software Engineering graduate living in Ottawa, Canada. I like agile software development and Ruby on Rails.
I write this blog in Canadian English and don't use a spell checker. Typos happen.
» Full-time Ruby on Rails freelancer
» Full-time with Rails since May 2005
» Former committer for RadRails (now Aptana)
» I also have a few Rails side-projects in development:
1. wheretogoinTO.com Toronto nightlife
2. Hey Heads Up! TODO list and sharing
3. Layered Genealogy family history research
4. foos for foosball scoring
5. fanconcert for music fans (on hold)
Hiring Rails developers? I can telecommute by the hour from Ottawa, Canada
»» Email: rails AT ryanlowe DOT ca
Now hosted on Hey! Heads Up -- check it out!
Derek Lowe's (Ryan's older brother) words at Ryan's funeral
[email protected] no more
Forging Email Headers: Good, Bad or Ugly?
Sarcastic Dictionary (Part 1 of Many)
Twisting Rails is Risky Business
Risky Business? My Take on Early Alphas
Whoa, it's August 2007
A Postscript to "Growth at the grassroots"
»» All Blog Posts
David Heinemeier Hansson
James Duncan Davidson
Signal vs. Noise
Amy Hoy: (24)slash7
Luis de la Rosa
# Week 01 Status Report
Development on AudioMan2 has officially started as of last week so here's a status update. It was mostly a relaxing weekend (my eyes needed a rest from computer monitors) but I did manage to get two plans down on paper. I tried to get the first iterations down to a minimum. Comments?
I still owe you guys that all-encompasing singing/dancing AudioMan2 architecture diagram. It's coming Real Soon Now.
The main priority this week is to get nightly builds going on my home machine (the link won't work until it's set up). That means I need to be able to build all of the AudioMan2 projects with Ant, run their test suites, generate code coverage reports and package it all in a nice ZIP file.
What's the deal with that? It keeps the AudioMan2 committers honest, which definitely includes myself. It doesn't matter who breaks the build -- having broken code just ruins it for everyone else. We have to be able to monitor the health of the AudioMan2 projects and the best way to do it is to make sure it compiles and passes all of the tests every night. For more on this stance, see Broken Window Theory. Later on we may use something even better: continuous integration.
Will I distribute nightly builds on the AudioMan website? I don't think that's such a great idea. Of course we can discuss that on the mailing list but I don't think nightly builds are suitable for general release. See my past comments about releasing unstable test software for more on that.
Will contributors have to write unit tests? No, but contributed patches have to have a certain level of quality. It will be a lot easier for us to confirm a patch works properly if it has unit tests. Why the code coverage tool? To identify weakly tested areas and give them more attention. I'm hoping that these quality tools will facilitate outside contributions by reducing our risk of accepting patches while lowering the barrier of contribution by making testing optional. Ultimately as project manager I'll be responsible for the overall quality of AudioMan2, which is why I need to monitor it closely.
On the Java development side: this week I'm also going to work on Durham's property declaration support using test-driven development as much as I can. Wish me luck. Hey Jim, pass that Kool-Aid!
# Iteration 1 Weekend
- a list of things to do in this iteration
User feedback of iteration x determines which things from the short and long term buckets go into iteration x+1.
To make things easy it's probably best if AudioMan2, Durham and jid3v2 all have synchronized release schedules with the same version numbers. It will be just like how the Rich Client Platform (RCP) is released with Eclipse and they have the same version numbers.
AudioMan2 has dependencies on Durham so either Durham will hold up implementation of features in AudioMan2 or AudioMan will drive Durham development and iteration priorities. It might even be better for a Durham feature to be implemented one iteration before it is needed, so it has a chance to stabilize.
How long is an iteration? Open source projects are less predictable here because you never know how much free time people will have to contribute. The early iterations will definitely be "when it's done" but later we could release more often and on a regular schedule. We don't want iterations to get too long or too big.
The other thing I want to do is make a nice diagram of how all of the AudioMan projects and components will work together. Hopefully that will help to answer questions like: where does the database (data persistence) fit in?
# Communicating and Collaborating on a Vision
The redesigned AudioMan website is all about communicating a vision of what the first version of AudioMan2 (and its supporting projects) will do. This vision will be refined over time by all of the people that work with and comment on AudioMan2.
The website will always communicate that vision it to whoever wants to read it instead of having something exclusively in my head. This kind of communication encourages collaboration.
My intention is to make the site easy to navigate and read. No fancy menus and not a lot of colour -- just content.
The drafts of specifications (see each project page) are still in the early stages but this is the best time to get your two cents in. The specs will probably keep changing well into the development stage too -- in true agile and open source hybrid style. As long as I keep the site updated with these changes, the vision is communicated.
Yes, the site is an artifact I have to maintain. The time it takes to do that will be saved from having to answer emails about the direction of the project. If the vision is up to date the direction will always be clear.
If you read the site and have questions, go ahead and give me a shout. Better yet, join the AudioMan Development mailing list and send us your comments.
# Object-File Persistence Layers and Durham
That's a darn good question -- maybe I shouldn't be. What if Hibernate could be extended to do file persistence instead of database persistence? That might save me a lot of work. What if there are already file persistence libraries out there that could do the job?
Clearly I should investigate more but that doesn't mean the Durham specification is a waste. It will provide a basis for requirements I need to look for in a third party library and give me an idea of the amount of work to get that library the way that I want it.
Same goes for the AudioMan2 functional spec because it depends on Durham and has requirements of its own.
# The Dilution of Beta
Forgive the rant, but it's interesting that there's so much attention being given to the term "beta" when it comes to software. This could be yet another great example of how insular technical people can be, not that we needed more examples.
The vast majority of people -- and it's vast -- had no idea what beta meant in the first place. None. So why is such a big deal being made now?
I'm not quite sure. Maybe it's because developers are really concerned about covering their asses. And what better way to cover your ass than by saying your released software is beta? Or by giving it some version number less than 1.0 and hiding that version number in the Help menu. Then if people run into catastrophic problems you can say "didn't you see you were using Product beta version 0.94? Didn't you read the EULA?" Yeah right.
It's all meaningless. People won't and do not care about version numbers or alpha, beta, gamma or even epsilon. If you release software people will use it. They'll inadvertently find 1000 different ways to break it. They'll pack it in their kid's lunches and use it as a hat on cold days. Maybe it will even do what it's supposed to do. Once it's out there though, it's out there.
If the software is free they might forgive you if it eats their homework or makes a mess on the rug. If they ever run into a catastrophic data munging bug in something they paid for they'll probably be less forgiving. Until they read the cover-your-ass EULA they didn't read in the first place they might even consider suing you.
Labels like beta on software are completely meaningless to almost everyone but a very small minority of keyboard-wielding geeks. Unless you can be absolutely positive your audience is only other geeks who know how to use a test release, care should be taken with each release you put out.
Once you make your software publicly available -- even a so-called alpha or beta -- people will grab it and break it. You have released it.
Like many other people, I've only pointed out the problem -- I haven't come up with a solution. Blaming users for being ignorant of industry jargon is not a solution. Blaming companies for messing up the meaning of industry jargon no one else cares about is not a solution either.
# Spec-tacularly Productive Weekend
This weekend I wrote an initial draft functional specification for AudioMan2 and updated the Durham spec I've been working on. If you're interested in the direction of these two projects now's best the time to get your two cents in. By posting these specs I'm welcoming any and all feedback either directly to me or the AudioMan-dev mailing list.
AudioMan2 might need a technical spec to complement this functional (end user level) spec so the architecture can be fleshed out. On the other hand AudioMan2 won't have much architecture per se because it's just going to use Durham, which is doing all of the heavy lifting when it comes to metadata and data persistence. As far as AudioMan2 is concerned everything is an object so I could see AudioMan2 being mostly UI code. Not that UI code is easy but it doesn't require a whole custom architecture.
Why all of this work up front? Well, it's a communication problem. I have to be able to maintain a centralized vision for the projects so everyone (users/developers/testers) is on the same page. If anyone wants to join the project and help out, they can read a few pages of spec and be in the loop faster. It also helps to get my thoughts organized and criticized by my peers. Specs like these are sort of like a business plans for software.
Just when you thought I was done I'm also working on a specification for a tool to generate Microsoft Windows installers for Eclipse Rich Client Platform (RCP) applications. It's about half-way done right now, but you can still check it out while it's in progress. The installers will be generated by the open source WiX toolset which is used internally by many Microsoft projects, runs on the .NET framework and licensed under the CPL.
There seems to be a division on Windows installers. Some people really like giving people a
I'm still looking for a decent comparision of MSI/WiX and NSIS tools like XtremeJ RCP Builder. I look at the bootloader-style
I'm calling the tool WiX-RCP for now but I could try to mix the WiX tool naming pattern (so far there's
By the way, I'm writing the WiX-RCP spec for fun to flesh out the idea and I have no plans to implement it yet. WiX-RCP won't necessarily be open source software.
# Learning WiX: Part Two
After more learning I think I need to backtrack even further with the WiX toolset and Windows Installer SDK. There are some fundamental decisions my team needs to make about how the product installation will work and it's something that should be decided before the first release. Once that first release is out the door from an installation perspective you're stuck with it.
Why? Uninstallation and upgrading. If a user installs version 14 of the product over version 13 what do you want to happen? Do you want to support patching to reduce upgrade installation sizes? Do you want to be able to gracefully uninstall the product? If so, what files do you want the uninstall to leave behind (ie. user data and application data) and where will they be? Do you want to be able to repair your product with the installer? These factors can affect how the installer is organized and written from day one.
Now's the time to get acquainted with some Windows Installer upgrade terminology. There are different options for updating and patching:
Given these options I think I'm only going to support installing, uninstalling and repairing and not any type of upgrading. Updating, upgrading and patching with Windows Installer seems like a lot more work and pain than it's worth at this point. If a user tries to install the product over an older version I can either prompt them to uninstall the older version before they install the newer version or maybe even do it automatically as an installer step. Their user and application data will be preserved, of course.
To support uninstallation I need to use managed Components with universally unique GUIDs (also called the component code). The uninstaller will use the Component GUIDs to figure out which components (and their resources) to remove, and according to component rules will reference count resources (MSDN seems to recommend having a reference count of no more than one).
Backwards compatibility is something I'm not too concerned about either. Each new version of our product will have resources that are almost certainly not backwards compatible (ie.
To make things really easy, each new version of the product can have new GUIDs for every Component so they aren't confused with old versions. This seems like the easiest way. The good news is that I can almost use tallow to automatically generate new Fragments with Components for each new version so I don't have to maintain the Fragments (and more importantly generate Component GUIDs) by hand. The only problem is that tallow doesn't generate Component GUIDs yet -- and therefore doesn't generate managed components that can be uninstalled -- so I made an RFE for that.
A very important aspect of installers is how they are tested and deployed. As I understand it even if I did support upgrading between file versions, a human tester would have to uninstall and reinstall the product to test daily builds because the daily builds within a version would contain the same Component GUIDs pointing to the same resource paths.
If the tester tried to install today's build over yesterday's build the installer would say: hey, the component GUID didn't change and the keyfile version number didn't change so don't update the component. Yeah, that seems bad for daily testing. I think it's safe to say that graceful upgrading is only between released final versions of the product. And that's all about keeping track of version numbers and Component GUIDs.
It's interesting that some places say the date is a factor in deciding which file to install and this diagram showing how it picks between two versioned files does not. Are dates in or out?
In a future blog post I'll try to go over some of the special issues I'm having creating a Windows installer with WiX for an Eclipse RCP app. I'm especially curious if a plugin's
# Learning WiX: Part One
Learning WiX is not just about learning the WiX toolset. It appears that in order to use the toolset properly you need to know quite a bit of background of setup on Microsoft Windows in general including the structure of Windows Setup MSI files.
This approach for WiX, while flexible and powerful for developers and setup crews that need it, doesn't help the little guy very much. Third party applications that create MSI setups seem to be too inflexible to do what I want or impractical to insert into an automatic build script.
So in order to make a proper Windows setup you need to put the time into it. The good side-effect of this is that you're more likely to take setup and project deployment seriously, which I get the impression often becomes a project afterthought.
As an aside, my bottom line when it comes to builds is: can I do it in one step?. I shouldn't have to assemble anything "by hand". Without going into a long rant about it the most important aspect of this is consistency of the build process which ultimately contributes to the overall quality of the final product.
A toolset like WiX seems like the perfect toolset to integrate into an automated build process done with another build tool like make or Ant. I'm in a bit of uncharted territory because I'm trying to use WiX with Ant (not NAnt) on Windows to make an MSI installer for an Eclipse Rich Client Platform (RCP) application written in Java and deployed in the way that RCP applications must be deployed. Most of the people I've seen in the blogosphere using WiX are either still using Visual Studio with C++ or have moved on to .NET.
I started with the documentation distributed with Wix and the well-written WiX Tutorial by Gábor DEÁK JAHN. Then I realised a) how much I didn't know about setup on Windows and b) how much this knowledge was required for Wix and was forced to backtrack.
The good news was that the author of WiX, Rob Mensching, has a great blog and has been posting since late 2003. I've been reading the archives from the beginning and they provide a very good starting point for people interested in learning about Microsoft Setup details and some insider background behind it and Wix.
The problem is that project I'm making an installer for is quite large (1500 files, 180 folders). The impression I get is that I'm going to have to manage each file and directory as explicit resources in the WiX .wxs file(s) so that it knows where all of the project files are and how they will be grouped/updated/versioned in their respective components. No wildcards like Ant.
Having done all of that work though, the generated MSI setup file will then not only install and uninstall but also sucessfully install the next version of the product right over an existing one using component GUIDs to track versions of components that have or haven't changed. That's how I understand it so far.
An interesting side effect of the one component to one directory guideline is that things get simpler component-wise if your directory structure is shallow. I don't have much control over directories because a large portion of the files and directories (1000 files and 100 directories) I'm packaging are coming either from the Eclipse project or are part of Sun's Java Runtime Environment (JRE) redestributable.
You might say "have you heard of tallow?". Yup, it produces a nice <Fragment> with 1500 <File>, 180 <Directory> and 180 <Component> tags in it.
The file generated from tallow is a good starting point but from what I understand I have to go into the generated file and give each Component a proper GUID so that its files can be properly uninstalled and I have to manage these GUIDs from product version to version, updating them when needed. For example, if I add a file to a directory the component has changed and it requires a new GUID. This is the distinction between managed and unmanaged Components. I can't see much use for the raw unmanaged Components generated by tallow in an installer if you want uninstall. What good is an installer if you can't uninstall?
I could definitely be way off base on my understanding of it but that seems like a lot of leg work -- not only for a setup engineer but also for individual developers if they are expected to update the WiX .wxs file if they add, remove or change files. The developer would also have to worry about the Component GUIDs if they make changes.
Once I generate that initial tallow file, that's it. From then on I have to manage that massive <Fragment> by hand without tallow's help. I can't use tallow again because it only generates unmanaged Components and I would have to reinsert all of the GUIDs again.
There could also be interesting installer problems if a file in a Component changes but someone doesn't change the Component's GUID in the Wix file. If that happens as far as Microsoft Setup is concerned, the Component didn't change and it won't be updated during the install. That seems to be a quality problem related to deployment waiting to happen.
If this is the case it seems like some sort of source control software (ie. CVS, Subversion) and WiX integration could be nice. The source control system could tell a WiX tool like tallow that a file in a certain Component had been added, removed or updated since the last release and the tool would automatically generate a new GUID for that Component for the next build.
At any rate it's good to know all of this background information just like it was good to know Ant really well. My practise with Ant on AudioMan really paid off and practise with WiX could pay off just as well in the long run.
# Suggestions for the Next AudioMan?
To help create a better functional specification for Durham I think it would be useful to create a functional spec for AudioMan 2 -- should it be renamed to avoid confusion? -- as well.
What features did you like in the original AudioMan? What would you like to see in the new one? Is there something missing in AudioMan 1 that you think hurt its success?
The sky's the limit for suggestions but think very high here on the feature level. That will give us a good starting point for an initial AudioMan draft functional spec.
BTW, the Durham spec is changing almost daily as refinements are made. I won't make announcements every time I change it because that would just be annoying. :) Thanks to everyone for their comments so far!
# First Draft of Durham Spec
I've posted the first draft of the Durham specification. This spec is the central vision for Durham to help get people on the same page.
Since Durham is a developer API the specification is written from the developer's perspective and is not a typical informal spec that might be written for an end user application like AudioMan. At the same time I think the philosophy of the spec should be to explain what Durham is going to do, not necessarily how it will do it. In that sense the spec will stay simple.
The spec is written purposely in a very Joel on Software type of way. It's informal, it has diagrams and is in a single skinny column in a large font.
How it looks is actually quite important because it indirectly encourages brevity. An informal spec like this should be complete but not overly verbose and it should be balanced on the vague/specific scale -- too specific and it will go stale faster but if it's too vague it's not useful enough to go by. Spec writing is a skill that's going to take a lot of practise!
This is a first draft. The features of Durham are not locked down and I definitely welcome feedback. This is your chance to stretch your software engineering legs and show people how smart you are. :)
What kinds of things does an application writer using Durham (like AudioMan) need? What kinds of things does a metadata type plugin (like id3v2 for MP3, or Microsoft Word DOC, or JPEG...) writer need? What kinds of things does a local cache plugin (like hooks to a MySQL or hsqldb database) writer need? What types of security problems could Durham have with files?
Durham will be pulled in all directions by these concerns. It doesn't hurt to think ahead a bit and get everything together in one place.
# WiX Plugin for Eclipse?
I've been learning WiX over the past week and so far have found nothing in terms of using it with Java or Eclipse. So here's a blog entry to be picked up by Google to gauge interest in the idea and let people post comments pointing to resources for Wix and Java combos.
WiX is a tool to generate setup files for Windows released as open source (CPL) by Microsoft last year. It is maintained/lead by Rob Mensching. The generated setup files are a relatively new format (MSI) that can be executed just like setup EXE files.
WiX uses XML markup to specify the installer settings and uses a compiler and linker-like system to get to the final MSI file. WiX runs on .NET and has dependencies on the Windows Setup APIs so it cannot be run on Mono (a Linux port of the .NET framework and Common Language Runtime (CLR)).
The good thing about WiX is that it's already being used internally by several large Microsoft teams (including Office) as well as externally by teams like MySQL. Even though it's still officially in beta this really looks good!
There are already people working on better RCP application export functionality for Eclipse. It would be even better if you could just export a Windows installer straight from Eclipse. Eclipse already knows all of the required files for each plugin.
Even though WiX is an open source tool there are a few minor issues I see integrating it with a development environment like Eclipse.
First off, WiX is Windows only and that makes a WiX Eclipse plugin Windows only. There are very few examples of Eclipse plugins that only work on one platform. To counter this there could be RCP installer exporters specifically for Mac OS X and Linux as well.
Secondly, WiX requires the .NET framework and Microsoft Setup APIs that cannot be deployed with the plugin and must be installed separately. This is a minor annoyance but certainly not a barrier. Eclipse already requires the separate installation of Java, of course.
Lastly, there are people that prefer to use Ant to build Java projects and do not necessarily setup their project plugins properly to export installers. They may also want to generate an installer for a timestamped/versioned Ant build after the fact, something completely independent from Eclipse. Ant builds are nice because a) they can be automated (ie. nightly) and b) they are more consistent and are less error-prone than "hand-building" an installer. I haven't seen any WiX integration with Ant either unfortunately.
There's already buzz about WiX integration in Visual Studio 2005. Someone -- possibly even the Eclipse team -- may want to try to beat Microsoft at their own game on this one.
A real Windows installer exporter from Eclipse would be excellent!
That's funny -- just before I published this I noticed this post from Ed Burnette pointing out XtremeJ RCP Builder which uses NSIS instead of WiX. I wonder how NSIS features and WiX features stack up against each other...
# Durham Spec in the Pipe
There's a Durham specification on the way, it's just not fully baked yet. I'm no writer, and it's hard to organize a two (at least) dimensional architecture into one literary dimension in a way that makes sense. I hope to get a draft done by the end of the week and it will be posted to the AudioMan-dev Google Group. Go ahead and join up, it's quick.
The main thing I had to get organized in my head was Durham's role and how it played out. Durham should not be an object persistence layer though it could easily tie into one. The main focus of Durham I think should be to offer object to file metadata synchronization.
This kind of automatic synchronization makes metadata look like objects to end user apps, and those objects are always the most up to date because Durham handles the file I/O automagically at lower layers. That -- and the metadata objects and API -- could be Durham's small and targetted focus. Seems more managable when I look at it that way.
Another spec will have to be written for AudioMan2 ... but that's later. :)
# Mozilla Releng and Openness
It's funny and yet so sad how software professionals are secretive; protecting methods of "modern" software development like they're trade secrets. Then people write books about ten year old methods which lets the cat out of the bag for everyone and the rest of the people using twenty year old development methods "catch up" to ten years ago. Repeat cycle.
The funny thing is that the methods are not revolutionary; it's how they are used in the right combinations and adjusted according to development conditions. Software development is controlled and constant microevolution -- it's a learning process, but what's learned should be applied and not forgotten as development progresses. The process that's used should allow breathing room for this learning and changing.
No one wants their competitor to exploit their competitive edge but public discussion about software development often ends up ten years behind in yawn territory. Then along came open source projects and blogs.
Take Asa Dotzler's blog for example, where he's starting to talk more about the details of Firefox development. Firefox is part of a larger group of Mozilla projects that use common code, so all of this common code has to be coordinated and the CVS branching done in such a way as to not interrupt active development in one project while another is preparing for a release.
A specific Mozilla example is how Firefox 1.0 used a very old but stable branch of Gecko, the Mozilla project's rendering engine used in The Mozilla Suite (Seamonkey), Firefox, Thunderbird, etc. When Firefox 1.0 was finally ready to release Gecko was already half a year of development down the road and not held up by the Firefox stabilization and testing effort. Now that Firefox 1.0 is out the door the Firefox team will catch up to a more current version Gecko with the 1.1 release and then probably be well behind the bleeding edge of Gecko again by the time Firefox 2.0 arrives.
The Mozilla project is truly a great public example of modern release engineering (releng) practices. The Linux kernel is actually very straightforward and uninteresting in comparison because kernel development uses multiple active and completely independent development trees instead of Mozilla's CVS branching, committer access, bug tracking, patching and fairly formal peer review process.
A lot can be learned from Mozilla releng if you're paying attention. This is how big companies are doing releng and when the projects get this big and interdependent, you have to. But even for small projects a good releng plan can make development smoother and the team more flexible. Why doesn't your company have a releng plan?