Keeping Track of Sporadic Development Projects

I have been working on a personal project for about the last year, off and on.  It is written in Python using Django.  I started with Java and J2EE and JSF but I was spending a lot more time trying to figure out weird Java problems and not enough time getting things done, so I finally abandoned the Java code base and switched to Python.  The project itself is what I think is an interesting approach to knowledge management on the web, but that's a topic for another day.

The challenge I am running into is that I am the sole developer and it's something I pick up and work on furiously for a few weeks and then life gets busy and I don't look at the code for 2 months.  Unfortunately all the great ideas and state of mind that I had when I put the project down are long gone by the time I pick it up again.  So I needed to find a way to keep track of that state.

I started with a simple TODO.md file in my source tree, but that turned out not to be ideal.  I want to capture a lot more than just a simple TODO list.  I want to keep notes on things that I've tried, things that I want to try, technical approaches I'm thinking of, additional features I'd like to add, etc.  But I don't want an infinitely long text document that I have to parse every time I pick up the project.

Plus I want to keep track of temporality.  It's really useful to refer back to what I was thinking about six months ago, but I don't want that mixed in with the stuff from three months ago.  And I want to quickly look back at what I was thinking about the last time I picked up the project.  Which may be yesterday or a month ago.

So I have started a journal for the project in MacJournal.  I've been using MacJournal for some time for other purposes but I have not tried it as a "lab notebook".  So we'll see how that works out.  My initial thought was to use a blogging platform but I don't really want all my thoughts about this out in the ether (even if it were a private blog), since this is a personal project and not something I'm ready to share yet.  Hopefully a local MacJournal will give me the best of both worlds.  I've only written the inaugural entry so it's too soon to say if it is a good approach or not.  The one complaint I have so far is that MacJournal apparently does not support headings which is not something I've ever missed before but would be very handy for this usage.  For now I'm simply marking my headings bold, but it would be nice to actually have things stored semantically and with hierarchy.

If I find this to be wildly successful or a catastrophic failure I'll be sure to post a follow-up.

Really Impressed with SquareSpace

While this is not the most high traffic site on the Internet and a week or two of downtime would not really matter, I have been impressed with SquareSpace's ability to keep their infrastructure going during the Hurricane Sandy situation in New York.  SquareSpace has kept up a very transparent public blog on status.squarespace.com and while they originally expected to suffer a multi-day outage as a result of losing power and their colo building's basement being flooded, they have managed to keep everything online working closely with their data center provider Peer1.

It does warrant keeping in mind, as I wrote in my last update, that this is not actually highly available infrastructure.  If SquareSpace really wanted high availability they would need to have redundant systems in place provided by a data center in another geographic locale.  But that may be cost prohibitive for them to implement.  So definite bonus points for making the most of the resources they have!  And even more bonus points for the transparent and candid status updates.

"Cloud" infrastructure customers do still need to engineer availability

Today Amazon Web Services had another partial outage, impacting a number of their customer sites.  This has happened a few times and it is always met with declarations of the apocalypse by tech reporters.  I find that irresponsible and silly.  Overall AWS has a good availability record but any IT infrastructure can and will fail from time to time.  In the case of "cloud" providers like Amazon, these failures are amplified by the fact that they impact a significant number of independent companies relying on their services.

But that doesn't mean that companies shouldn't use cloud service providers.  It's still a fantastic value proposition, enabling small companies to compete with large incumbents without prohibitive capital requirements and larger companies to handle surge traffic without having to significantly overbuild their internal infrastructure.

Still, whether you are relying on your own infrastructure or someone else's, you have to engineer for availability if that's important to your business.  AllThingsD has an article on the outage with the incendiary title, "Amazon's Cloud Is Down Again, Taking Heroku With It".  But that's factually incorrect.  Amazon's "cloud" was not down.  There was a partial outage with EBS and several other services within the Northern Virginia Availability Zone.  Sites that are using the affected services within that availability zone are indeed impacted by the outage.  But an Amazon quote from the AllThingsD article states:

2:20 PM PDT We’ve now restored performance for about half of the volumes that experienced issues. Instances that were attached to these recovered volumes are recovering. We’re continuing to work on restoring availability and performance for the volumes that are still degraded.

We also want to add some detail around what customers using ELB may have experienced. Customers with ELBs running in only the affected Availability Zone may be experiencing elevated error rates and customers may not be able to create new ELBs in the affected Availability Zone. For customers with multi-AZ ELBs, traffic was shifted away from the affected Availability Zone early in this event and they should not be seeing impact at this time.

If Heroku was impacted by this outage it's a function of availability choices they made.  I have no experience with or relationship to Heroku so I don't know if their engineering choices were made based on cost, engineering complexity, or other factors, but the bottom line is that they are apparently choosing to host their service within a single AWS Availability Zone.  That is not a solution for high availability and it's unfair to blame the service provider when that provider offers solutions to mitigate these types of single Avaiability Zone failures.

Just because you're using someone else's infrastructure does not mean you don't need to make engineering choices to see maximum availability and it's irresponsible reporting to place all of the blame on the cloud provider.

A Few Minor Things

A few relatively minor updates for today.

Firstly, I think I am switching my primary browser from Firefox to Safari (at least on my personal Macs).  It's strange- I have used Firefox almost exclusively for about 10 years so it's bizzare to contemplate a change given that I am such a creature of habit.  I've always had Chrome and Safari and a few other browsers installed for cross-platform testing but my go-to has been Firefox for a long time.

I'm not terribly unhappy with Firefox, but their release cycle has become a froth which makes half my extensions broken most of the time.  I literally just opened Firefox to check the About box because it occured to me that I might be on the development channel and could be making an unfair characterization.  I am on the "release" channel and Firefox is now downloading an update.  I used Firefox earlier today.  Point made, I think.

Besides, given my investment in iOS I want to try out Safari on a full time basis to see if the iCloud tab syncing and the like will be useful for me.  I plan to use Safari at least until some time after iOS 6 comes out and if I don't find my life improved I may return to Firefox.  No big deal.

Second update, I signed up for an app.net charter account/membership.  I'm not really sure why.  I barely publish anything using the existing free services but I suppose I like the idea of there being paid services on the Internet which are accountable to their paying customers as opposed to having the incentive to probe and exploit their users to satisfy other companies' marketing whims.  Not that I'm against the online advertising business model; I just think there should be more than one prevailing way of doing things.  So I am supporting once such endeavour.  I have no idea if it will get traction.  They really need a different name though.

Third update, I looked at moving this site to the new SquareSpace 6 platform because it offers some things I like such as automatic mobile support.  I haven't switched because it's not currently possible to customize the layout of the templates on SS6 and that's too limiting for my taste.  Once they get their developer platform launched I will take a look again.

Fourth update, I'm kind of over social networking.  I've deactivated my social profiles, except for Twitter which I mostly use for consumption and LinkedIn which I mostly use for nothing at all (seriously, what is the point of LinkedIn? I really don't know).  I might reactive the other ones at some point.  I just realized that it had changed from being about tools to help me engage with my social connections to being a substitute or voyeurism in place of actual social interaction.  Plus I just wasn't finding much to relate to in the endless stream of baby and wedding pictures.  In the words of The Smiths, "the music that they constantly play says nothing to me about my life."

Fifth update, nobody reads this site so I don't worry much about my voice or my choice of topics, but actually this blog is more or less about technology.  So I think I might add a second tab at the top to talk about music.  Because I like music.  And that was the one thing that I did like to do on Facebook- share music videos from time to time.  So maybe I'll just do that here instead.  From time to time.