Stop writing for loops in Javascript

You have an array; you want to find an element in the array. What do you do?

Most developers go with the old procedural for loop:

var items = [
   { Name = "Matthew", Id = 1 },
   { Name = "Mark", Id = 2 },
   { Name = "Luke", Id = 3 },
   { Name = "John", Id = 4 }
];

// Find the item with Id === 3
var matchingItem = null;
for (i = 0; i < items.length; i++) {
     if (items.Id === 3) {
       matchingItem = item;
       break;
     }
}

  // Do something with matchingItem
}

What’s wrong with this?

It’s the classic procedural programming affliction: over-specification. It painfully specifies all the steps the program must take to reach the desired state.

All you want is the item with an Id of 3, but instead you create a new loop variable, increment that variable each loop, use special loop iteration language syntax, index into the array, check if it matches your condition, store the match in a mutable variable.

The how obscures the what.

Thank you sir, may I have another.

This obfuscates the intent of your code via over-specificying pageantry.

It’s also error prone. The above code omitted the var declaration for the loop variable i, (did you catch that?), thus adding it to the global scope (oops!). Developers often forget to break; out of the for loop when a match is found, resulting in superfluous loop iterations at best, unintended behavior at worst.

Can we do better?

How about .filter?

Today’s JavaScript (1.5+) includes the built-in .filter function:

var items = [
     { Name = "Matthew", Id = 1 },
     { Name = "Mark", Id = 2 },
     { Name = "Luke", Id = 3 },
     { Name = "John", Id = 4 }
  ];

  // Find the item with Id === 3
  var matchingItem = items.filter(function(item) {
     return item.Id === 3;
})[0];

  // Do something with matchingItem...

.filter is a higher-order function that returns all elements in the array that match your predicate. The above code calls .filter(…) and then gets the first matching item.

This is better, but still not right.

What’s wrong with this? Like a for loop that forgot to break; out of the loop after finding a match, .filter will iterate over your entire array, even if it already found a match. This will result in superfluous iterations. And if your code doesn’t account for multiple items matching your condition, you’ll get unexpected behavior.

Solution

There’s no way to do this efficiently or succinctly using any of the built-in JavaScript array functions.

Because of this, most developers just do the ugly one, or the inefficient one.

A proper solution would be a function added to the Array prototype chain. In lieu of the browser vendors doing that, we can do it like this:

if (!Array.prototype.first)
{
   Array.prototype.first = function(predicate)
   {
     "use strict";   
     if (this == null)
       throw new TypeError();      
     if (typeof predicate != "function")
       throw new TypeError(); 
     
     for (var i = 0; i < this.length; i++) {
       if (predicate(this[i])) {
         return match;
       }
     }      
     return null;
   }
}

Here, we’ve defined a .first method on the Array prototype. It uses the efficient-but-ugly for loop. But instead of coding up this for loop every time, we can now just call our higher-order function:

var items = [
   { Name = "Matthew", Id = 1 },
   { Name = "Mark", Id = 2 },
   { Name = "Luke", Id = 3 },
   { Name = "John", Id = 4 }
];

// Find the item with Id === 3
var matching = items.first(function(i) { return i.Id === 3 });

// Do something with match...

Much cleaner, syntactically similar to .filter, but without the superfluous iterations.

JavaScript has no lambda syntax (yet), so it’s not as clear as it could be. Ideally, we’d write it more succinct, something like:

var matchingItem = items.first(item => item.Id === 3);

We’ll have to wait for the proposed arrow function syntax for the next version of JavaScript before we can get that succinct.

In the meantime, Array.prototype.first is a nice polyfill with decent syntax, allowing your code to focus on what, rather than how.

TypeScript, CoffeeScript, and the state of web development in late 2012

Summary: The problems facing web development today. Microsoft’s new TypeScript language, what problems it addresses, and what it leaves wanting.


Earlier this year, I predicted Microsoft would be utilizing their still-baking Roslyn compiler service to transform C# into targets besides IL, such as JavaScript. That prediction didn’t come true, but I was close: Microsoft is indeed getting into the compile-to-JavaScript games, but they’re doing so with a brand new language announced today, TypeScript.

JavaScript is a messy, powerful language.

The whole compile-to-JavaScript trend addresses a real problem in web development today: JavaScript just isn’t a great language for big web apps. Some argue it’s not a great language for anything besides quick DOM + event handler glue. And honestly, they kind of have a point.

Half baked, escaped the lab too early, leaving us silly things like implicit conversions for equality comparison. Quick, what’s the result of this?

foo = 0 == (100 + 23 === “123”)

If you guessed the boolean true, you’re right (I think!) but it shows kind of the trouble you get into when languages try to do too much for the developer.

But silly syntax puzzles aren’t the real reason JavaScript is bad for big apps.

No, it’s for a number of more deeply-seated reasons the difficulties arise when coding a large JS app with a team of developers:

  • The automatic, often accidental, inclusion in global scope. (The above snippet adds foo to the window object, woops!) This results in code littering the global namespace at best, and overwriting other code’s functionality at worst.
  • No modules or namespaces for code organization. In addition to contributing to the above problem, a lack of built-in modules means we have to resort to 3rd party libraries for dependency detection and loading. Often, developers will just take the easy route and load everything, resulting in web apps that load slowly and use excessive memory.
  • A 10-ways-to-do-this inheritance pattern that kinda sorta looks like classical inheritance but is really prototypal. Some codebases standardize on a 3rd party library (such as MooTools or Prototype), while other codebases become the Wild West of inheritance problems.
  • Without any types, refactoring and symbol analysis becomes painful. For example, want to find all callers of a function? Have fun with CTRL+F, and weed through who’s calling your function and who’s calling a function of the same name. Want to rename a property? CTRL+H and pray that no other code file has a property of the same name. And so on.

Plus other uglies like optional semicolons to end lines, resulting in production-level frameworks being filled with, in the words of JavaScript creator Douglas Crockford, “insanely stupid code.”

But one thing JavaScript is, is capable. It’s a functional, prototypal, object-oriented language. General purpose.

So rather than wait for JavaScript to slowly evolve through standards bodies and bickering, we build our own fast-evolving languages that compile down to JavaScript.

CoffeeScript: A (slightly) better JavaScript

At the time of this writing, the compiles-to-JavaScript language de jour is CoffeeScript: a superset of JavaScript that adds things like classes, and removes things like semicolons, parenthesis, and other staples of the C family of languages.

But CoffeeScript still lacks a proper compiler. Oh, sure, there’s the CoffeeScript compiler, but it’s not really a compiler. It can’t tell you if you’re passing the wrong type into a function. It can’t tell you if foo.Blah will error at runtime, because there’s no symbol analysis.

CoffeeScript also suffers from its straddle-the-fence stance on compatibility with JavaScript. Want to include parens? OK, valid! Want to exclude parens? OK, valid! While it sounds nice in theory, in practice you end up with an inconsistent codebase, and an awkward rule set about when optional syntax isn’t really optional.

And though CoffeeScript tried its hand at list comprehensions, it falls far short of what Python and C#/LINQ devs have been enjoying for years.

To make matters worse, there’s still no real standard library for JavaScript. Just a mash of tiny libraries that do things differently ($.map? or array.map?)

And finally, we still have terrible debugging support. Got Coffee? OK! How do I set a breakpoint on this CoffeeScript line? Silly developer, you can’t! You have to look at your Coffee, figure out which .js files were emitted, find those in your favorite web browser developer debugger tool, and break into that. Grahgghhlhlhlhsl.

End result? As of late 2012, we don’t have any great options for building big web applications. We need a language that supports symbol analysis (refactoring and the like), type error detection, modules, classes. We need a web development platform with a standard library. We need a web development platform that supports debugging in the language you authored.

Enter Microsoft’s TypeScript

TypeScript is a Microsoft attempt at solving these problems, in particular through the adding of types. It’s JavaScript with types.

Does it solve all the problems?

Too early to say.

It was announced only today, and I just downloaded the damn thing. At a glance, they solved the classes and modules problem. And by adding types and tooling support in Visual Studio, you can actually perform refactorings in your big codebase; something Java and C# developers have been enjoying for over a decade, but web developers have been suffering without.

It doesn’t appear to have a standard library – sorry, LINQ fans.

Nor does it appear to support debugging in Visual Studio.

UPDATE: The TypeScript compiler supports an experimental –sourcemap command line argument that generates a source map file. These source map files can then be used to debug TypeScript in the browser. I predict that Microsoft will use these source map files to integrate into Visual Studio’s debugging experience, likely through a special debugging browser similar to the existing Page Inspector.

So that answer for now appears to be, TypeScript is an incremental improvement upon CoffeeScript: types and modules and symbol analysis are a nice addition. We’re still missing some important things, namely, a standard library and a powerful source-language debugger. Until we get these things, web development is still a second class experience.

Code Camp interview

Summary: Joining the Twin Cities Code Camp leadership team. Being interviewed by Mike Hodnick, a well-known Minnesota developer and Code Camp leader. We talk software, the future, code camps, and, of course, llamas. If you find this amusing, you may want to follow up when the tables are turned and I interview Mike.

Who are you?

I’m Judah Gabriel Himango, I work for Avtex. I like programming. I like thinking about technology futures and the evolution of software. But mostly I like programming. Creating things is fun.

What are you most excited about in joining the TCCC leadership team?

The existing TCCC leadership is composed of some of the most well-known, respected names in the Minnesota software community. If I surround myself with excellent people, I might trick people into thinking I’m excellent, too.

Why is TCCC a great event to you?

Code Camp gave me perspective on the Minnesota dev community. Helps you see the broader picture, outside of your little technology ghettos and outside your company. I love talking with nerdly folk; with them I am in my element! 🙂 Really, the reason Code Camp rocks is the people. You get to talk to other devs you’d normally never be exposed to. Good networking comes out of that, but it’s not all about career — it’s good to pick other people’s brains, hear what other people are building, expand your own ideas and horizons. Code Camp does that. Oh, and the tech presentations are fun, too.

Llamas or Alpacas?

In my last Code Camp talk, I built, in front of a live audience, Twin Cities Llama Radio. Oh yeah! May the llamas reign forever and ever.

As a programmer, what is your technology comfort zone (what do you typically work with)?

Oh man. I typically work web dev these days — everything is moving to the web. But, honestly, even though I’ve done web dev for 2 years, I’m not really comfortable, in fact, I feel inadequate. It’s like I should really know more than I do, and someday, somebody’s gonna find me out and expose me as an imposter. Even in my talks, I preface my them with, “Hey, I am not an expert! Forgive me for blatant factual errors. But listen to me anyways.” 🙂

But, yeah, web dev is where I’m working now, where I think the most innovation is happening, and where I’ll be for the foreseeable future.

What is a hidden gem of programming that you think every programmer should know about, but most don’t?

Distributed map/reduce function in Erlang. Yes, Bob, I did.

Do you contribute to any open source projects?

Yeah, I’m a big believer in open source, honestly. I’ve contributed to some libraries I’ve found useful, such as HtmlAgility PackRhinoMocksLucene.NET, and Ninject. I’ve also created a few open source projects for my own stuff, like Chavah Radio. I’m currently in the process of launching a startup, and my main product will be open source as well.

Second try: llamas or alpacas?

Alpacas are llama-wannabes with an inferiority complex. Also, did you know *anyone* can start a Alpaca farm? That’s what an infomercial told me. I mean, are Alpacas so depraved and needy that they resort to late night infomercials? Llamas, dignified and honored as they are, would never stoop to such a low.

Five years ago, did you see yourself doing what you are doing now?

No way! I was really sheltered, actually. I was at a small company writing software; they treated me really well, and I have fond memories of my time there, but I was kind of living in a technology ghetto, ignorant of the state of technology, jobs, and trends in the industry. Code Camp helped me break out of that and see the broad software community, engage with other software people, exchange ideas. Since then, I’ve started speaking at software conferences and user groups, growing my skills, expanding my horizons. I think the last 5 years have been awesome, I’ve advanced myself professionally, I look back and am astounded at how far I’ve come. (Woohoo, go me! :pats self on back:). I’m looking forward to the next 5.

Why does a mouse when it spins?

LOL. At first, I read that, and I was like — wait, is that a typo? OH LOLZ HE MADE A MISTAKE!!111 But then I Googled it, and now I see the light and higher the few. Probably the same difference as a duck, one has wheels. Yee-ahhh!

Apple: Please fix HTML5 on iOS

I’ve built an HTML5 audio player, Chavah Messianic Jewish Radio, that “works” on iOS.

By “works“, I mean, it plays the best it can on Apple’s crippled iOS <audio> implementation. At the time of this writing, this includes iOS 5, tested on iPhone 3 and 4, and iPad1 through iPad 3.

How is <audio> broken on iOS? Let me count the ways:

  • Apple does not let you play audio until user interaction. For me, this means detecting iOS, then showing the music as paused until the user clicks play. Their reasoning is that this will consume data and battery; in practice, though, it cripples web apps, forcing us to resort to platform-specific hoop-jumping  to get the web to work on your platform, Apple. It effectively stifles the evolution of the web in the same way IE did in the late 1990s.
  • Apple does not allow you to play concurrent audio files. This is a huge crippler for gaming apps that need to play sound effects, for example.
  • Apple doesn’t support OGG format. Patent-encumbered formats work, of course, but without support for free and open formats like OGG, Apple is effectively creating a long-term thorn in the side of the opened web.
  • Every dynamic <audio> element must receive user interaction. If you want to play successive sounds (one after another) without receiving user interaction for each one, you must use a single <audio> element, then do the iOS <audio> dance: set existingAudio.src, then call existingAudio.load(), then call existingAudio.play().
  • Audio events don’t fire unless Safari is in the fore. While audio will continue playing if the user switches to a different app, the .ended event won’t fire. This means it’s practically impossible to build a music player web app.

Apple, please, please, please give us better support for HTML5 in iOS Safari. Here are your action items, Apple:

  1. Let HTML5 audio work in the background.
  2. Support OGG.
  3. Support pre-loading audio.
  4. Support concurrent audio.
  5. Let us play audio without user interaction.

Do those things, Apple, you’ll be the industry leader in mobile HTML5 audio, everyone will emulate you, you’ll once again be leading the way, and HTML5 web apps will work best on your platforms. Wouldn’t that be good for Apple?

Yes, these features will use data and the battery, but native apps already do this, so why cripple web apps? Stop stifling the evolution of the web, and start being the leader. Make HTML5 <audio> a first-class citizen on iOS.

Visual Studio 11 crashes with InvalidCastException

UPDATE: This issue can be solved by running the steps described here. There’s also a Microsoft Connect ticket describing the problem.

Just got Visual Studio 11 beta (with recent VS March Update) to crash consistently on start-up:

Launch VS, immediate crash:

Debugging that brings up the following exception:

The exception detail reads:

System.InvalidCastException was unhandled
HResult=-2147467262
Message=Unable to cast COM object of type ‘System.__ComObject’ to interface type ‘Microsoft.VisualStudio.OLE.Interop.IServiceProvider’. This operation failed because the QueryInterface call on the COM component for the interface with IID ‘{6D5140C1-7436-11CE-8034-00AA006009FA}’ failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).
Source=mscorlib
StackTrace:
at System.StubHelpers.StubHelpers.GetCOMIPFromRCW(Object objSrc, IntPtr pCPCMD, IntPtr& ppTarget, Boolean& pfNeedsRelease)
at Microsoft.VisualStudio.OLE.Interop.IServiceProvider.QueryService(Guid& guidService, Guid& riid, IntPtr& ppvObject)
at Microsoft.VisualStudio.Shell.ServiceProvider.QueryService(Guid guid, Type serviceType, Object& service)
at Microsoft.VisualStudio.Shell.ServiceProvider.QueryService(Type serviceType, Object& service)
at Microsoft.VisualStudio.Shell.ThreadHelper.get_Invoker()
at Microsoft.VisualStudio.Shell.ThreadHelper.InvokeOnUIThread(InvokableBase invokable)
at Microsoft.VisualStudio.Shell.ThreadHelper.Invoke(Action action)
at Microsoft.VisualStudio.Services.TaskSchedulerService.VsUIThreadScheduler.<>c__DisplayClass2.<QueueTask>b__0(Object _)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
InnerException:

How did I get in this state? I’m unsure, but I think I accidentally hit F1 key. (Oh noes, the dreaded Visual Studio F1 key!) Somehow, I think that wired up the VS help, which resulted in a consistent crash at boot.

Running devenv.exe /resetskippkgs didn’t fix it. Running a full VS repair doesn’t fix it.

Anyone else hitting this issue?

5:01 developers are visionless hacks

The 501 Manifesto states that:

  • Our families are more important than the commercial goals of business owners
  • Free time is more important than free snacks
  • Living our lives is more important than maintaining our personal brands
  • Sustainable pace is more important than muscle-man heroics
  • Our personal creative projects are more important than commercial products the world doesn’t need
  • Having money for nice clothes is more important than getting free t-shirts from Microsoft
  • Playing fußball in the pub with our friends is more important than playing fußball in the office with our team leader
  • Not being a dick is more important than being a rockstar

“501” refers to programmers who punch out at 5:01pm and never think about technology until 9am the next morning. They don’t read programming blogs or books, nor do they have a desire to advance their craft. Technology is nothing special to them, software is just another job. They might as well be bagging groceries if the pay was comparable.

There’s some good to be gleaned from this manifesto. If your software job is all you do, your life will pass you by and you’ll wonder why you spent so much of your life living for a corporation. You’ll realize what a waste of a life it is to make all that money but not, you know, live. The young & ambitious tend to overlook this wisdom.

Having a life outside of tech is important and rewarding. For me, I play music, I help run a religious congregation, I spend time with my kids and my wife. Those things are hugely rewarding.

I sympathize with the 501 manifesto, but there’s something not quite right here.

The manifesto confuses corporate interference into personal life with passion for technology. That is to say, it incorrectly implies the people who are passionate about technology are those enslaved to their corporate masters. It makes this confusion when it says those who “write technical blogs, attend user groups, or contribute to open source projects” are those who are enslaved by their corporate masters.

It’s silly, because my corporation didn’t ask me to write this blog, attend Code Camp last week, go to user groups, or contribute to open source projects. In fact, many corporations frown on such things, since it takes time away from working on The Project. My corporate masters didn’t ask me to do those things, nor did I do those things in order to impress my corporate masters.

Instead, I do these things because I love technology. And I suspect those of you who do these things do so because, like me, you’re passionate about software and technology, not because some evil corporation forced your hand.

I entered technology because it’s my passion. I take joy in writing software, and in a bigger sense, I consider technology a frontier for humanity and the advancer of civilization. More on that in a minute.

If you entered the technology field for other reasons like money or easy labor, I suggest finding a career you are passionate about. It’s hard to put a price on happiness, and if you aren’t happy with your choice of career, get out before you waste any more of your life on it.

Why be passionate about technology?

I find it freakin’ awesome that we can make new things. We’re in a new age, an age of technology, and we’re the inventors at the fore. We build things from scratch! Then put those new things in front of people. And some of those things might be so useful, that people will use those things regularly. What a rare and precious opportunity we have, that we can build and invent new things, things that can impact humanity.

Hubris? Not at all:

We explore the universe because of technology.

We fix sick people because of technology.

We can communicate with anyone in the world via the push of a button because of technology.

We prevent disease because of technology.

We have the sum total of humanity’s knowledge in a single repository — the internet — because of technology. That we have a single repository containing the sum of human knowledge for the first time in our history is an accomplishment in itself, but just wait until you see the ramifications of this knowledge freely available to all. It will result in an explosion of learning, new technology, new ideas; raw civilization advancement.

I want to be part of that. It’s why I have a passion for technology. It’s why I don’t consider software, the brains behind technology, just another job. I don’t stop thinking about it at 5:01pm. Technology is a frontier for humanity. And I’m going to add to it, even if through some insignificantly small contribution.

Shortsighted and visionless is the person who looks at software as just another job, technology as just another field. If that’s you, find a job you take joy in, and in a field you are passionate about. Your life will be happier.

For the rest of us, the 21st century is a great time to be alive. We, the technologists, are the new inventors. We are the future, and the future, my friends, is bright. Watch and see.

Nerds, Code Camps, and You

Summary: Reflections on Twin Cities Code Camp 12, and my talk on the death of Silverlight and the triumph of HTML5. Why code camps and developer user groups are great for the happiness and careers of all who love technology.

This past weekend marked the 3rd Twin Cities Code Camp conference I’ve attended. Each year gets better.

Why are code camps so great?

You get to hang with nerdy people. That’s me in the blue shirt:portrait of me and another nerd at Code Camp

…You get to talk nerd crap with other nerds:

Cats (both LOL and Nyan variety)? Llamas? Star Wars? Sailor Moon? The nerdliness abounds:

…You get to give fun/silly talks. I spent a good 30 minutes talking serious about the death of web plugins and the triumph of HTML5, but ended with a from-scratch demo building an HTML5 radio station that played…llama music. Oh yes!


I rocked that talk hard, and had a blast doing so:

Move Your Silverlight Skills to the Native Web with KnockoutJS [slideshare id=12574854&w=425&h=355&sc=no]

View more PowerPoint from JudahGabriel

Doesn’t hurt to hear great feedback:


Oh yeah!

Code Camps are total nerd fests, and it’s a blast. You learn tech, you grow your career, you have fun.

If you’re a dev who loves software and technology, step outside your job and have some fun with technology. Try a Code Camp, or even just a local user group. Self-improve and have fun while doing it. Good for career, good for life happiness.

Move Your Silverlight Skills to the Web with KnockoutJS

Summary: Posted the slides for my recent, unconventional talk, “Move Your Silverlight Skills to the Web with KnockoutJS.” And a note about web plug-ins like Flash and Silverlight.

I spoke at the Twin Cities Silverlight User Group yesterday afternoon on a rather uncomfortable topic: Silverlight’s imminent irrelevance. I told Silverlight developers they should stop writing Silverlight code and move to the native web.  🙂 It definitely raised some eyebrows.

Fortunately, they didn’t throw rotten fruit at me; we actually had some great discussion back and forth about the software industry, Silverlight, and the future of the web. Overall, it was a blast! You can view my Rage Comic-infused slides online: Move Your Silverlight Skills to the Web with KnockoutJS

So why should Silverlight developers think about moving their skills to the web? See my earlier post, The Bell Tolls for Flash, and Silverlight Isn’t Far Behind. In a nutshell, we are witnessing in our industry a major shift away from web plugin technology like Flash, Silverlight, and JavaFX. With the explosion of mobile computing, I believe this trend will grow, such  that the only real platform for the web is the native web technologies: HTML, JavaScript, CSS.

If you’re a Silverlight dev, it’s time to move on. The war’s over and, for better or worse, web plugins lost. The native web is where it’s at.

Opened web FTW! How I used opened web APIs and standards to build a beautiful HTML5 web app

Summary: It’s a great time to be a web developer. HTML5, KnockoutJS, MVC3, RavenDB, and opened web APIs and services like Google Docs, IfThisThenThat, and RSS are combined to build a beautiful, useful web app. The opened web is coming to fruition.

I recently released MessianicChords.com, an HTML 5 site for finding lyrics and guitar chords to Messianic Jewish religious music.

Using opened web APIs to build this thing proved effective. Do web developers know these APIs exist? They should; these opened web APIs and standards are powerful tools that will make your app shine.

The raison d’etra of MessianicChords is to provide a way to find lyrics and guitar chords to Messianic Jewish music. This sounds simple, but building it presented a number of challenges:

  • Precise character formatting required. Guitar chords sheets require precise formatting; the G chord must be precisely above the “Let us exalt” line, for example.  This makes automatic conversion to straight HTML unfeasible. This lack of formatting preservation has plagued existing guitar chord sites like ultimate-guitar.com.
  • Proprietary file formats. Guitar chord sheets are often in .doc or .pdf formats due to precise formatting requirements, as well as for historical reasons. This is problematic, because I wanted the chord sheets to be viewable on all devices and platforms, especially tablets like iPad, seeing as how many artists and guitarists use tablets to view the chords while playing music.
  • Searching. I wanted to be able to type a title, artist, or even a partial lyric, and have results come back instantly. Today, users have come to expect Google-like instant search results. In reality, it’s very difficult to make search work like Google: partial matches, incomplete search terms, search qualifiers (such as placing quotes around a group of words) — these all present additional technical challenges.
  • Letting users upload content. While myself and a few contributors have created a large library of Messianic Jewish music chord sheets, approximately several hundred chord sheets, letting the community add chord sheets was important.
  • Automated announcement of new chord sheets. When new chord sheets are added, we’d like to announce it to the world. Better yet, make it automated.
  • An interactive, clean HTML5 app. The existing chords and lyrics sites are littered with ads, have a messy navigation system, and look as though they were designed in the 1990s. Can we do better?

Getting complex, proprietary documents onto the opened web

The formatting issue was the most important to tackle: how can we take complex documents of proprietary file formats and make them available on the opened web?

Enter Google Docs. With Google Docs, you can upload documents, including .doc and .pdf formats, and view them right in your web browser without having to install any special software or plugins. Google’s servers do the heavy lifting on the back end, and you with your web browser reap the fruit.

Additionally, Google provides a sharing model for documents on Google Docs: you can share the document with individuals (specified by email address), you can make the document semi-public by generating a link to the document, or you can make the document entirely public, discoverable by search engines and the like. For MessianicChords, I chose the latter: documents that are public, discoverable by search engines, with a public URL. In addition to sharing, docs can have multiple authors and collaborators. For MessianicChords, I made the chord sheets visible on the web, but read-only to the general public, editable only by myself and some trusted admins.

Finally, and most importantly, Google Docs comes with a powerful API to find documents, metadata, and grab URLs to them. For MessianicChords, having chose ASP.NET MVC3 on the server, there’s a free .NET wrapper for the Google Docs APIs, available as a NuGet package.

With Google Docs, it allowed me to preserve the special formatting of guitar chord sheets, while still making them available on the web. With the Google Docs API, I was able to use Google Docs as a kind of database for the chord sheets. It solved the proprietary file format problem not by lossy conversion, as so many other chord sites do, but by making them viewable via pure HTML/JS.

Seaching

Users have come to expect Google-like search capabilities: instant, partial search terms, close matches, special search terms such as words grouped in quotes, and so on.

With Google Docs as our pseudo-database for chord sheets, I get the best of the search world: Google’s own search algorithms via the Google Docs API. Send over a query with bless God should return all documents with the word bless and God in them; many results. However, if you search for that same phrase, but in quotes, “bless God”, you get only documents that contain that exact phrase.

Best of all, the performance of these queries is excellent: near instant, under a second.

Using Google Docs as a storage mechanism for MessianicChords results in a free, web-accessible database with built-in Google search functionality. Win.

User-generated content and automating the “tell the world about it” part

Since MessianicChords allows users to contribute their own chord sheets, wouldn’t it be nice if we could announce to the world new chord sheets as they’re added?

Sure, I could send out an email, but email blasts are so 1990.

And yeah, I could put an announcement on the Facebook fan page, but Facebook is ultimately a walled garden that will eventually pass away.

Keeping with our spirit of living on the opened web, can we do some sort of announcement of new content that lives on the opened web?

Enter RSS. This is a long-standing, UI-agnostic technology that keeps track of updates to blogs, news sites, or other content that is frequently updated. Boiled down, it’s just an XML file that describes the what, who, and when of updates in a standardized way. The result can be consumed in any RSS aggregator, such as Google Reader.

For MessianicChords, I wanted to publish an RSS feed of new chord sheets. This way, updates to the website are broadcast in a standardized way on the opened web, not limited to Facebook walled gardens, private email inboxes, or proprietary services like Twitter. (Though we can push to those services if we please. More on that later.)

So, the first problem: we need to persistently store the list of updated documents. I could probably have used Google Docs for this, too, but it felt right to have a real database here to keep track of purely application-level things. For this, I used RavenDB Embedded. RavenDB is an open source document database. It’s a great, free database for .NET development, and the embedded version can run inside web apps without any special machine configuration or installation. I plopped that sucker right into my web app, following the lead from the excellent MSDN article, Embedding RavenDB into an MVC3 app.

Within 5 minutes, I was up and running with RavenDB, storing the user’s uploaded chord sheets as records in Raven.

With the upload records in place,  how do you generate an RSS feed from ASP.NET MVC3?

Turns out, it’s a piece of cake, thanks in part to Microsoft’s Windows Communication Foundation (WCF) APIs, in particular, the SyndicationFeedFormatter class.

We’re looking to have a URL where the updates feed resides. For this, you can create a custom MVC ActionResult that wraps up an RSS feed.

public class FeedResult : ActionResult
{
    public Encoding ContentEncoding { get; set; }
    public string ContentType { get; set; }

    private readonly SyndicationFeedFormatter feed;
    public SyndicationFeedFormatter Feed
    {
        get { return feed; }
    }

    public FeedResult(SyndicationFeedFormatter feed)
    {
        this.feed = feed;
    }

    public override void ExecuteResult(ControllerContext context) {
        if (context == null)
        {
            throw new ArgumentNullException("context");
        }

        HttpResponseBase response = context.HttpContext.Response;
        response.ContentType = !string.IsNullOrEmpty(ContentType) ? ContentType : "application/rss+xml";

        if (ContentEncoding != null)
        {
            response.ContentEncoding = ContentEncoding;
        }

        if (feed != null)
        {
            using (var xmlWriter = new XmlTextWriter(response.Output))
            {
                xmlWriter.Formatting = Formatting.Indented;
                feed.WriteTo(xmlWriter);
            }
        }
    }
}

This gives us an ActionResult we can then return from an MVC3 action to render an RSS feed. From there, we return an instance of this class, populating it with the upload records stored in RavenDB. End result? See it in action: MessianicChords.com/Home/UploadsFeed. As you can see, each upload record shows up in the RSS feed, as populated from Raven.

Wooohooo! We now have an RSS feed for user-generated content.

Pushing RSS to proprietary services like Facebook and Twitter

Is RSS enough? How many people know what RSS is, let alone know how to subscribe to it? Sadly, the current state of the web and web browsers makes it awkward at best to subscribe to an RSS feed.

How can we get our nice, standards-compliant, opened web feed of updates to regular, non-techie people?

Enter IfThisThenThat. In a nutshell, this is a free service that puts the web to work for you. In more concrete terms, it lets you listen for actions on the web — a new blog post, an RSS feed updated, a Tweet posted, a Facebook image tagged, whatever you can imagine — and then do some other action in response to that trigger. If [this], then do [that]. Scott Hanselman has a great write-up on the technology.

For MessianicChords, I decided my “this“, that is, my trigger, would be “when the MessianicChords RSS feed is updated”. And my “that“, which is my reaction to the trigger, would be, “Post a new Tweet announcing the new chord sheet.” It looks like this:

I also set up a trigger to announce the new chord sheet to Facebook. Also, to direct message my main Twitter account with the news, so that myself or other admins will quickly send it through the approval process.

Bottom line: we used the If This Then That cloud service to consume our “user uploads” RSS feed, then automatically tweet the news of the new user-generated content and spread it around the social networks. Awesome!

Interactive, clean HTML 5

Last but certainly not least, an important part of all this is the user experience. I want to differentiate my site from other chord sites by having zero ads, and a clean, responsive UI.

Yes, when search results come in, I want a subtle fade-in effect.

Yes, I want nice prompt text that disappears as you enter the textbox.

Yes, I want an inline upload functionality that doesn’t navigate away from the page or your current task.

Yes, I want each search to have its own URL.

Yes, I want search results to appear as I type. But if I type too fast, it should wait a moment before trying to search.

Yes, I want the whole thing to feel smooth and elegant.

While these things can be accomplished “manually”, as it were, with jQuery and straight DOM manipulation, this whole thing would become rather unwieldy.

Enter KnockoutJS. Knockout is a client-side library that makes it easy to build interactive web apps. It accomplishes this through data-binding and the MVVM pattern that separates your UI from your logic. Instead of a mash of UI DOM manipulation and logic in your JS code, with Knockout, you can write apps where your JS code free from DOM manipulation. Your code is cleaner — focused only on logic — allowing you to focus on doing awesome things in your UI.

For MessianicChords, I used data-binding to show the results of the search. My HTML looks like this:

<div class="search-results-container" data-bind="foreach: searchResults>
   <div>
      <a data-bind="text: Name, attr: { href: DocumentUrl }" />
   </div>
</div>

So simple! Our JS, then, creates an array called searchResults, populates it at runtime. When it gets populated, Knockout renders a template for every item in searchResults array. The search results come back with objects containing a Name, a DocumentURL, and those get rendered inside our search-results-container as shown. Awesome!

How about things like automatically searching when we type? And only searching if the typing has been idle for awhile? Knockout to the rescue again:

<input data-bind="value: searchText, valueUpdate: 'keydown'" />

And in our JavaScript:

var searchResults = ko.observableArray();
var searchText = ko.observable();
var searchTextThrottled = ko.computed(function() { return searchText(); }).extend({ throttle: 250 });
searchTextThrottled.subscribe(function(newSearchText) {
   // perform AJAX to MVC action to fetch search results from Google Docs
   $.getJSON("/Home/Search", { searchTerm: newSearchText }).success(function(results) {
      searchResults.pushAll(results);
   });
});

As soon as we push results into our searchResults array, Knockout automatically renders the search results onto the page, thanks to data-binding. Sweet!

And notice how we subscribed to the throttled value of the search text. The searchText observable is updated as the user types, then the searchTextThrottled value waits for a no-typing period of 250ms before sending the query to the server. Complex UI interaction pattern, but took just 3 or 4 lines of JS code to accomplish, thanks Knockout.

Bottom line: KnockoutJS gave us the ability to write an interactive UI with ease.

Summary: It’s a great time to be a web developer!

The web is evolving and getting better. The technologies now available to us as web developers is staggering; tools like HTML5, KnockoutJS, Modernizr, RSS make it easy to build web apps that shine. And  external services like IfThisThenThat, RSS, and Google Docs, makes building services that integrate with the opened web a piece of cake.

These opened web APIs grant web developers powerful capabilities not in previous generations. The opened web is coming to fruition, and we’re reaping the benefits.

Web dev for the win.

How to partially solve the blogs-are-dying problem: comments

One of the reasons that blogs are dying is that blogs no longer represent an individual online. Instead, Twitter, Facebook, or even email represents a person online. Blogs are relegated as unimportant side shows.

One problem with this — and forgive my verbing a noun — is that it silos a person into a single commercial entity. If your Facebook profile is you, no one can know about you except your Facebook friends. The only people than come in and go out are Facebook users. That’s a problem, because if Facebook ever becomes passe — not unlike MySpace did in recent memory — your identity online is scrapped.

So what other options do we have, and how can we make blogging relevant again?

Respected technologist Dave Winer recently posited that, on the net, your feed is you. What is your feed? I’d broaden Winer’s definition and say, it’s all the public stuff you do on the web: your blog posts. Your tweets. Your Facebook comments and status. Your comments on other blog posts. All of it.

That’s you. On the web, you are your feed. 

The important part is the content you write, which is ultimately your blog. Everything else is secondary; Twitter can be an announcement mechanism for your blog posts. Facebook can, too. Today, that’s what content producers tend to use Twitter and Facebook for. You author your content on your site, then spam Twitter and Facebook. Then people come out of those silos and read your content on the opened web. This addresses the first problem with blogging, and that is makes your blog your primary authoring tool on the web, which helps push it back into its spot as your online identity.

The Comments Problem

Another problem with blogs is nobody comments on them. It’s too hard to comment on a blog.

To be sure, an audience isn’t a problem: in your blogging, if you keep jabbing, and throw the occasional haymaker, and you do this consistently, you’ll get an audience, no matter your niche  topic. The problem isn’t from a lack of people.

Yes, even with an audience, people don’t comment on blogs. And part of the reason is the insane roadblocks you must navigate through to add a blog comment: OpenID provider? Do we know you? No? Sign up! Yes? Name, please! Password, please! Url, please! Type this hard to read CAPTCHA! Preview, please! Submit! Oh what’s that, we don’t support that HTML tag! Try again! OK! Now submit!

This commenting barrier problem has created an amusing phenomenon: the announcement of the blog post on Facebook gets more comments than the blog itself. Maybe 15 people will go back and forth about your blog…but they’ll do so on Facebook, not your blog. Ditto for Twitter. After all, it’s so easy to comment on Facebook; there’s no typing your name, no typing your URL, no preview, no CAPTCHA, no confusing OpenID provider selection, none of that. Just type your comment and hit enter.

One solution I’d like to propose to this problem is automated duplication of Facebook comments into blog post comments. Yes, if Daria M. writes a comment on your post announcement on Facebook, why not do her a favor and copy that comment onto the opened web, onto your blog? Make it automated. Suddenly, your blog posts have 40 comments instead of 5. And they’re all on the opened web.

Here’s what this would look like:

  1. Write a piece of software that looks at your Facebook feed.
  2. When that software detects a feed item that contains a link, unwrap the link (almost all services now, including Facebook and Twitter use web-breaking URL shortening services)
  3. If the unwrapped link points to a post on your blog, check the comments.
  4. If the comments exist on that feed item, talk to the blog services (e.g. WordPress, Blogger) APIs to post a comment to linked blog post.

End result: when someone comments on your Facebook status announcing your new blog post, comments to that status get reproduced on your blog. The same thing goes for Twitter and Google+.

Here’s what’s involved with making this work:

For #1, we’d need an app that talks to [social media service]. A Facebook app, in this case.

For #2, our software would need the ability to unwrap links from URL shortening services.

For #3, our software would need to work with the social media service’s APIs to fetch comments on a feed item. Facebook and Twitter already support this.

For #4, we’d need our software to communicate with [blogging service] to post a new comment to the blog. WordPress and Blogger likely already support this.

The feels like a problem that could be solved with the IfThisThenThat (IFTTT) web service. There, you can create recipes for [when something happens on the web] then [do this other thing on the web]. The problem, however, is it appears that IFTTT doesn’t have any tasks adequately powerful. While Twitter (and Facebook?) have RSS feeds for your feed, there’s no built-in IFTTT task for unwrapping shortened hyperlinks (step #2), fetching comments to feed items (step #3), or posting comments to blogs via blog APIs.

Could it be that this could be done with Yahoo Pipes? That feels a bit more powerful than IFTTT.

In any case, I’d like to see this done. I might just have to do it myself, but I’d love to find that someone’s already done the heavy lifting.