javascript – Debugger.Break()

I built a PWA and published it in 3 app stores. Here’s what I learned.

Summary: Turning a web app into a Progressive Web App (PWA) and submitting it to 3 app stores requires about a month of work, a few hundred dollars, and lots of red tape.

I recently published Chavah Messianic Radio, a Pandora-like music player, as a Progressive Web App and submitted it to the 3 app stores (Google Play, iOS App Store, Windows Store).

The process was both painful and enlightening. Here’s what I learned.

Why?

First, you might wonder, “Why even put your app in the app stores? Just live on the opened web!”

The answer, in a nutshell, is because that’s where the users are. We’ve trained a generation of users to find apps in proprietary app stores, not on the free and open web.

For my web app, there were 2 big reasons to get in the app store:

User demand
Web app restrictions by ~~Apple~~ hostile mobile platforms

User demand: My users have been asking me for years, “Is there an app for Chavah? I don’t see it in the store.”

They ask that because we’ve trained users to look for apps in proprietary app stores.

My response to my users has up until now been,

“Aww, you don’t need an app – just go to the website on your phone! It works!”

But I was kind of lying.

Real web apps only kinda-sorta work on mobile. Which brings me to the 2nd reason: web app restrictions by ~~Apple~~ hostile mobile platforms.

Mobile platform vendors, like Apple, are totally cool with apps that use your phone to its fullest. Access your location, play background audio, get your GPS coordinates, read all your contacts, play videos or audio without app interaction, read your email, intercept your typing, play more than one thing at a time, use your microphone and camera, access your pictures, and more.

Apple’s totally cool with that.

But only if you pay Apple $99/year for the privilege.

If you want to do any of those things in a regular old web app, well, goshdarnit, Apple won’t just deny you these things, it prevents you from even asking permission.

For my Pandora-like music player app, this horrible brokenness showed up in numerous ways.

From minor things like “iOS Safari won’t let you play audio without first interacting with the page” to major, show-stopping things like, “iOS Safari won’t let you play the next song if your app is in the background or if your screen is off.”

Oh, plus weird visual anomalies like typing in a textbox and seeing your text appear elsewhere on screen.

So, to make my HTML5 music app actually functional and working on people’s mobile devices, it was necessary to turn my PWA into an app in app store.

Barriers to entry

In the ideal world, publishing your web app to the app stores would look like this:

Your Web/Cloud Host or CI Provider

You’ve published a Progressive Web App. Publish to app stores?

☑ iOS App Store
☑ Google Play
☑ Windows Store

(Or alternately, as Microsoft is experimenting with, your PWA will just automatically appear in the app store as Bing crawls it.)

But alas, we don’t live in this ideal world. Instead, we have to deal with all kinds of proprietary native BS to get our web apps in the stores.

Each of the app stores has a barrier to entry: how difficult it is to take an existing web app and it in the app store.

I list some of the barriers below.

Cost

Apple: $99/year to have your app listed in the iOS app store.
Google: One-time $25 fee to list your app in the Google Play Store.
Microsoft: Free!

Don’t make me pay you to make my app available to your users. My app enriches your platform. Without good apps, your platform will be abandoned.

Apple used to understand this. When it first introduced the iPhone, Steve Jobs was adamant that HTML5 was the future and that apps will simply just be web apps. There was no native iPhone SDK for 3rd parties. Apple has since abandoned this vision.

Google asked for a token $25 one-time fee. Probably to avoid spammers and decrease truly junk apps from entering the store.

Microsoft seems determined to just increase the total number of apps in their app store, regardless of quality.

Winner: Microsoft. It’s hard to beat free.

Adding native capabilities

In an ideal world, I wouldn’t have to write a single extra line of code for my web app to integrate into the OS. Or, as Steve Jobs said back in 2007,

“The full Safari engine is inside of iPhone. And so, you can write amazing Web 2.0 and Ajax apps that look exactly and behave exactly like apps on the iPhone. And these apps can integrate perfectly with iPhone services. They can make a call, they can send an email, they can look up a location on Google Maps.”

-Steve Jobs, 2007

For me, that means my web app plays background audio using standard HTML5 audio; that works just fine on all OSes.

My web app declares what audio is playing, and the OSes pick up on that, show currently playing song info on the lock screen.

My app controls audio using standard HTML5 audio API; the OS picks up on that and provides play/pause/next/volume/trackbar controls on the lock screen.

But sadly, we don’t live in this ideal world. All the things listed above don’t actually work out of the box on all 3 platforms.

My web app needs to play audio in the background. And load URLs from my CDN. Sounds reasonable, right? And bonus, how about showing currently playing song info on the lock screen? And controlling the audio (play/pause/next, etc.) from the lock screen? How hard is this?

Three very different approaches taken here:

Apple: We don’t give web apps a way to declare such capabilities; you’ll need to write a native wrapper (e.g. with Cordova) to interact with the OS.
Google: Web FTW! Let’s create a new web standard that shows audio & controls from the lock screen. Background audio? Sure, go ahead!
Microsoft: We’ll inject our proprietary API, window.Windows.*, into your JavaScript global namespace and you can use that to do the things you want to do.

Going into more details here for each store:

For iOS app store, does your web app need to play background audio? Use a Cordova plugin. Need to show currently playing song on the lock screen? Use a Cordova plugin. Need to control the currently playing song from the lock screen? Use a Cordova plugin. You get the idea. Basically, Cordova tricks Apple into thinking you’re a native app. And since you’re not a yucky web app, Apple lets you do all the things native apps can do. You just need native tricks – Cordova plugins – to let you do it.

For Google Play, it’s nice that I can just write JS code to make this work; no Cordova plugins required here. Of course, that JS won’t work anywhere except Chrome on Android…but hey, maybe one day (in an ideal world!) all the mobile browsers will implement these web APIs…and the world will live as one. I’m almost ready to bust out some John Lennon hippie utopia tunes.

For Windows Store, do you want to play background audio? Sorry! That is, unless you declare your intentions in our proprietary capabilities manifest file (easy) AND you implement this proprietary media interface using window.Windows.SystemMediaTransportControls (not so easy). Otherwise we’ll mute you when your app goes to the background.

Winner: Google. I want to be able to just write JavaScript, and let the OS pick up cues from my app.

Runner-up: Windows. I can still write plain old JavaScript, but I need to talk to a proprietary Windows JS API that was injected into my process when running on Windows. Not terrible.

Loser: Apple. They don’t care about web apps. Actually, it’s worse than that. It feels like they are actually hostile to web apps. iOS Safari is the new Internet Explorer 6. It has lagged behind in nearly every web standard, especially around Progressive Web Apps. This is probably for business reasons: web apps disrupt their $99/year + 33% in-app purchases racket. So to make my web app work on their platform, I have to basically pretend I’m a native app.

App Store Registration

Submitting your PWA to the app store requires registration, business verification, and more red tape. Here’s how the 3 app stores fared:

Apple: You must prove that you’re a legal, registered business. This verification isn’t done by us – but by a 3rd party, which may or may not know about your business.
Google: You want your app in our store? Cool by us.
Microsoft: You want your app in our store? Cool by us.

The biggest pain point for me was getting verified as a legal business by Apple.

First, I went to the site and registered for Apple’s Developer Program. I filled out my name and company information. (Aside: I guess Apple won’t let you submit an app unless you have a registered, legal company?)

I click next.

“The information you entered did not match your D&B profile.”

My…what?

A bit of Googling showed that “D&B profile” is Dun and Bradstreet. I’ve never heard of this group before, but I find out that Apple is using them to verify you legal corporation details.

And apparently, my D&B profile didn’t match what I put in my Apple Dev registration.

I google some more and find the Apple dev forums littered with similar posts. Nobody had a good answer.

I contact Apple Dev support. 24 hours later, I’m contacted by email saying that I should contact D&B.

I decide to contact them…but Apple says it will take up to a few days for them to respond.

At this point, I’m thinking of abandoning the whole idea.

While waiting for D&B support to get back to me, I decide to go to the D&B site, verify my identity, and update my company information which, I assume, they had taken from government registration records.

Did I mention how sucky this is? I just want to list my existing web app in the store. Plz help.

I go to D&B to update my business profile. Surprise! They have a JavaScript bug in their validation logic that prevents me from updating my profile.

Thankfully, I’m a proficient developer. I click put a breakpoint in their JavaScript, click submit, change the isValid flag to true, and voila! I’ve updated my D&B profile.

Back to Apple Dev –> let’s try this again. Register my company…

“Error: The information you entered did not match your D&B profile.”

AREYOUFREAKINKIDDINGME.

Talk to Apple again. “Oh, it may take 24-48 hours for the updated D&B information to get into our system.”

You know, because digital information can take 2 days to travel from server A to server B. Sigh.

Two days later, I try to register…finally it works! Now I’m in the Apple Developer program and can submit apps for review.

Winner: Google and Microsoft; both took all of 5 minutes to register.

Loser: The Apple Developer registration was slow and painful. It took about a week to actually get registered with their developer program. It required me contacting support from 2 different freaking companies. And it required me to runtime debug the JavaScript code on a 3rd party website just so that I can get past their buggy client-side validation, so that my info will flow to Apple, so that I can submit my app to the store. Wow, just…wow.

If there is any saving grace here for Apple, it’s that they have a 501c3 non-profit program, where non-profits can have their $99 annual fee waived. I took advantage of that. And perhaps this extra step complicated matters.

App Packaging, Building, Submitting

Once you have a web app, you have to run some magic on it to turn it into something you can submit for App Store review.

Apple: First, buy a Mac; you can’t build an iOS app without a Mac. Install XCode and these build tools and frameworks, acquire a certificate from our developer program, create a profile on a separate website called iTunes Connect, link it up with the certificate you generated on the Apple Dev center, then submit using XCode. Easy as one, two, three…thirty-seven…
Google: Download Android Studio, generate a security certificate through it, then package it using the Studio. Upload the package to Android Developer website.
Microsoft: Generate an .appx package using these free command line tools, or Visual Studio. Upload to the Microsoft Dev Center website.

The good news is, there’s a free tool to do the magic of turning your web app into app packages. That awesome free tool is called PWABuilder. It analyzes a URL, tells you what you need to do (e.g. maybe add some home screen icons to your PWA web manifest). And in a 3 step wizard, it lets you download packages that contain all the magic:

For Windows, it actually generates the .appx package. You can literally take that and submit it on the Windows Dev Center site.
For Google, it generates a wrapper Java app that contains your PWA web app. From Android Studio, you build this project, which generates the Android package that can be uploaded to the Android Dev Center site.
For Apple, it generates an XCode project which can be built with XCode. Which requires a Mac.

Once again, Apple was the most painful of all of these. I don’t have a Mac. But you cannot build the XCode project for your PWA without a Mac.

I don’t want to pay several thousand dollars to publish my free app in Apple’s app store. I don’t want to pay for the privilege of enriching Apple’s iOS platform.

Thankfully, MacInCloud costs about $25/month, and they give you a Mac machine with XCode already installed. You can remote into it using Windows Remote Desktop, or even via a web interface.

It wasn’t enough to just build the XCode project and submit. I had to generate a security certificate on the Apple Developer site, then create a new app profile in a separate site, iTunes Connect, where you actually submit the package.

And that wasn’t all: since Apple is hostile to web apps, I had to install some special frameworks and add Cordova plugins that allow my app to do things like to play audio in the background, add the current song to the lock screen, control the song volume and play status from the lock screen, and more.

This took at least a week of finagling to get my app into a working state before I could submit it to the app store.

Winner: Microsoft. Imagine this: you can go to a website that generates an app package for your web app. And if that’s not your thing, you can download command line tools that will do the job. GUI person? The free Visual Studio will work.

Runner-up: Google. Requires Android Studio, but it’s free, runs everyone, and was simple enough.

Loser: Apple. I shouldn’t have to buy a proprietary computer – a several thousand dollar Mac – in order to build my app. The Apple Dev Center –> iTunes Connect tangling seems like an out-of-touch manager’s attempt to push iTunes onto developers. It should simply be part of Apple Developer Center site.

App Testing

Once you finally did all the magic incantations to turn your existing web app into a mobile app package, you probably want to send it out to testers before releasing your app on the unwashed masses.

Apple: For testing, you have your testers install Test Flight on their iOS device. Then you add the tester’s email in iTunes Connect. The tester will get a notification and can test your app before it’s available in the app store.
Google: In Android Dev Center, you add email addresses of testers. Once added, they can see your alpha/beta version in the App Store.
Microsoft: I didn’t actually use this, so I won’t comment on it.

Winner: Toss up. Apple’s Test Flight app is simple and streamlined. You can control alpha/beta expiration simply on the admin side. Google wasn’t far behind; it was quite painless, not even requiring a separate app.

App Review

Once your app is ready for prime time, you submit your app for review. The review is done using both a programmatic checklist (e.g. do you have a launch icon?) and by real people (“your app is a clone of X, we reject it”)

Apple: Prior to submission, XCode warns you about potential problems during build. The human app review takes about 24-48 hours.
Google: Anybody home? Android Studio didn’t tell me about any potential problems, and my app was approved within minutes of submission. I don’t think a real human being looked at my app.
Microsoft: Upon submission, a fast programmatic review caught an issue pertaining to wrong icon formats. After passing, a human reviewed my app within 4 days.

Winner: Apple.

Sure, as a developer, I like the fact that my app was instantly in the Google Play store. But that’s only because, I suspect, it wasn’t actually reviewed by a human.

Apple had the quickest turnaround time for actual human review. Updates also passed review within 24 hours.

Microsoft was hit or miss here. The initial review took 3 or 4 days. An later update took 24 hours. Then another update, where I added XBox platform, took another 3-4 days.

Conclusion

It’s painful, and costs money, to take an existing PWA and get them functional on mobile platforms and listed in the App Store.

Winner: Google. They made it the easiest to get into the app store. The made it the easiest to integrate into the native platform, by attempting to standardize web APIs that OS platforms can pick up on (hello, lovely navigator.mediaSession)

Runner-up: Microsoft. They made it the easiest to sprinkle your web app with magic, turning it into a package that can be submitted to their store. (Can be done for free using the PWABuilder site!) Integrating with their platform means using the auto-injected window.Windows.* JavaScript namespace. Not bad.

Loser: Apple. Don’t require me to buy a Mac to build an iOS app. Don’t force me to use native wrappers to integrate with your platform. Don’t require me to screw around with security certificates; let your build tools make them for me, and store them automatically in my Dev Center account. Don’t make me use 2 different sites: Apple Dev Center and iTunes Connect.

Final thoughts: The web always wins. It defeated Flash. It killed Silverlight. It destroyed native apps on desktop. The browser is the rich client platform. The OS is merely a browser-launcher and hardware-communicator.

The web will win, too, on mobile. Developers don’t want to build 3 separate apps for the major platforms. Companies don’t want to pay for development of 3 apps.

The answer to all this is the web. We can build rich web apps – Progressive Web Apps – and package them for all the app stores.

Apple in particular has a perverse incentive to stop the progress of the web. It’s the same incentive that Microsoft had in the late ‘90s and early 2000s: it wants to be the platform for good apps. PWAs undermine that; they run everywhere.

My software wisdom is this: PWAs will eventually win and overtake native mobile apps. In 5-10 years, native iOS apps will be as common as Win32 C apps. Apple will go kicking and screaming, keeping iOS Safari behind the curve, blocking PWA progress where they can. (Even their recent “support” for PWAs in iOS Safari 11.1 actually cripple PWAs.)

My suggestion to mobile app platforms is embrace the inevitable and either automatically add quality PWAs to your app store, or allow developers to easily (e.g. free, and with 3 clicks or less) submit a PWA to your store.

Readers, I hope this has been helpful glance at PWAs in App Stores in 2018.

Have you submitted a PWA to an app store? I’d love to hear your experience in the comments.

It’s almost 2017, and HTML5 audio is still broken on iOS

Summary: Back in 2012, I wrote that HTML5 audio is broken on iOS. Now as we enter 2017, it turns out things are still horribly broken. It is currently impossible to build a music player web app that works on iOS.

Update June 2017: I’ve filed a bug with the iOS WebKit team to address the major blocking issue. Here’s to hoping they fix it!

Last week I published a big update to Chavah Messianic Jewish radio. It’s an HTML5 music player in the vein of Pandora (users can thumb up songs, influencing what gets played) for the Messianic Jewish faith.

And thanks to the magic of the web and HTML5 audio, it works flawlessly on PCs, Macs, and Linux. Sweet!

What about mobile? Well, yeah. Umm. Apple’s mobile implementation of HTML5 audio is still busted.

After releasing the new version last week, my users reported things still busted on iOS devices:

Well, I did have a look. And what I found is that iOS still cripples web apps that use HTML5 audio.

I was hoping that the new (July 2016) relaxed web media restrictions in iOS 10+ would un-cripple HTML5 audio in iOS.

I was disappointed to find it’s still broken. It’s currently impossible to write a working audio player using modern web technologies.

Here are the working things:

You can play MP3 audio* ** *** ****

* Only after user interaction
** Only while the page is active and in the fore
*** Only while the phone screen is on
**** There’s no way to keep the screen from turning off, so your audio will stop after the first song.

So yes, you can play audio (10 asterisks here). Or more precisely, you can play a single audio track, but not much more.

Here are the busted things:

Minor: Audio doesn’t play until user interaction.
Major: Audio can’t play the next track when the page is in the background. (JavaScript execution is suspended; no way to set audio.src for the next song.)
Major, blocking: Audio can’t play the next track when the phone screen is off.
(JavaScript execution is suspended when the phone screen is off; no way to set audio.src for the next song)

Details on each of these below:

Audio doesn’t play until user interaction

This is the most minor of the busted things. But it’s an artificial restriction by Apple, likely for user experience and battery life reasons.

None of the other operating systems, mobile or desktop, do this. So we have to have special handling for iOS to require the user to interact with the UI before playing.

As for battery life, 3d content, muted <video>, gifs, ads and more don’t require interaction before start. Why hurt real web apps and real users by singling out audio apps?

Audio can’t play next track in background

Common user scenario: they go to my web app, hit play, and the music starts streaming in. Now they switch over to Twitter. When the current song ends, the music just stops. No new track is played.

Why?

Upon investigation, HTMLAudioElement ended event is never fired. Why? Because Apple suspends all JavaScript execution in the name of performance.

This sounds good in theory: you’re not using a web site, so Safari will just stop executing any JavaScript since you’re not using it anyways.

But in practice, this kills real web apps on iOS. Music playing apps need to know when the song is finished in order to play the next song. So, we call

audio.addEventListener(“ended”, playNextSong)

But the playNextSong function never fires; JavaScript has been suspended, and users are disappointed.

Audio can’t play the next track when the phone screen is off.

The most common scenario for my app: user goes to my web app, the app starts playing music. Then, the user leaves her phone alone; perhaps she’s in the car driving while listening to the music.

After a short period of time, the phone screen turns off. The music keeps playing…until the current song ends. Once again, iOS has suspended JavaScript execution, resulting in the audio.ended event never firing, meaning I can’t set audio.src to the next song.

Upon further investigation, I tried to find out if it’s possible to prevent the screen from sleeping.

For a native iOS app, you can set application.idleTimerDisabled = YES. Super simple.

But for a web app? Nope, there’s no supported way to tell iOS, “Hey, keep the screen on, the user is in the middle of listening to music so don’t disrupt them.”

Some dated information on StackOverflow suggests looping a silent audio or video may prevent sleep. I built a little test app to try this out, and it appears to no longer work on iOS 10.

Additional answers on StackOverflow suggest pseudo page navigation every few seconds to prevent phone sleep/lock. I tried this as well, and it likewise appears to no longer work on iOS 10. The phone still sleeps even with page navigation going on.

Bottom line: there appears to be no way for a web app to prevent an iPhone from sleeping/locking.

And since sleeping/locking will cause a suspension of JavaScript execution, there’s no way to play the next song. End result is your audio web app stops playing audio, making it pretty useless.

Apple Webkit team action items

Here’s what we web developers need to make audio web apps a first class citizen on iOS:

Don’t suspend JavaScript execution for web apps playing audio. Don’t suspend JS execution if we’re in the background. Don’t suspend JS if the phone is locked. The user is playing our audio for a reason, don’t disrupt the user.

It’s not enough to let the audio finish and then suspend JS; this breaks the user experience and causes audio web apps to stop working.

This would solve the other problems.

Alternately, don’t sleep/lock the phone if an active web app is playing audio. While this alternate solution doesn’t fix all the problems, it would address the most blocking use case for using audio web apps unattended.

RavenDB Studio 3.0, and why we moved from Silverlight to HTML5

Summary: A big step for RavenDB: a new HTML5 Studio. Plus, some thoughts on the move from Silverlight to HTML5 and our experience in the transition.

Yesterday, I pulled the covers off something I’ve been working on for a few months, something I’m very proud of.

RavenDB, the most popular NoSQL database in the .NET world, announced a brand new RavenDB Management Studio, Raven Studio 3.0, built from the ground up using HTML5 and modern web technologies. Yes, we’re moving away from Silverlight and onto HTML5.

Ayende and myself demoed the new Raven Studio just yesterday in a live webinar:

[youtube=http://www.youtube.com/watch?v=FNDKuiftKcc&w=448&h=252&hd=1]

This has been my pet project for the last few months, and it’s something I’m quite proud of! I believe this is a huge step forward for RavenDB (more on that in a minute), and the reception from the Raven community has been awesome, ego-stoking, and totally energizing.

The old Raven Studio was built in Silverlight. Some Silverlight fans have asked, why did we move to HTML5? Are we making a big mistake moving away from Silverlight and to HTML5?

No, on the contrary, we believe HTML5 is a damn good option.

The RavenDB community wants an HTML5 Studio. This has probably been the most-requested item from the RavenDB community. It came up multiple times in the Raven 3.0 Wishlist, it’s come up whenever Oren talks about the Studio, it comes up when we speak to the external developer community, heck, when I was in Israel for RavenDB training last year, one of the students brought it up right there in the Hibernating Rhinos office. Silverlight has served us well over the years, but Silverlight is a dying technology that our community doesn’t want to be tied to any longer.
Silverlight tooling is a perceived barrier to Raven adoption. I speak at Code Camps and user groups, and when I speak on RavenDB, the love flows and the excitement grows…until I show Silverlight tooling. I get the raised eyebrow. “Silverlight? Oh. I see.” Others in the Raven community have reported this as well. For some, Silverlight is a stumbling stone.
HTML5 is a step towards cross-platform Raven. RavenDB is the best NoSQL database for .NET. But, in time, we want Raven to spread her wings and be not just the best NoSQL solution for .NET, but the best NoSQL database, period. Moving to an HTML5 toolset is a step towards this goal.
The software industry is moving away from plugins. Plugins like Silverlight added abilities you couldn’t do on the native web, such as audio, video, gaming, 2d drawing, documents, voice, and more. Plugins filled these gaps, but with HTML5, these gaps are disappearing. We don’t need Adobe Acrobat plugins anymore to view that high fidelity document. We don’t need Java applets anymore to run that simulation. We don’t need websites built entirely with Flash. And we don’t need Silverlight for Raven Studio. There is little reason today to build something in JavaFX, Flash, or Silverlight: the native web has supplanted them. Just as it’s rare – and often undesired – to see a Java applet out in the wild, so too it will be with Silverlight in the coming years.
The native web platform is a solid foundation for the future. Microsoft products come and go. 3 years ago, Microsoft was pushing Silverlight as the platform for line of business apps and islands of richness on the web. Today, Silverlight is prevented from running in the default browser of their newest operating systems.
HTML, on the other hand, has been a stable, ever-evolving technology for decades, and because it is the very fabric of the web, things built in HTML live indefinitely. There’s a reason you can still visit and use the 17-year old Space Jam Website. But your MS Silverlight app from last year? It won’t run even on the latest MS operating system’s default browser.

As a Silverlight developer who has built professional apps (e.g. 3M Visual Attention Service) and spoken at Code Camps and user groups on Silverlight, truth be told, Silverlight is a great developer platform. C# is an excellent language, Visual Studio probably the best development environment.

But, in the words of Miguel de Icaza, creator of Moonlight (open source, cross-platform Silverlight),

“I felt that Silverlight had a bright future, and that it could turn to fill an important void, not only for web development, but for desktop development in general. And this was largely one of my motivators. I am very sad that Microsoft strategy cut the air supply to Silverlight.”

This, coupled with the mobile computing explosion and the software industry’s shift away from plugins, results in a sickly future for Silverlight and Silverlight apps.

RavenDB rocks, and we want the tooling to rock as well. Having our tooling tied to this technology was not an attractive proposition, and it was time for us to move on.

A new technology stack for Raven Studio 3.0

After much deliberation and considering all the options available to us, we moved off of Silverlight.

Instead of Silverlight, HTML5.

Instead of C#, TypeScript.

TypeScript is awesome. TypeScript is new language, a superset of JavaScript designed for building apps on the web. Silverlight fans will be happy to know it’s built by none other than Anders Heijlsberg, the much-respected language designer and author of C#.

In TypeScript, all JavaScript is valid TypeScript code, so it’s familiar to any web developer, but it gives us nice things like an optional, flexible type system, classes, modules, and enums, and features proposed for future versions of JavaScript, but compiles to plain old JavaScript that runs in every browser.

TypeScript tooling is Visual Studio, with all the nice debugging and refactoring that brings, but it can also be written in any text editor and debugged in any browser.

For infrastructure, because we wanted the look & feel of a web application, rather than a set of web pages, we opted to build a single page application (SPA). Durandal.js gives us exactly that: a nice means to load pages on demand and compose them into a cohesive web application.

For UI, Durandal uses Bootstrap for a consistent, pleasing aesthetic, and KnockoutJS for data binding and MVVM.

Using data binding, MVVM, and Durandal makes a great developer experience, one not too foreign to the MVVM stuff in Silverlight. (Indeed, the author of Durandal.js is the same author of the popular Silverlight MVVM framework Caliburn Micro.) Look at the code and judge for yourself; you’ll see classes separated out into small, logical view models, and a clean separation between view and logic.

What has been our experience moving to HTML5?

One immediate, measurable gain was performance:

Memory usage dropped from 140MB to 20MB.
Cold starts dropped from ~7s to ~2s.
Warm starts dropped from ~3s to ~1s.
General snappiness: XAML is rather heavyweight, and you’ll notice just moving around the application, loading your documents, collections, or editing – it’s all faster in HTML5. Snappy and responsive.
This doesn’t happen:
This doesn’t happen either:

A lot of the above we get for free simply by Doing Less Stuff™. No .xap files to download, no dlls to load, no CLR runtime to start, no plugin host process for the browser, no browser-to-plugin communication, no managed code to start executing.

This translates into faster start times and less memory usage.

Another free item we get is JavaScript and the blazing-hot modern JS browser runtimes. The major browsers – IE, Firefox, Chrome, Safari – are in a cut-throat competition to get the fastest JavaScript runtime, to squeeze every possible ounce of performance out of JavaScript. You’ll regularly see these browser vendors advertising their JS benchmarks as proof of performance improvement. This is a free win for the new HTML5 Raven Studio: as browser vendors continue to improve their engines in this cut-throat, cross-company competition of speed, Raven Studio will reap the performance improvements.

Moving to the native web platform fixes some plugin-induced workflow hiccups. For example, keyboard shortcuts: Silverlight and other plugins eat the keyboard. So say you’re got Raven Studio opened, and you want to open a new browser tab, so you hit CTRL+T. Surprise, nothing happens. Why? Silverlight ate your keyboard shortcuts, your browser never received them, and your workflow was just interrupted.

If you’ve ever used one of those old all-Flash websites, or full-page Java applets, you’ve probably noticed some things just don’t feel right. So it was with the old Silverlight Studio. Moving to HTML5 fixes these issues.

Conclusion

Transitioning out of the plugin ghetto and moving to HTML5 has been a delight, but more importantly, it’s good for RavenDB users as we move to a faster, more lightweight tool. It’s good for the future of RavenDB to have our tooling built on the solid rock of the native web.

I understand the Silverlight fans who are sad to see the old Silverlight Studio go. I’m a Silverlight fan myself, I understand their concerns. The most I can ask of you guys is to give us the opportunity to earn your trust. It will take time, but with a faster, more lightweight, stable tool that does what you need and gets out of your way, I believe that trust will be earned.

The new HTML5 Raven Studio is on GitHub and we’d love for you to give it a spin or even contribute to the code. I’m pleased to say we already have had a few contributions since it was released just yesterday. I’m proud of this work, and I really hope you guys enjoy it!

My beautiful web development soup

Summary: Web development is chaotic, overwhelming, and beautiful. How many technologies does it take to build something useful on the web? You might be surprised. Advice for developers getting into web development.

This week I was working on my open source startup project, BitShuva Radio. I acquired a new client this week, but this client was unique because he’s a Java developer and wanted to know how to edit the open source code for his new radio station.

As I typed out my answer, it dawned on me that the sheer number of technologies utilized in building a useful web application is staggering. That they actually work together is a testament of the beautiful soup that is web development today.

How many technologies does it take to build something useful on the web? Here’s my answer:

I use TypeScript, rather than plain JavaScript, for all client-side code. A superset of JavaScript with optional types, TypeScript is a well-structured, typed language with excellent tooling. I use it for practical reasons: as the app grew in complexity, the need for a more structured language became apparent. I ported the JS to TypeScript last month, and I’m quite pleased with the results.
KnockoutJS is ~~used heavily~~ absolutely relied on. BitShuva is a dynamic single page application, meaning we’re doing lots of dynamic updates to the UI in real time. Knockout lets us make dynamic UIs very easily without resorting to tons of DOM manipulation code. Instead of DOM manipulation, Knockout lets you change your objects, and your UI automatically updates. For example, songsOnScreen.push(new Song()) will automatically show the song in the UI, no jQuery DOM manipulation required.
I use knockout.postbox for decoupled pub/sub communication between client-side code components. For example, when I hit the play button, the click handler doesn’t need to know about HTML5 audio or any infrastructure concerns. Instead, it calls ko.postbox.publish(“Play”). Whoever is in charge of playing audio simply calls ko.postbox.subscribe(“Play”, …). This way, the click handler and the HTML5 audio component don’t need to know about each other. Nice and clean. Bonus: it works with Knockout observables, so you can say currentSong.publishOn(“SongChanged”) to automatically publish messages on a particular topic.
UbaPlayer is used for playing HTML5 audio with a Flash fallback.
For styling, we use LESS, the CSS superset. We use LESS because CSS is redundant; LESS lets us use variables and functions for code reuse, but still compiles down to plain CSS.
jQuery is still there, but not as much as you might think. Because KnockoutJS handles the DOM <—> code interaction, jQuery is only needed in rare cases where we need special DOM manipulation (e.g. to fade in/out an element). We also use jQuery to do AJAX calls.
The site generally uses Twitter Bootstrap UI toolkit for styling and consistency across the UI.
We’re using Google Web Fonts for typography.
For images like thumb-up/down, play button, etc. these are actually font characters, from the special Font Awesome web font. Because it’s just a font, they scale infinitely without losing fidelity, and you can change their appearance (size, color, etc.) using plain old CSS stylings. Icon fonts are awesome.
We use Asp.NET MVC Razor view engine to actually render our UI. Because of heavy reliance on KnockoutJS, Razor isn’t needed much; it’s mostly just plain HTML that is dynamically swapped in through AJAX and KnockoutJS.
The server code is written in C# 5 using Asp.NET MVC 4 + MVC Web API.
ASP.NET Web Optimization Framework is used for minifying and bundling the scripts and CSS.
The database is RavenDB, a modern, 2nd generation document database, easily the best NoSQL database in the .NET world. I use it because it’s blazing fast and simple to just start dumping objects into.
ReactiveExtensions (RX) is used on the C# server end to treat events as observable streams, which can then write LINQ queries on top of. We also use its sister project, Interactive Extensions, for some useful extension methods to IEnumerable objects (such as .Do and .ForEach).
To load up the whole shebang and edit the project top-to-bottom, I use Visual Studio 2012 + TypeScript tools plugin (for editing/compiling TypeScript) + Web Essentials (LESS compilation, better tooling for HTML5 web standards, and more).

Holy cow!

Here’s the final tally of languages, frameworks, and tools used:

5 languages – C#, TypeScript, LESS, .cshtml Razor syntax, plus bits of plain JavaScript here and there
5 UI JavaScript frameworks – KnockoutJS, jQuery, UbaPlayer, Knockout.postbox, Twitter Bootstrap.
6 server-side .NET frameworks – RavenDB, MVC 4, MVC Web API, Reactive Extensions, Interactive Extensions, Web Optimization Framework.
3 tools – Visual Studio 2012, TypeScript tools plugin, Web Essentials plugin.

The sheer number of tools required to really build something useful on the web is staggering. You get the sense of why desktop developers — who often work in just a single language + UI framework (e.g. C# + WPF) — are hesitant to embrace the web. It requires massive retooling.

And to be truthful, this is only my beautiful web soup. I chose C# + KnockoutJS + TypeScript + LESS, but other web devs might prefer Ruby + BackboneJS + CoffeeScript + SASS. Or something entirely different.

The great thing is, there is so much evolution going on in web development space today, and this produces unique and specialized toolsets that help move the web forward. For example, languages like CoffeeScript and TypeScript are influencing the future direction of JavaScript.

It’s a fun time to be in web development.

My advice for developers? Learn a few web technologies that pique your interest and cook up your own beautiful web soup.

Stop writing for loops in Javascript

You have an array; you want to find an element in the array. What do you do?

Most developers go with the old procedural for loop:

     
var items = [        
   { Name = "Matthew", Id = 1 },         
   { Name = "Mark", Id = 2 },         
   { Name = "Luke", Id = 3 },         
   { Name = "John", Id = 4 }         
];
// Find the item with Id === 3        
var matchingItem = null;         
for (i = 0; i < items.length; i++) {         
     if (items.Id === 3) {         
       matchingItem = item;         
       break;         
     }         
}
  // Do something with matchingItem        
}

What’s wrong with this?

It’s the classic procedural programming affliction: over-specification. It painfully specifies all the steps the program must take to reach the desired state.

All you want is the item with an Id of 3, but instead you create a new loop variable, increment that variable each loop, use special loop iteration language syntax, index into the array, check if it matches your condition, store the match in a mutable variable.

The how obscures the what.

Thank you sir, may I have another.

This obfuscates the intent of your code via over-specificying pageantry.

It’s also error prone. The above code omitted the var declaration for the loop variable i, (did you catch that?), thus adding it to the global scope (oops!). Developers often forget to break; out of the for loop when a match is found, resulting in superfluous loop iterations at best, unintended behavior at worst.

Can we do better?

How about .filter?

Today’s JavaScript (1.5+) includes the built-in .filter function:

     
var items = [        
     { Name = "Matthew", Id = 1 },         
     { Name = "Mark", Id = 2 },         
     { Name = "Luke", Id = 3 },         
     { Name = "John", Id = 4 }         
  ];
  // Find the item with Id === 3        
  var matchingItem = items.filter(function(item) {         
     return item.Id === 3;         
})[0];
  // Do something with matchingItem...

.filter is a higher-order function that returns all elements in the array that match your predicate. The above code calls .filter(…) and then gets the first matching item.

This is better, but still not right.

What’s wrong with this? Like a for loop that forgot to break; out of the loop after finding a match, .filter will iterate over your entire array, even if it already found a match. This will result in superfluous iterations. And if your code doesn’t account for multiple items matching your condition, you’ll get unexpected behavior.

Solution

There’s no way to do this efficiently or succinctly using any of the built-in JavaScript array functions.

Because of this, most developers just do the ugly one, or the inefficient one.

A proper solution would be a function added to the Array prototype chain. In lieu of the browser vendors doing that, we can do it like this:

     
if (!Array.prototype.first)        
{         
   Array.prototype.first = function(predicate)         
   {         
     "use strict";    
     if (this == null)         
       throw new TypeError();       
     if (typeof predicate != "function")         
       throw new TypeError();  
      
     for (var i = 0; i < this.length; i++) {         
       if (predicate(this[i])) {         
         return match;         
       }         
     }       
     return null;         
   }         
}

Here, we’ve defined a .first method on the Array prototype. It uses the efficient-but-ugly for loop. But instead of coding up this for loop every time, we can now just call our higher-order function:

var items = [      
   { Name = "Matthew", Id = 1 },       
   { Name = "Mark", Id = 2 },       
   { Name = "Luke", Id = 3 },       
   { Name = "John", Id = 4 }       
];       
// Find the item with Id === 3       
var matching = items.first(function(i) { return i.Id === 3 });

// Do something with match...

Much cleaner, syntactically similar to .filter, but without the superfluous iterations.

JavaScript has no lambda syntax (yet), so it’s not as clear as it could be. Ideally, we’d write it more succinct, something like:


var matchingItem = items.first(item => item.Id === 3);

We’ll have to wait for the proposed arrow function syntax for the next version of JavaScript before we can get that succinct.

In the meantime, Array.prototype.first is a nice polyfill with decent syntax, allowing your code to focus on what, rather than how.

TypeScript, CoffeeScript, and the state of web development in late 2012

Summary: The problems facing web development today. Microsoft’s new TypeScript language, what problems it addresses, and what it leaves wanting.

Earlier this year, I predicted Microsoft would be utilizing their still-baking Roslyn compiler service to transform C# into targets besides IL, such as JavaScript. That prediction didn’t come true, but I was close: Microsoft is indeed getting into the compile-to-JavaScript games, but they’re doing so with a brand new language announced today, TypeScript.

JavaScript is a messy, powerful language.

The whole compile-to-JavaScript trend addresses a real problem in web development today: JavaScript just isn’t a great language for big web apps. Some argue it’s not a great language for anything besides quick DOM + event handler glue. And honestly, they kind of have a point.

Half baked, escaped the lab too early, leaving us silly things like implicit conversions for equality comparison. Quick, what’s the result of this?

foo = 0 == (100 + 23 === “123”)

If you guessed the boolean true, you’re right (I think!) but it shows kind of the trouble you get into when languages try to do too much for the developer.

But silly syntax puzzles aren’t the real reason JavaScript is bad for big apps.

No, it’s for a number of more deeply-seated reasons the difficulties arise when coding a large JS app with a team of developers:

The automatic, often accidental, inclusion in global scope. (The above snippet adds foo to the window object, woops!) This results in code littering the global namespace at best, and overwriting other code’s functionality at worst.
No modules or namespaces for code organization. In addition to contributing to the above problem, a lack of built-in modules means we have to resort to 3rd party libraries for dependency detection and loading. Often, developers will just take the easy route and load everything, resulting in web apps that load slowly and use excessive memory.
A 10-ways-to-do-this inheritance pattern that kinda sorta looks like classical inheritance but is really prototypal. Some codebases standardize on a 3rd party library (such as MooTools or Prototype), while other codebases become the Wild West of inheritance problems.
Without any types, refactoring and symbol analysis becomes painful. For example, want to find all callers of a function? Have fun with CTRL+F, and weed through who’s calling your function and who’s calling a function of the same name. Want to rename a property? CTRL+H and pray that no other code file has a property of the same name. And so on.

Plus other uglies like optional semicolons to end lines, resulting in production-level frameworks being filled with, in the words of JavaScript creator Douglas Crockford, “insanely stupid code.”

But one thing JavaScript is, is capable. It’s a functional, prototypal, object-oriented language. General purpose.

So rather than wait for JavaScript to slowly evolve through standards bodies and bickering, we build our own fast-evolving languages that compile down to JavaScript.

CoffeeScript: A (slightly) better JavaScript

At the time of this writing, the compiles-to-JavaScript language de jour is CoffeeScript: a superset of JavaScript that adds things like classes, and removes things like semicolons, parenthesis, and other staples of the C family of languages.

But CoffeeScript still lacks a proper compiler. Oh, sure, there’s the CoffeeScript compiler, but it’s not really a compiler. It can’t tell you if you’re passing the wrong type into a function. It can’t tell you if foo.Blah will error at runtime, because there’s no symbol analysis.

CoffeeScript also suffers from its straddle-the-fence stance on compatibility with JavaScript. Want to include parens? OK, valid! Want to exclude parens? OK, valid! While it sounds nice in theory, in practice you end up with an inconsistent codebase, and an awkward rule set about when optional syntax isn’t really optional.

And though CoffeeScript tried its hand at list comprehensions, it falls far short of what Python and C#/LINQ devs have been enjoying for years.

To make matters worse, there’s still no real standard library for JavaScript. Just a mash of tiny libraries that do things differently ($.map? or array.map?)

And finally, we still have terrible debugging support. Got Coffee? OK! How do I set a breakpoint on this CoffeeScript line? Silly developer, you can’t! You have to look at your Coffee, figure out which .js files were emitted, find those in your favorite web browser developer debugger tool, and break into that. Grahgghhlhlhlhsl.

End result? As of late 2012, we don’t have any great options for building big web applications. We need a language that supports symbol analysis (refactoring and the like), type error detection, modules, classes. We need a web development platform with a standard library. We need a web development platform that supports debugging in the language you authored.

Enter Microsoft’s TypeScript

TypeScript is a Microsoft attempt at solving these problems, in particular through the adding of types. It’s JavaScript with types.

Does it solve all the problems?

Too early to say.

It was announced only today, and I just downloaded the damn thing. At a glance, they solved the classes and modules problem. And by adding types and tooling support in Visual Studio, you can actually perform refactorings in your big codebase; something Java and C# developers have been enjoying for over a decade, but web developers have been suffering without.

It doesn’t appear to have a standard library – sorry, LINQ fans.

Nor does it appear to support debugging in Visual Studio.

UPDATE: The TypeScript compiler supports an experimental –sourcemap command line argument that generates a source map file. These source map files can then be used to debug TypeScript in the browser. I predict that Microsoft will use these source map files to integrate into Visual Studio’s debugging experience, likely through a special debugging browser similar to the existing Page Inspector.

So that answer for now appears to be, TypeScript is an incremental improvement upon CoffeeScript: types and modules and symbol analysis are a nice addition. We’re still missing some important things, namely, a standard library and a powerful source-language debugger. Until we get these things, web development is still a second class experience.