html5 – Debugger.Break()

It’s almost 2017, and HTML5 audio is still broken on iOS

Summary: Back in 2012, I wrote that HTML5 audio is broken on iOS. Now as we enter 2017, it turns out things are still horribly broken. It is currently impossible to build a music player web app that works on iOS.

Update June 2017: I’ve filed a bug with the iOS WebKit team to address the major blocking issue. Here’s to hoping they fix it!

Last week I published a big update to Chavah Messianic Jewish radio. It’s an HTML5 music player in the vein of Pandora (users can thumb up songs, influencing what gets played) for the Messianic Jewish faith.

And thanks to the magic of the web and HTML5 audio, it works flawlessly on PCs, Macs, and Linux. Sweet!

What about mobile? Well, yeah. Umm. Apple’s mobile implementation of HTML5 audio is still busted.

After releasing the new version last week, my users reported things still busted on iOS devices:

Well, I did have a look. And what I found is that iOS still cripples web apps that use HTML5 audio.

I was hoping that the new (July 2016) relaxed web media restrictions in iOS 10+ would un-cripple HTML5 audio in iOS.

I was disappointed to find it’s still broken. It’s currently impossible to write a working audio player using modern web technologies.

Here are the working things:

You can play MP3 audio* ** *** ****

* Only after user interaction
** Only while the page is active and in the fore
*** Only while the phone screen is on
**** There’s no way to keep the screen from turning off, so your audio will stop after the first song.

So yes, you can play audio (10 asterisks here). Or more precisely, you can play a single audio track, but not much more.

Here are the busted things:

Minor: Audio doesn’t play until user interaction.
Major: Audio can’t play the next track when the page is in the background. (JavaScript execution is suspended; no way to set audio.src for the next song.)
Major, blocking: Audio can’t play the next track when the phone screen is off.
(JavaScript execution is suspended when the phone screen is off; no way to set audio.src for the next song)

Details on each of these below:

Audio doesn’t play until user interaction

This is the most minor of the busted things. But it’s an artificial restriction by Apple, likely for user experience and battery life reasons.

None of the other operating systems, mobile or desktop, do this. So we have to have special handling for iOS to require the user to interact with the UI before playing.

As for battery life, 3d content, muted <video>, gifs, ads and more don’t require interaction before start. Why hurt real web apps and real users by singling out audio apps?

Audio can’t play next track in background

Common user scenario: they go to my web app, hit play, and the music starts streaming in. Now they switch over to Twitter. When the current song ends, the music just stops. No new track is played.

Why?

Upon investigation, HTMLAudioElement ended event is never fired. Why? Because Apple suspends all JavaScript execution in the name of performance.

This sounds good in theory: you’re not using a web site, so Safari will just stop executing any JavaScript since you’re not using it anyways.

But in practice, this kills real web apps on iOS. Music playing apps need to know when the song is finished in order to play the next song. So, we call

audio.addEventListener(“ended”, playNextSong)

But the playNextSong function never fires; JavaScript has been suspended, and users are disappointed.

Audio can’t play the next track when the phone screen is off.

The most common scenario for my app: user goes to my web app, the app starts playing music. Then, the user leaves her phone alone; perhaps she’s in the car driving while listening to the music.

After a short period of time, the phone screen turns off. The music keeps playing…until the current song ends. Once again, iOS has suspended JavaScript execution, resulting in the audio.ended event never firing, meaning I can’t set audio.src to the next song.

Upon further investigation, I tried to find out if it’s possible to prevent the screen from sleeping.

For a native iOS app, you can set application.idleTimerDisabled = YES. Super simple.

But for a web app? Nope, there’s no supported way to tell iOS, “Hey, keep the screen on, the user is in the middle of listening to music so don’t disrupt them.”

Some dated information on StackOverflow suggests looping a silent audio or video may prevent sleep. I built a little test app to try this out, and it appears to no longer work on iOS 10.

Additional answers on StackOverflow suggest pseudo page navigation every few seconds to prevent phone sleep/lock. I tried this as well, and it likewise appears to no longer work on iOS 10. The phone still sleeps even with page navigation going on.

Bottom line: there appears to be no way for a web app to prevent an iPhone from sleeping/locking.

And since sleeping/locking will cause a suspension of JavaScript execution, there’s no way to play the next song. End result is your audio web app stops playing audio, making it pretty useless.

Apple Webkit team action items

Here’s what we web developers need to make audio web apps a first class citizen on iOS:

Don’t suspend JavaScript execution for web apps playing audio. Don’t suspend JS execution if we’re in the background. Don’t suspend JS if the phone is locked. The user is playing our audio for a reason, don’t disrupt the user.

It’s not enough to let the audio finish and then suspend JS; this breaks the user experience and causes audio web apps to stop working.

This would solve the other problems.

Alternately, don’t sleep/lock the phone if an active web app is playing audio. While this alternate solution doesn’t fix all the problems, it would address the most blocking use case for using audio web apps unattended.

Lap Around HTML5 RAvenDB Studio

Last month, I was honored to give a talk at the very first (!) RavenDB conference, the first of many, as I believe Raven is poised for greatness as it gains more exposure and spreads its wings outside the .NET niche.

My talk was on the new Raven Studio – built from the ground-up using modern web technologies, HTML5, LESS, TypeScript – oh yes!

[youtube https://www.youtube.com/watch?v=wlbAj-qEuV4]

I hope you guys enjoy the new RavenDB Studio! It’s now available as unstable 3.0 build. Browse the code (or heck, send me pull requests) over at the GitHub repo.

RavenDB Studio 3.0, and why we moved from Silverlight to HTML5

Summary: A big step for RavenDB: a new HTML5 Studio. Plus, some thoughts on the move from Silverlight to HTML5 and our experience in the transition.

Yesterday, I pulled the covers off something I’ve been working on for a few months, something I’m very proud of.

RavenDB, the most popular NoSQL database in the .NET world, announced a brand new RavenDB Management Studio, Raven Studio 3.0, built from the ground up using HTML5 and modern web technologies. Yes, we’re moving away from Silverlight and onto HTML5.

Ayende and myself demoed the new Raven Studio just yesterday in a live webinar:

[youtube=http://www.youtube.com/watch?v=FNDKuiftKcc&w=448&h=252&hd=1]

This has been my pet project for the last few months, and it’s something I’m quite proud of! I believe this is a huge step forward for RavenDB (more on that in a minute), and the reception from the Raven community has been awesome, ego-stoking, and totally energizing.

The old Raven Studio was built in Silverlight. Some Silverlight fans have asked, why did we move to HTML5? Are we making a big mistake moving away from Silverlight and to HTML5?

No, on the contrary, we believe HTML5 is a damn good option.

The RavenDB community wants an HTML5 Studio. This has probably been the most-requested item from the RavenDB community. It came up multiple times in the Raven 3.0 Wishlist, it’s come up whenever Oren talks about the Studio, it comes up when we speak to the external developer community, heck, when I was in Israel for RavenDB training last year, one of the students brought it up right there in the Hibernating Rhinos office. Silverlight has served us well over the years, but Silverlight is a dying technology that our community doesn’t want to be tied to any longer.
Silverlight tooling is a perceived barrier to Raven adoption. I speak at Code Camps and user groups, and when I speak on RavenDB, the love flows and the excitement grows…until I show Silverlight tooling. I get the raised eyebrow. “Silverlight? Oh. I see.” Others in the Raven community have reported this as well. For some, Silverlight is a stumbling stone.
HTML5 is a step towards cross-platform Raven. RavenDB is the best NoSQL database for .NET. But, in time, we want Raven to spread her wings and be not just the best NoSQL solution for .NET, but the best NoSQL database, period. Moving to an HTML5 toolset is a step towards this goal.
The software industry is moving away from plugins. Plugins like Silverlight added abilities you couldn’t do on the native web, such as audio, video, gaming, 2d drawing, documents, voice, and more. Plugins filled these gaps, but with HTML5, these gaps are disappearing. We don’t need Adobe Acrobat plugins anymore to view that high fidelity document. We don’t need Java applets anymore to run that simulation. We don’t need websites built entirely with Flash. And we don’t need Silverlight for Raven Studio. There is little reason today to build something in JavaFX, Flash, or Silverlight: the native web has supplanted them. Just as it’s rare – and often undesired – to see a Java applet out in the wild, so too it will be with Silverlight in the coming years.
The native web platform is a solid foundation for the future. Microsoft products come and go. 3 years ago, Microsoft was pushing Silverlight as the platform for line of business apps and islands of richness on the web. Today, Silverlight is prevented from running in the default browser of their newest operating systems.
HTML, on the other hand, has been a stable, ever-evolving technology for decades, and because it is the very fabric of the web, things built in HTML live indefinitely. There’s a reason you can still visit and use the 17-year old Space Jam Website. But your MS Silverlight app from last year? It won’t run even on the latest MS operating system’s default browser.

As a Silverlight developer who has built professional apps (e.g. 3M Visual Attention Service) and spoken at Code Camps and user groups on Silverlight, truth be told, Silverlight is a great developer platform. C# is an excellent language, Visual Studio probably the best development environment.

But, in the words of Miguel de Icaza, creator of Moonlight (open source, cross-platform Silverlight),

“I felt that Silverlight had a bright future, and that it could turn to fill an important void, not only for web development, but for desktop development in general. And this was largely one of my motivators. I am very sad that Microsoft strategy cut the air supply to Silverlight.”

This, coupled with the mobile computing explosion and the software industry’s shift away from plugins, results in a sickly future for Silverlight and Silverlight apps.

RavenDB rocks, and we want the tooling to rock as well. Having our tooling tied to this technology was not an attractive proposition, and it was time for us to move on.

A new technology stack for Raven Studio 3.0

After much deliberation and considering all the options available to us, we moved off of Silverlight.

Instead of Silverlight, HTML5.

Instead of C#, TypeScript.

TypeScript is awesome. TypeScript is new language, a superset of JavaScript designed for building apps on the web. Silverlight fans will be happy to know it’s built by none other than Anders Heijlsberg, the much-respected language designer and author of C#.

In TypeScript, all JavaScript is valid TypeScript code, so it’s familiar to any web developer, but it gives us nice things like an optional, flexible type system, classes, modules, and enums, and features proposed for future versions of JavaScript, but compiles to plain old JavaScript that runs in every browser.

TypeScript tooling is Visual Studio, with all the nice debugging and refactoring that brings, but it can also be written in any text editor and debugged in any browser.

For infrastructure, because we wanted the look & feel of a web application, rather than a set of web pages, we opted to build a single page application (SPA). Durandal.js gives us exactly that: a nice means to load pages on demand and compose them into a cohesive web application.

For UI, Durandal uses Bootstrap for a consistent, pleasing aesthetic, and KnockoutJS for data binding and MVVM.

Using data binding, MVVM, and Durandal makes a great developer experience, one not too foreign to the MVVM stuff in Silverlight. (Indeed, the author of Durandal.js is the same author of the popular Silverlight MVVM framework Caliburn Micro.) Look at the code and judge for yourself; you’ll see classes separated out into small, logical view models, and a clean separation between view and logic.

What has been our experience moving to HTML5?

One immediate, measurable gain was performance:

Memory usage dropped from 140MB to 20MB.
Cold starts dropped from ~7s to ~2s.
Warm starts dropped from ~3s to ~1s.
General snappiness: XAML is rather heavyweight, and you’ll notice just moving around the application, loading your documents, collections, or editing – it’s all faster in HTML5. Snappy and responsive.
This doesn’t happen:
This doesn’t happen either:

A lot of the above we get for free simply by Doing Less Stuff™. No .xap files to download, no dlls to load, no CLR runtime to start, no plugin host process for the browser, no browser-to-plugin communication, no managed code to start executing.

This translates into faster start times and less memory usage.

Another free item we get is JavaScript and the blazing-hot modern JS browser runtimes. The major browsers – IE, Firefox, Chrome, Safari – are in a cut-throat competition to get the fastest JavaScript runtime, to squeeze every possible ounce of performance out of JavaScript. You’ll regularly see these browser vendors advertising their JS benchmarks as proof of performance improvement. This is a free win for the new HTML5 Raven Studio: as browser vendors continue to improve their engines in this cut-throat, cross-company competition of speed, Raven Studio will reap the performance improvements.

Moving to the native web platform fixes some plugin-induced workflow hiccups. For example, keyboard shortcuts: Silverlight and other plugins eat the keyboard. So say you’re got Raven Studio opened, and you want to open a new browser tab, so you hit CTRL+T. Surprise, nothing happens. Why? Silverlight ate your keyboard shortcuts, your browser never received them, and your workflow was just interrupted.

If you’ve ever used one of those old all-Flash websites, or full-page Java applets, you’ve probably noticed some things just don’t feel right. So it was with the old Silverlight Studio. Moving to HTML5 fixes these issues.

Conclusion

Transitioning out of the plugin ghetto and moving to HTML5 has been a delight, but more importantly, it’s good for RavenDB users as we move to a faster, more lightweight tool. It’s good for the future of RavenDB to have our tooling built on the solid rock of the native web.

I understand the Silverlight fans who are sad to see the old Silverlight Studio go. I’m a Silverlight fan myself, I understand their concerns. The most I can ask of you guys is to give us the opportunity to earn your trust. It will take time, but with a faster, more lightweight, stable tool that does what you need and gets out of your way, I believe that trust will be earned.

The new HTML5 Raven Studio is on GitHub and we’d love for you to give it a spin or even contribute to the code. I’m pleased to say we already have had a few contributions since it was released just yesterday. I’m proud of this work, and I really hope you guys enjoy it!

My beautiful web development soup

Summary: Web development is chaotic, overwhelming, and beautiful. How many technologies does it take to build something useful on the web? You might be surprised. Advice for developers getting into web development.

This week I was working on my open source startup project, BitShuva Radio. I acquired a new client this week, but this client was unique because he’s a Java developer and wanted to know how to edit the open source code for his new radio station.

As I typed out my answer, it dawned on me that the sheer number of technologies utilized in building a useful web application is staggering. That they actually work together is a testament of the beautiful soup that is web development today.

How many technologies does it take to build something useful on the web? Here’s my answer:

I use TypeScript, rather than plain JavaScript, for all client-side code. A superset of JavaScript with optional types, TypeScript is a well-structured, typed language with excellent tooling. I use it for practical reasons: as the app grew in complexity, the need for a more structured language became apparent. I ported the JS to TypeScript last month, and I’m quite pleased with the results.
KnockoutJS is ~~used heavily~~ absolutely relied on. BitShuva is a dynamic single page application, meaning we’re doing lots of dynamic updates to the UI in real time. Knockout lets us make dynamic UIs very easily without resorting to tons of DOM manipulation code. Instead of DOM manipulation, Knockout lets you change your objects, and your UI automatically updates. For example, songsOnScreen.push(new Song()) will automatically show the song in the UI, no jQuery DOM manipulation required.
I use knockout.postbox for decoupled pub/sub communication between client-side code components. For example, when I hit the play button, the click handler doesn’t need to know about HTML5 audio or any infrastructure concerns. Instead, it calls ko.postbox.publish(“Play”). Whoever is in charge of playing audio simply calls ko.postbox.subscribe(“Play”, …). This way, the click handler and the HTML5 audio component don’t need to know about each other. Nice and clean. Bonus: it works with Knockout observables, so you can say currentSong.publishOn(“SongChanged”) to automatically publish messages on a particular topic.
UbaPlayer is used for playing HTML5 audio with a Flash fallback.
For styling, we use LESS, the CSS superset. We use LESS because CSS is redundant; LESS lets us use variables and functions for code reuse, but still compiles down to plain CSS.
jQuery is still there, but not as much as you might think. Because KnockoutJS handles the DOM <—> code interaction, jQuery is only needed in rare cases where we need special DOM manipulation (e.g. to fade in/out an element). We also use jQuery to do AJAX calls.
The site generally uses Twitter Bootstrap UI toolkit for styling and consistency across the UI.
We’re using Google Web Fonts for typography.
For images like thumb-up/down, play button, etc. these are actually font characters, from the special Font Awesome web font. Because it’s just a font, they scale infinitely without losing fidelity, and you can change their appearance (size, color, etc.) using plain old CSS stylings. Icon fonts are awesome.
We use Asp.NET MVC Razor view engine to actually render our UI. Because of heavy reliance on KnockoutJS, Razor isn’t needed much; it’s mostly just plain HTML that is dynamically swapped in through AJAX and KnockoutJS.
The server code is written in C# 5 using Asp.NET MVC 4 + MVC Web API.
ASP.NET Web Optimization Framework is used for minifying and bundling the scripts and CSS.
The database is RavenDB, a modern, 2nd generation document database, easily the best NoSQL database in the .NET world. I use it because it’s blazing fast and simple to just start dumping objects into.
ReactiveExtensions (RX) is used on the C# server end to treat events as observable streams, which can then write LINQ queries on top of. We also use its sister project, Interactive Extensions, for some useful extension methods to IEnumerable objects (such as .Do and .ForEach).
To load up the whole shebang and edit the project top-to-bottom, I use Visual Studio 2012 + TypeScript tools plugin (for editing/compiling TypeScript) + Web Essentials (LESS compilation, better tooling for HTML5 web standards, and more).

Holy cow!

Here’s the final tally of languages, frameworks, and tools used:

5 languages – C#, TypeScript, LESS, .cshtml Razor syntax, plus bits of plain JavaScript here and there
5 UI JavaScript frameworks – KnockoutJS, jQuery, UbaPlayer, Knockout.postbox, Twitter Bootstrap.
6 server-side .NET frameworks – RavenDB, MVC 4, MVC Web API, Reactive Extensions, Interactive Extensions, Web Optimization Framework.
3 tools – Visual Studio 2012, TypeScript tools plugin, Web Essentials plugin.

The sheer number of tools required to really build something useful on the web is staggering. You get the sense of why desktop developers — who often work in just a single language + UI framework (e.g. C# + WPF) — are hesitant to embrace the web. It requires massive retooling.

And to be truthful, this is only my beautiful web soup. I chose C# + KnockoutJS + TypeScript + LESS, but other web devs might prefer Ruby + BackboneJS + CoffeeScript + SASS. Or something entirely different.

The great thing is, there is so much evolution going on in web development space today, and this produces unique and specialized toolsets that help move the web forward. For example, languages like CoffeeScript and TypeScript are influencing the future direction of JavaScript.

It’s a fun time to be in web development.

My advice for developers? Learn a few web technologies that pique your interest and cook up your own beautiful web soup.

Apple: Please fix HTML5 on iOS

I’ve built an HTML5 audio player, Chavah Messianic Jewish Radio, that “works” on iOS.

By “works“, I mean, it plays the best it can on Apple’s crippled iOS <audio> implementation. At the time of this writing, this includes iOS 5, tested on iPhone 3 and 4, and iPad1 through iPad 3.

How is <audio> broken on iOS? Let me count the ways:

Apple does not let you play audio until user interaction. For me, this means detecting iOS, then showing the music as paused until the user clicks play. Their reasoning is that this will consume data and battery; in practice, though, it cripples web apps, forcing us to resort to platform-specific hoop-jumping to get the web to work on your platform, Apple. It effectively stifles the evolution of the web in the same way IE did in the late 1990s.
Apple does not allow you to play concurrent audio files. This is a huge crippler for gaming apps that need to play sound effects, for example.
Apple doesn’t support OGG format. Patent-encumbered formats work, of course, but without support for free and open formats like OGG, Apple is effectively creating a long-term thorn in the side of the opened web.
Every dynamic <audio> element must receive user interaction. If you want to play successive sounds (one after another) without receiving user interaction for each one, you must use a single <audio> element, then do the iOS <audio> dance: set existingAudio.src, then call existingAudio.load(), then call existingAudio.play().
Audio events don’t fire unless Safari is in the fore. While audio will continue playing if the user switches to a different app, the .ended event won’t fire. This means it’s practically impossible to build a music player web app.

Apple, please, please, please give us better support for HTML5 in iOS Safari. Here are your action items, Apple:

Let HTML5 audio work in the background.
Support OGG.
Support pre-loading audio.
Support concurrent audio.
Let us play audio without user interaction.

Do those things, Apple, you’ll be the industry leader in mobile HTML5 audio, everyone will emulate you, you’ll once again be leading the way, and HTML5 web apps will work best on your platforms. Wouldn’t that be good for Apple?

Yes, these features will use data and the battery, but native apps already do this, so why cripple web apps? Stop stifling the evolution of the web, and start being the leader. Make HTML5 <audio> a first-class citizen on iOS.