RavenDB.Identity – an ASP.NET Core Identity provider for RavenDB

I’ve recently been doing some greenfield development with ASP.NET Core; I feel the framework, now in version 2, is sufficiently stablized to warrant new work in.

As my database of choice is RavenDB, I want to store my users, logins, and claims/roles in the database. For that, I’ve built the NuGet package RavenDB.Identity. While there are some existing packages for doing Identity with RavenDB, I designed my package to be easy to use and get-out-of-the-way:

Nice and simple, eh?

Once it’s setup, you use [Authorize] just like normal:

Likewise, signing in uses the built-in Identity APIs:

More details, and a sample project, over at the RavenDB.Identity GitHub repo. Enjoy!

Making TypeScript async/await play nice with AngularJS 1.x, even on old ES5 browsers

Summary: How to use TypeScript async/await with AngularJS 1.x apps, compiling down to ES5 browsers.

With TypeScript 2.1+, you can start using the awesome new async/await functionality today, even if your users are running old browsers. TypeScript will compile it down to something all browsers can run. Hot smile

I’m using Angular 1.x for many of my apps. I wanted to use the sexy new async/await functionality in my Angular code. I didn’t find any examples online how to do this, so I did some experimenting and figured it out.

For the uninitiated, async/await is a big improvement on writing clean async code:

Getting this to work with Angular is pretty simple, requiring only a single step.

1. Use $q for Promise

Since older browsers may not have a global Promise object, we need to drop in a polyfill. Fortunately, we can just use Angular’s $q object as the Promise, as it’s A+ compatible with the Promise standard.

This kills two birds with one stone: we now have a Promise polyfill, and when these promises resolve, the scope will automatically be applied.

2. You’re done! Sort of…

That’s actually enough to start using async/await against Promise-based code, such as ng.IPromise<T>:

Cool. We’re cooking with gas. Except…

Making it cleaner.

If you look at the transpiled javascript, you’ll see that TypeScript is generating 2 big helper functions at the top of every file that uses an async function:

Yikes! Sure, this is how the TypeScript compiler is working its magic: simulating async/await on old platforms going back to IE8 (and earlier?).

Love the magic, but hate the duplication; we’re generating this magic for every TS file that uses async functions. Ideally, we’d just generate the magic once, and have all our async functions reuse it.

We can do just that, explained in steps 3 and 4 below.

3. Use noEmitHelpers TS compiler flag

The TypeScript 2.1+ compiler supports the noEmitHelpers flag. This will isntruct TypeScript not to emit any of its helpers: not for async, not for generators, not for class inheritance, …nuttin’.

Let’s start with that. In my tsconfig.json file, I add the flag:

You can see we’ve set noEmitHelpers to true in line 8. Now if we compile our app, you’ll notice the transpiled UsersController.js (and your code files that use async functions) no longer has all the magic transpiler stuff. Instead, you’ll notice your async functions are compiled down to something like this:

Ok – that actually looks fairly clean. Except if you run it, you’ll get an error saying __awaiter is undefined. And that’s because we just told TypeScript to skip generating the __awaiter helper function.

Instead of having TypeScript compiler generate that in each file, we’re just going to define those magic helper functions once.

4. Use TsLib.js to define the magic helper functions once.

Microsoft maintains tslib, the runtime helpers library for TypeScript apps. It’s all contained in tslib.js, single small file (about 200 lines of JS) that defines all helper functions TypeScript can emit. I added this file to my project, and now all my async calls work again. Party smile

Alternately, you can tell the TypeScript compiler to do that for you using the importHelpers flag.

It’s almost 2017, and HTML5 audio is still broken on iOS

Summary: Back in 2012, I wrote that HTML5 audio is broken on iOS. Now as we enter 2017, it turns out things are still horribly broken. It is currently impossible to build a music player web app that works on iOS.


Update June 2017: I’ve filed a bug with the iOS WebKit team to address the major blocking issue. Here’s to hoping they fix it!

Last week I published a big update to Chavah Messianic Jewish radio. It’s an HTML5 music player in the vein of Pandora (users can thumb up songs, influencing what gets played) for the Messianic Jewish faith.

chavahv3-small

And thanks to the magic of the web and HTML5 audio, it works flawlessly on PCs, Macs, and Linux. Sweet!

What about mobile? Well, yeah. Umm. Apple’s mobile implementation of HTML5 audio is still busted.

After releasing the new version last week, my users reported things still busted on iOS devices:

image

Well, I did have a look. And what I found is that iOS still cripples web apps that use HTML5 audio.

I was hoping that the new (July 2016) relaxed web media restrictions in iOS 10+ would un-cripple HTML5 audio in iOS.

I was disappointed to find it’s still broken. It’s currently impossible to write a working audio player using modern web technologies.

Here are the working things:

  1. You can play MP3 audio* ** *** ****

* Only after user interaction
** Only while the page is active and in the fore
*** Only while the phone screen is on
**** There’s no way to keep the screen from turning off, so your audio will stop after the first song.

So yes, you can play audio (10 asterisks here). Or more precisely, you can play a single audio track, but not much more.

Here are the busted things:

  1. Minor: Audio doesn’t play until user interaction.
  2. Major: Audio can’t play the next track when the page is in the background. (JavaScript execution is suspended; no way to set audio.src for the next song.)
  3. Major, blocking: Audio can’t play the next track when the phone screen is off.
    (JavaScript execution is suspended when the phone screen is off; no way to set audio.src for the next song)

Details on each of these below:

Audio doesn’t play until user interaction

This is the most minor of the busted things. But it’s an artificial restriction by Apple, likely for user experience and battery life reasons.

None of the other operating systems, mobile or desktop, do this. So we have to have special handling for iOS to require the user to interact with the UI before playing.

As for battery life, 3d content, muted <video>, gifs, ads and more don’t require interaction before start. Why hurt real web apps and real users by singling out audio apps?

Audio can’t play next track in background

Common user scenario: they go to my web app, hit play, and the music starts streaming in. Now they switch over to Twitter. When the current song ends, the music just stops. No new track is played.

Why?

Upon investigation, HTMLAudioElement ended event is never fired. Why? Because Apple suspends all JavaScript execution in the name of performance.

This sounds good in theory: you’re not using a web site, so Safari will just stop executing any JavaScript since you’re not using it anyways.

But in practice, this kills real web apps on iOS. Music playing apps need to know when the song is finished in order to play the next song. So, we call

audio.addEventListener(“ended”, playNextSong)

But the playNextSong function never fires; JavaScript has been suspended, and users are disappointed.

Audio can’t play the next track when the phone screen is off.

The most common scenario for my app: user goes to my web app, the app starts playing music. Then, the user leaves her phone alone; perhaps she’s in the car driving while listening to the music.

After a short period of time, the phone screen turns off. The music keeps playing…until the current song ends. Once again, iOS has suspended JavaScript execution, resulting in the audio.ended event never firing, meaning I can’t set audio.src to the next song.

Upon further investigation, I tried to find out if it’s possible to prevent the screen from sleeping.

For a native iOS app, you can set application.idleTimerDisabled = YES. Super simple.

But for a web app? Nope, there’s no supported way to tell iOS, “Hey, keep the screen on, the user is in the middle of listening to music so don’t disrupt them.”

Some dated information on StackOverflow suggests looping a silent audio or video may prevent sleep. I built a little test app to try this out, and it appears to no longer work on iOS 10.

Additional answers on StackOverflow suggest pseudo page navigation every few seconds to prevent phone sleep/lock. I tried this as well, and it likewise appears to no longer work on iOS 10. The phone still sleeps even with page navigation going on.

Bottom line: there appears to be no way for a web app to prevent an iPhone from sleeping/locking.

And since sleeping/locking will cause a suspension of JavaScript execution, there’s no way to play the next song. End result is your audio web app stops playing audio, making it pretty useless.

Apple Webkit team action items

Here’s what we web developers need to make audio web apps a first class citizen on iOS:

  • Don’t suspend JavaScript execution for web apps playing audio. Don’t suspend JS execution if we’re in the background. Don’t suspend JS if the phone is locked. The user is playing our audio for a reason, don’t disrupt the user.

It’s not enough to let the audio finish and then suspend JS; this breaks the user experience and causes audio web apps to stop working.

This would solve the other problems.

Alternately, don’t sleep/lock the phone if an active web app is playing audio. While this alternate solution doesn’t fix all the problems, it would address the most blocking use case for using audio web apps unattended.

Enabling TypeScript 2.0 strict null checks in a Visual Studio project

TypeScript 2.0 beta was released today. Among the big list of 2.0 awesomeness, the headline feature is non-null types.

Non-null types is currently opt-in: you pass a –strictNullChecks flag to the compiler to enable non-null types.

Enabling this feature in a Visual Studio project wasn’t obvious to me. This post shows how to do that.

Once you’ve downloaded the TypeScript 2.0 beta for VS 2015, open your project in Visual Studio. It will prompt you to upgrade the project to the new TypeScript tooling.

Once you’ve done this, open your .csproj in a text editor. Scroll down and find the TypeScript property group:

You’ll want to add the line:

<TypeScriptStrictNullChecks>True</TypeScriptStrictNullChecks>

inside the property group, as shown above.

Save the .csproj, reload it in Visual Studio, and the feature will be enabled.

Bonus: this is also how you’d enable the new 2.0 compiler flags, noUnusedParameters and noUnusedLocals:

<TypeScriptNoUnusedParameters>True</TypeScriptNoUnusedParameters>
<TypeScriptNoUnusedLocals>True</TypeScriptNoUnusedLocals>

Working with View Models in RavenDB

Summary: RavenDB opens up some new possibilities for working with view models: objects that contain pieces of data from other objects. With Raven, we can use .Include to simulate relational joins. But we can go beyond that for superior performance by exploiting Raven’s transformers, custom indexes, and even complex object storage.

Modern app developers often work with conglomerations of data cobbled together to display a UI page. For example, you’ve got a web app that displays tasty recipes, and on that page you also want to display the author of the recipe, a list of ingredients and how much of each ingredient. Maybe comments from users on that recipe, and more. We want pieces of data from different objects, and all that on a single UI page.

For tasks like this, we turn to view models. A view model is a single object that contains pieces of data from other objects. It contains just enough information to display a particular UI page.

In relational databases, the common way to create view models is to utilize multiple JOIN statements to piece together disparate data to compose a view model.

But with RavenDB, we’re given new tools which enable us to work with view models more efficiently. For example, since we’re able to store complex objects, even full object graphs, there’s less need to piece together data from different objects. This opens up some options for efficiently creating and even storing view models. Raven also gives us some new tools like Transformers that make working with view models a joy.

In this article, we’ll look at a different ways to work with view models in RavenDB. I’ll also give some practical advice on when to use each approach.

The UI

We’re building a web app that displays tasty recipes to hungry end users. In this article, we’ll be building a view model for a UI page that looks like this:

Figure 1

At first glance, we see several pieces of data from different objects making up this UI.

  • Name and image from a Recipe object
  • List of Ingredient objects
  • Name and email of a Chef object, the author of the recipe
  • List of Comment objects
  • List of categories (plain strings) to help users find the recipe

A naive implementation might query for each piece of data independently: a query for the Recipe object, a query for the Ingredients, and so on.

This has the downside of multiple trips to the database and implies performance overhead. If done from the browser, we’re looking at multiple trips to the web server, and multiple trips from the web server to the database.

A better implementation makes a single call to the database to load all the data needed to display the page. The view model is the container object for all these pieces of data. It might look something like this:

How do we populate such a view model from pieces of data from other objects?

How we’ve done it in the past

In relational databases, we tackle this problem using JOINs to piece together a view model on the fly:

It’s not particularly beautiful, but it works. This pseudo code could run against some object-relational mapper, such as Entity Framework, and gives us our results back.

However, there are some downsides to this approach.

  • Performance: JOINs and subqueries often have a non-trivial impact on query times. While JOIN performance varies per database vendor, per the type of column being joined on, and whether there are indexes on the appropriate columns, there is nonetheless a cost associated with JOINs and subqueries. Queries with multiple JOINs and subqueries only add to the cost. So when your user wants the data, we’re making him wait while we perform the join.
  • DRY modeling: JOINs often require us to violate the DRY (Don’t Repeat Yourself) principle. For example, if we want to display Recipe details in a different context, such as a list of recipe details, we’d likely need to repeat our piece-together-the-view-model JOIN code for each UI page that needs our view model.

Can we do better with RavenDB?

Using .Include

Perhaps the easiest and most familiar way to piece together a view model is to use RavenDB’s .Include.

In the above code, we make a single remote call to the database and load the Recipe and its related objects.

Then, after the Recipe returns, we can call session.Load to fetch the already-loaded related objects from memory.

This is conceptually similar to a JOIN in relational databases. Many devs new to RavenDB default to this pattern out of familiarity.

Better modeling options, fewer things to .Include

One beneficial difference between relational JOINs and Raven’s .Include is that we can reduce the number of .Include calls due to better modeling capabilities. RavenDB stores our objects as JSON, rather than as table rows, and this enables us to store complex objects beyond what is possible in relational table rows. Objects can contain encapsulated objects, lists, and other complex data, eliminating the need to .Include related objects.

For example, logically speaking, .Ingredients should be encapsulated in a Recipe, but relational databases don’t support encapsulation. That is to say, we can’t easily store a list of ingredients per recipe inside a Recipe table. Relational databases would require us to split a Recipe’s .Ingredients into an Ingredient table, with a foreign key back to the Recipe it belongs to. Then, when we query for a recipe details, we need to JOIN them together.

But with Raven, we can skip this step and gain performance. Since .Ingredients should logically be encapsulated inside a Recipe, we can store them as part of the Recipe object itself, and thus we don’t have to .Include them. Raven allows us to store and load Recipe that encapsulate an .Ingredients list. We gain a more logical model, we gain performance since we can skip the .Include (JOIN in the relational world) step, and our app benefits.

Likewise with the Recipe’s .Categories. In our Tasty Recipes app, we want each Recipe to contain a list of categories. A recipe might contain categories like [“italian”, “cheesy”, “pasta”]. Relational databases struggle with such a model: we’d have to store the strings as a single delimited string, or as an XML data type or some other non-ideal solution. Each has their downsides. Or, we might even create a new Categories table to hold the string categories, along with a foreign key back to their recipe. That solution requires an additional JOIN at query time when querying for our RecipeViewModel.

Raven has no such constraints. JSON documents tend to be a better storage format than rows in a relational table, and our .Categories list is an example. In Raven, we can store a list of strings as part of our Recipe object; there’s no need to resort to hacks involving delimited fields, XML, or additional tables.

RavenDB’s .Include is an improvement over relational JOINs. Combined with improved modeling, we’re off to a good start.

So far, we’ve looked at Raven’s .Include pattern, which is conceptually similiar to relational JOINs. But Raven gives us additional tools that go above and beyond JOINs. We discuss these below.

Transformers

RavenDB provides a means to build reusable server-side projections. In RavenDB we call these Transformers. We can think of transformers as a C# function that converts an object into some other object. In our case, we want to take a Recipe and project it into a RecipeViewModel.

Let’s write a transformer that does just that:

In the above code, we’re accepting a Recipe and spitting out a RecipeViewModel. Inside our Transformer code, we can call .LoadDocument to load related objects, like our .Comments and .Chef. And since Transformers are server-side, we’re not making extra trips to the database.

Once we’ve defined our Transformer, we can easily query any Recipe and turn it into a RecipeViewModel.

This code is a bit cleaner than calling .Include as in the previous section; there are no more .Load calls to fetch the related objects.

Additionally, using Transformers enables us to keep DRY. If we need to query a list of RecipeViewModels, there’s no repeated piece-together-the-view-model code:

Storing view models

Developers accustomed to relational databases may be slow to consider this possibility, but with RavenDB we can actually store view models as-is.

It’s certainly a different way of thinking. Rather than storing only our domain roots (Recipes, Comments, Chefs, etc.), we can also store objects that contain pieces of them. Instead of only storing models, we can also store view models.

This technique has benefits, but also trade-offs:

  • Query times are faster. We don’t need to load other documents to display our Recipe details UI page. A single call to the database with zero joins – it’s a beautiful thing!
  • Data duplication. We’re now storing Recipes and RecipeViewModels. If an author changes his recipe, we may need to also update the RecipeViewModel. This shifts the cost from query times to write times, which may be preferrable in a read-heavy system.

The data duplication is the biggest downside. We’ve effectively denormalized our data at the expense of adding redundant data. Can we fix this?

Storing view models + syncing via RavenDB’s Changes API

Having to remember to update RecipeViewModels whenever a Recipe changes is error prone. Responsibility for syncing the data is now in the hands of you and the other developers on your team. Human error is almost certain to creep in — someone will write new code to update Recipes and forget to also update the RecipeViewModels — we’ve created a pit of failure that your team will eventually fall into.

We can improve on this situation by using RavenDB’s Changes API. With Raven’s Changes API, we can subscribe to changes to documents in the database. In our app, we’ll listen for changes to Recipes and update RecipeViewModels accordingly. We write this code once, and future self and other developers won’t need to update the RecipeViewModels; it’s already happening ambiently through the Changes API subscription.

The Changes API utilizes Reactive Extensions for a beautiful, fluent and easy-to-understand way to listen for changes to documents in Raven. Our Changes subscription ends up looking like this:

Easy enough. Now whenever a Recipe is added, updated, or deleted, we’ll get notified and can update the stored view model accordingly.

Indexes for view models: let Raven do the hard work

One final, more advanced technique is to let Raven do the heavy lifting in mapping Recipes to RecipeViewModels.

A quick refresher on RavenDB indexes: in RavenDB, all queries are satisfied by an index. For example, if we query for Recipes by .Name, Raven will automatically cretate an index for Recipes-by-name, so that all future queries will return results near instantly. Raven then intelligently manages the indexes it’s created, throwing server resources behind the most-used indexes. This is one of the secrets to RavenDB’s blazing fast query response times.

RavenDB indexes are powerful and customizable. We can piggy-back on RavenDB’s indexing capabilities to generate RecipeViewModels for us, essentially making Raven do the work for us behind the scenes.

First, let’s create a custom RavenDB index:

In RavenDB, we use LINQ to create indexes. The above index tells RavenDB that for every Recipe, we want to spit out a RecipeViewModel.

This index definition is similiar to our transformer definition. A key difference, however, is that the transformer is applied at query time, whereas the index is applied asynchronously in the background as soon as a change is made to a Recipe. Queries run against the index will be faster than queries run against the transformer: the index is giving us pre-computed RecipeViewModels, whereas our transformer would create the RecipeViewModels on demand.

Once the index is deployed to our Raven server, Raven will store a RecipeViewModel for each Recipe.

Querying for our view models is quite simple and we’ll get results back almost instantaneously, as the heavy lifting of piecing together the view model has already been done.

Now whenever a Recipe is created, Raven will asynchronously and intelligently execute our index and spit out a new RecipeViewModel. Likewise, if a Recipe, Comment, or Chef is changed or deleted, the corresponding RecipeViewModel will automatically be updated. Nifty!

Storing view models is certainly not appropriate for every situation. But some apps, especially read-heavy apps with a priority on speed, might benefit from this option. I like that Raven gives us the freedom to do this when it makes sense for our apps.

Conclusion

In this article, we looked at using view models with RavenDB. Several techniques are at our disposal:

  • .Include: loads multiple related objects in a single query.
  • Transformers: reusable server-side projections which transform Recipes to RecipeViewModels.
  • Storing view models: Essentially denormalization. We store both Recipes and RecipeViewModels. Allows faster read times at the expense of duplicated data.
  • Storing view models + .Changes API: The benefits of denormalization, but with code to automatically sync the duplicated data.
  • Indexes: utilize RavenDB’s powerful indexing to let Raven denormalize data for us automatically, and automatically keeping the duplicated data in sync. The duplicated data is stashed away as fields in an index, rather than as distinct documents.

For quick and dirty scenarios and one-offs, using .Include is fine. It’s the most common way of piecing together view models in my experience, and it’s also familiar to devs with relational database experience. And since Raven allows us to store things like nested objects and lists, there is less need for joining data; we can instead store lists and encapsulated objects right inside our parent objects where it makes sense to do so.

Transformers are the next widely used. If you find yourself converting Recipe to RecipeViewModel multiple times in your code, use a Transformer. They’re easy to write, typically small, and familiar to anyone with LINQ experience. Using them in your queries is a simple one-liner that keeps your query code clean and focused.

Storing view models is rarely used, in my experience, but it can come in handy for read-heavy apps or for UI pages that need to be blazing fast. Pairing this with the .Changes API is an appealing way to automatically keep Recipes and RecipeViewModels in sync.

Finally, we can piggy-back off Raven’s powerful indexing feature to have Raven automatically create, store, and synchronize both RecipeViewModels for us. This has a touch of magical feel to it, and is an attractive way to get great performance without having to worry about keeping denormalized data in sync.

Using these techniques, RavenDB opens up some powerful capabilities for the simple view model. App performance and code clarity benefit as a result.

Make it modern: Web dev with RavenDB, Angular, TypeScript

Summary: A modern dev stack for modern web apps. See how I built a new web app using RavenDB, Angular, Bootstrap and TypeScript. Why these tools are an excellent choice for modern web dev.

image

Twin Cities Code Camp (TCCC) is the biggest developer event in Minnesota. I’ve written about it before: it’s a free event where 500 developers descend on the University of Minnesota to attend talks on software dev, learn new stuff, have fun, and become better at the software craft.

I help run the event and this April, we are hosting our 20th event. 20 events in 10 years. Yes, we’ve been putting on Code Camps for a decade! That’s practically an eternity in internet years. Smile

For the 20th event, we thought it was time to renovate the website. Our old website had been around since about 2006 – a decade old – and the old site was showing its age:

Screenshot of old Code Camp website

It got the job done, but admittedly it’s not a modern site. Rather plain Jane; it didn’t reflect the awesome event that Code Camp is. We wanted something fresh and new for our 20th event.

On the dev side, everything on the old site was hard-coded – no database – meaning every time we wanted to host a new event, add speakers or talks or bios, we had to write code and add new HTML pages. We wanted something modern, where events and talks and speakers and bios are all stored in a database that drives the whole site.

Dev Stack

Taking at stab at rewriting the TCCC websiite, I wanted to really make it a web app. That is, I want it database-driven, I want some dynamic client-side functionality; things like letting speakers upload their slides, letting attendees vote on talks, having admins login to add new talks, etc. This requires something more than a set of static web pages.

Additionally, most of the people attending Code Camp will be looking at this site on their phone or tablet. I want to build a site that looks great on mobile.

To build amazing web apps, I turn to my favorite web dev stack:

  • RavenDB – the very best get-shit-done database. Forget tables and sprocs and schemas. Just store your C# objects without having to map them to tables and columns and rows. Query them with LINQ when you’re ready.
  • AngularJS – front-end framework for building dynamic web apps. Transforms your website from a set of static pages to a coherent web application with client-side navigation, routing, automatic updates with data-binding, and more awesomeness. Turns this:
    $(".contentWrapper.container").attr("data-item-text", "zanzibar");

    into this:

    this.itemText = "zanzibar";

     

  • Bootstrap – CSS to make it all pretty, make it consistent, and make it look great on mobile devices. Turns this: image
    …into this:
    image
  • TypeScript – JavaScript extended with optional types, classes, and features from the future. This lets me build clean, easily refactored, maintainable code that runs in your browser. So instead of this ugly JavaScript code:image…we instead write this nice modern JavaScript + ES6 + types code, which is TypeScript:

    image

  • ASP.NET Web API – Small, clean RESTful APIs in elegant, asynchronous C#. Your web app talks to these to get the data from the database and display the web app.
  • FontAwesome – Icons for your app. Turns this: image into this:image

I find these tools super helpful and awesome and I’m pretty darn productive with them. I’ve used this exact stack to build all kinds of apps, both professional and personal:

And a bunch of internal apps at my current employer, 3M, use this same stack internally. I find this stack lets me get stuff done quickly without much ceremony and I want to tell you why this stack works so well for me in this post.

RavenDB

I’m at the point in my career that I don’t put up with friction. If there is friction in my workflow, it slows me down and I don’t get stuff done.

RavenDB is a friction remover.

My old workflow, when I was a young and naïve developer, went like this:

  1. Hmm, I think I need a database.
  2. Well, I’d guess I’d better create some tables.
  3. I should probably create columns with the right types for all these tables.
  4. Now I need to save my C# objects to the database. I guess I’ll use an object-relational mapper, like LINQ-to-SQL. Or maybe NHibernate. Or maybe Entity Framework.
  5. Now I’ve created mappings for my C# objects to my database. Of course, I have to massage those transitions; you can’t really do inheritance or polymorphism in SQL. Heck, my object contains a List<string>, and even something that simple doesn’t map well to SQL. (Do I make those list of strings its own table with foreign key? Or combine them into a comma-separated single column, and rehydrate them on query? Or…ugh.)
  6. Hmm, why is db.Events.Where(…) taking forever? Oh, yeah, I didn’t add any indexes. Crap, it’s doing a full table scan. What’s the syntax for that again?

This went on and on. It wasn’t until I tried RavenDB did I see that all this friction is not needed.

SQL databases were built for a different era in which disk space was at a premium; hence normalization and varchar and manual indexes. This comes at a cost: big joins, manual denormalization for speed. Often times on big projects, we have DBAs building giant stored procedures with all kinds of temp tables and weird optimization hacks to make shit fast.

Forget all that.

With Raven, you just store your stuff. Here’s how I store a C# object that contains a list of strings:

db.Store(codeCampTalk);

Notice I didn’t have to create tables. Notice I didn’t have to create foreign key relationships. Notice I didn’t have to create columns with types in tables. Notice I didn’t have to tell it how to map codeCampTalk.Tags – a list of strings – to the database.

Raven just stores it.

And when we’re ready to query, it looks like this:

db.Query<Talk>().Where(t => t.Author == "Jon Skeet");

Notice I didn’t have to do any joins; unlike SQL, Raven supports encapsulation. Whether that’s a list of strings, a single object inside another object, or a full list of objects. Honey Badger Raven don’t care.

And notice I didn’t have to create any indexes. Raven is smart about this: it creates an index for every query automatically. Then it uses machine learning – a kind of AI for your database – to optimize the ones you use the most. If I’m querying for Talks by .Author, Raven keeps that index hot for me. But if I query for Talks by .Bio infrequently, Raven will devote server resources – RAM and CPU and disk – to more important things.

It’s self optimizing. And it’s friggin’ amazing.

The end result is your app remains fast, because Raven is responding to how it’s being used and optimizing for that.

And I didn’t have to do anything make that happen. I just used it.

Zero friction. I love RavenDB.

If you’re doing .NET, there really is no reason you shouldn’t be using it. Having built about 10 apps in the last 2 years, both professional and side projects, I have not found a case where Raven is a bad fit. I’ve stopped defaulting to crappy defaults. I’ve stopped defaulting to SQL and Entity Framework. It’s not 1970 anymore. Welcome to modern, flexible, fast databases that work with you, reduce friction, work with object oriented languages and optimize for today’s read-heavy web apps.

AngularJS

In the bad ol’ days, we’d write front-end code like this:

image

JavaScript was basically used to wire up event handlers. And we’d do some postback to the server, which reloaded the page in the browser with the new email address. Yay, welcome to 1997.

Then, we discovered JQuery, which was awesome. We realized that the browser was fully programmable and JQuery made it a joy to program it. So instead of doing postbacks and having static pages, we could just update the DOM, and the user would see the new email address:

image

And that was good for users, because the page just instantly updated. Like apps do. No postback and full page refresh; they click the button and instantly see the results.

This was well and good. Until our code was pretty ugly. I mean, look at it. DOM manipulation isn’t fun and it’s error prone. Did I mention it was ugly?

What if we could do something like this:

image

image

Whoa! Now anytime we update a JavaScript variable, .emailAddress, the DOM instantly changes! Magic!

No ugly DOM manipulation, we just change variables and the browser UI updates instantly.

This is data-binding, and when we discovered it in the browser, all kinds of frameworks popped up that let you do this data-binding. KnockoutJS, Ember, Backbone, and more.

This was all well and good until we realized that while data-binding is great, it kind of sucks that we still have full page reloads when moving from page to page. The whole app context is gone when the page reloads.

What if we could wrap up the whole time a user is using our web app into a cohesive set of views which the user navigates to without losing context. And instead of a mess of JavaScript, what if we made each view have its own class. That class has variables in it, and the view data-binds to those variables. And that class, we’ll call it a controller, loads data from the server using classes called services. Now we’re organized and cooking with gas.

Enter AngularJS. Angular makes it a breeze to build web apps with:

  • Client-side navigation. This means as the app moves between pages, say between the Schedule and Talks pages, your app hasn’t been blown away. You still have all the variables and state and everything still there.
  • Data-binding. You put your variables in a class called a controller:
    image
    …and then in your HTML, you data-bind to those variables:
    imageThen, any time you change the variables, the DOM – that is, the web browser UI, automatically updates. Fantastic.

Angular also add structure. You load data using service classes. Those services classes are automatically injected into your controllers. Your controllers tell the service classes to fetch data. When the data returns, you set the variable in your controller, and the UI automatically updates to show the data:

image

Nice clean separation of concerns. Makes building dynamic apps – apps where the data changes at runtime and the UI automatically shows the new data – a breeze.

TypeScript

JavaScript is a powerful but messy language. Conceived in a weekend of wild coding in the late ‘90s, it was built for a different era when web apps didn’t exist.

TypeScript fixes this. TypeScript is just JavaScript + optional types + features from the future. Where features from the future = ES6, ES7, and future proposals – things that will eventually be added to JavaScript, but you won’t be able to use for 5, 10, 15 years. You can use them right now in TypeScript.

TypeScript compiles it all down to normal JavaScript that runs in everybody’s browser. But it lets you code using modern coding practices like classes, lambdas, properties, async/await, and more. Thanks to types, it enables great tooling experiences like refactoring, auto-completion, error detection.

So instead of writing ugly JavaScript like this:
image

We can instead write concise and clean, intellisense-enabled TypeScript like this:

image

Ahhh…lambdas, classes, properties. Beautiful. All with intellisense, refactoring, error detection. I love TypeScript.

There are few reasons to write plain JavaScript today. It’s beginning to feel a lot like writing assembly by hand; ain’t nobody got time for that. Use modern language features, use powerful tooling to help you write correct code and find errors before your users do.

Bootstrap

You don’t need to drink $15 Frappamochachino Grandes to design elegant UIs.

We’ve got code at our disposal that gives us a nice set of defaults, using well-known UI concepts and components to build great interfaces on the web.

Bootstrap, with almost no effort, makes plain old HTML into something more beautiful.

A plain old HTML table:
image

Add a Bootstrap class to the <table> and it’s suddenly looking respectable:

image

A plain HTML button:

image

Add one of a few button classes and things start looking quite good:

image

Bootstrap gives you a default theme, but you can tweak the existing theme or use some pre-built themes, like those at Bootswatch. For TwinCitiesCodeCamp.com, I used the free Superhero theme and then tweaked it to my liking.

Bootstrap also gives you components, pieces of combined UI to build common UI patterns. For example, here is a Bootstrap split-button with drop-down, a common UI pattern:
image

Bootstrap enables these components using plain HTML with some additional CSS classes. Super easy to use.

Bootstrap also makes it easy to build responsive websites: sites that look good on small phones, medium tablets, and large desktops.

Add a few classes to your HTML, and now your web app looks great on any device. For TwinCitiesCodeCamp, we wanted to make sure the app looks great on phones and tablets, as many of our attendees will be using their mobile devices at the event.

Here’s TwinCitiesCodeCamp.com on multiple devices:

Large desktop:
image

iPad and medium tablets:
image

And on iPhone 6 and small phones:
image

This is all accomplished by adding a few extra CSS classes to my HTML. The classes are Bootstrap responsive classes that adjust the layout of your elements based on available screen real-estate.

Summary

RavenDB, AngularJS, TypeScript, Bootstrap,. It’s a beautiful stack for building modern web apps.

TwinCitiesCodeCamp code is on GitHub.

My startup’s dead! 5 things I learned

Last week I sent the dreaded, “I’m going out of business” email to clients of my BitShuva Radio startup:

shuttingDown

An unintentional startup

A few years ago, I wrote a piece of software to solve a niche problem: the Messianic Jewish religious community had a lot of great music, but no online services were playing that music. I wrote a Pandora-like music service that played Messianic Jewish music, Chavah Messianic Radio was born, and it’s been great. (Chavah is still doing very well to this day; Google Analytics tells me it’s had 5,874 unique listeners this month – not bad at all!)

After creating Chavah, I wrote a programming article about the software: How to Build a Pandora Clone in Silverlight 4. (At the time, Silverlight was the hotness! I’ve since ported Chavah to HTML5, but I digress.)

Once that article was published, several people emailed me asking if I’d build a radio station for them. One after another. Turns out there many underserved music niches. Nigerian music. West African soul. Egyptian Coptic chants. Indie artists. Instrumentals. Ethiopian pop. A marketplace for beats. Local bands from central Illinois. All these clients came out of the woodwork, asking me to build clones of my radio station for their communities.

After these clients approached me – with no marketing or sales pitches on my part – it looked like a good business opportunity. I founded BitShuva Radio and got to work for these clients. I had founded a startup.

But after almost 2 years, making less than $100/month on recurring monthly fees, and spending hours every week working for peanuts, I’ve decided to fold the startup. It wasn’t worth my time, it was eating into my family life, preventing me from working on things I really wanted to work on. So this week, I cut the cord.

Along the way, I learned so much! Maybe this will help the next person who does their first startup.

Here’s what I learned:

1. Don’t be afraid to ask for a *lot* of money.

When I acquired my first client, I had no idea how much to charge. For me, the work involved forking an existing codebase, swapping out some logos and colors, and deploying to a web server. A few hours of work.

I dared to ask for the hefty sum of $75.

Yes, I asked for SEVENTY-FIVE WHOLE DOLLARS! I remember saying that figure to the man on the other end of the phone – what a thrill! – $75 for forking a codebase, ha! To my surprise, he agreed to this exorbitant charge.

In my startup newbie mind, $75 seemed totally reasonable for forking a codebase and tweaking some CSS. After all, it’s not that much work.

What I didn’t understand was, you charge not for how much work it is for you. You charge how much the service is worth. A custom Pandora-like radio station, with thumb-up and –down functionality, song requests, user registration, playing native web audio with fallbacks to Flash for old browsers – creating a community around a niche genre of music – that’s what you charge for. That’s the value being created here. The client doesn’t care if it’s just forking a codebase and tweaking CSS – to him, it’s a brand new piece of software with his branding and content. He doesn’t know what code, repo forking, or CSS is. All he knows is he’s getting a custom piece of software doing exactly what he wants. And that’s worth a lot more than $75.

It took me several clients to figure this out. My next client, I tried charging $100. He went for it. The next client $250. The next client $500. Then $1000.

I kept charging more and more until finally 3 clients all turned down my $2000 fee. So I lowered the price back to $1000.

Money is just business. It’s not insulting to ask for a lot of money. Change as much as you can. Had I knew this when I started, I’d have several thousand dollars more in my pocket right now.

2. Keep your head above the ever-changing technology waters

Don’t drown!

When I built my first the first radio software, Silverlight seemed like a reasonable choice. HTML5 audio was nascent, Firefox didn’t support native MP3 audio, IE9 was still a thing. So I turned to plugins.

Over time, plugins like Silverlight fell out of favor, particularly due to the mobile explosion. Suddenly, everyone’s trying to run my radio software on phones and tablets, and plugins don’t work there, so I had to act.

I ported my radio software code to HTML5, with Flash fallbacks for old browsers. KnockoutJS was the the new hotness, so I moved all our Silverlight code to HTML5+CSS3+KnockoutJS.

As the software grew in complexity, it became apparent you really need something more than data-binding, but Knockout was just data-binding. Single Page Application (SPA) frameworks became the new hotness, and I ported our code over to DurandalJS.

Soon, Durandal was abandoned by its creator, and said creator joined the AngularJS team at Google. Not wanting to be left on a dying technology, I ported the code to Angular 1.x.

Shortly after, the Durandal author left the Angular team over issues with the Angular vNext design, and founded Aurelia.

If I was continuing my startup today, I’d be looking at riding that wave and moving to Aurelia or Angular 2.

What am I saying? Staying on top of the technology wave is a balancing act: stand still and you’ll be dead within a year, but move to every new hotness, and you’ll be forever porting your code and never adding new features. My advice is to be fiscally pragmatic about it: if your paying clients have a need for new technology, migrate. Otherwise, use caution and a wait-and-see approach.

Applying this wisdom in hindsight to my startup, it was wise to move from Silverlight to HTML5 (my paying clients needed that to run on mobile). However, jumping around from Knockout to Durandal to Angular did little for my clients. I should have used more caution and used a wait-and-see approach.

3. Custom software is a fool’s errand. Build customizable software, not custom software builds

My startup grew out of clients asking for custom versions of my radio software. “Oh, you have a Pandora clone? Can you make one for my music niche?”

Naturally, I spent most of my time building custom software. They pay me a nice up-front sum ($1000 in the latter days), and we go our merry way, right?

Turns out, it’s a terrible business model. Here’s why:

Clients continually want more features, bug fixes, more customization. I charged that $1000 up-front fee to build a custom station, but then would spend many hours every week responding to customer complaints, customer requests, bug fixes, performance fixes, new features. And I didn’t charge a dime for that. (After all, the client’s perspective was, “I already paid you!”)

In hindsight, I should have built a customizable platform, ala WordPress, in which potential radio clients could go to my website, bitshuva.com, spin up a new radio station (mystation.bitshuva.com), customize it in-browser, let them use the whole damn thing for free, and when they reach a limit of songs or bandwidth, bring up a PayPal prompt. All of that is automated, it doesn’t require my intervention, and it’s not “custom” software, it’s software that the client themselves can customize to their OCDified heart’s content.

Had I done that, my startup probably would be making more money today, maybe even sustainably so.

Bottom line: Unless a client is paying for 25% of your annual salary, don’t go follow the “I’ll build a custom version just for you, dear client” business model. It’s a fool’s errand.

4. On Saying “No”

I’m a people-pleaser. So, when a person pays me money, I amplify that people pleasing by 10.

“Hey, Judah, can you add XYZ to my radio station this week?”

“Judah! Buddy! Did you fix that one thing with the logins?”

“How’s it going, Judah! Where is that new feature we talked about?”

“Hey Judah, hope it’s going well. When are you going to finish my radio station features? I thought we were on for last week.”

I wanted to please my precious clients. So of course I said “yes”. I said yes, yes, yes, until I had no time left for myself, my sanity, my family.

A turning point for me for over late December, at my in-laws. I was upstairs working, rather than spending the holidays with my kids, my wife. “What the hell am I doing?” The amount of money I was making was small beans, why am I blowing my very limited time on this earth doing *this*?

You see why folks in the YCombinator / Silicon Valley startup clique put so much emphasis on, “You should be young, single, work exclusively on your startup, all-in committal.” I can totally see why, but I also completely don’t want that lifestyle.

Maybe if I had followed YCombinator-level devotion to my startup, it would have grown. But the reality is, I value things outside of software, too. 🙂 I like to chill and watch shows and eat ice cream. I like to relax on the couch with my wife. I like to teach my son how to drive. I like to play My Little Ponies with my daughter. I like to play music on the guitar. I like to work on tech pet projects (like Chavah, MessianicChords, EtzMitzvot).

The startup chipped away at all that, leaving me with precious little time outside of the startup.

5. Startups force you to learn outside your technological niche

On a more positive note, running a startup taught me all kinds of things I would have never learned as a plain old programmer.

When I launched my startup, I was mostly a Windows desktop app developer (i.e. dead in the water). I didn’t know how to run websites in IIS, how to work with DNS, how to scale things, didn’t understand web development. I didn’t know how to market software, how to talk to clients, what prices to charge, didn’t have an eye for “ooh, I could monetize that…”

Building a useful piece of software — a radio station used by a few thousand people — forces you to learn all this crap and become proficient and building useful things.

In the end, getting all retrospective and and hindsight-y here, I’m glad I took the plunge and did a startup, even if it didn’t work out financially, because I learned so much and am a more rounded technological individual for it. Armed with all this knowledge, I believe I will try my hand at a startup again in the future. For now, I’m going to enjoy my temporary freedom. 🙂

Thanks for reading.

“Yet Another JS Framework” isn’t so bad after all

Summary: The web is evolving. Fast. The default reaction is framework fatigue: “Another JS framework to learn? Ugh!” But there’s a hidden upside to the rapid evolution in front-end web dev, and it benefits your happiness, your mind, and your wallet.

We’re seeing rapid evolution in the web development space. Not only is JavaScript and HTML evolving for today’s needs, but numerous frameworks are popping up to leverage the new bits and help you build better web apps.

This fast-paced evolution can be difficult for web developers. There’s a never-ending stream of new things to learn, and that gets overwhelming.

image

My experience In the last 2 years testifies to this: I’ve built web apps with plain old jQuery, then Knockout, then Durandal, then Angular. And with today’s announcement, I’ll probably write my next one with Aurelia. That’s 5 different libraries/frameworks in just 2 years! Web devleopment is a beautiful soup.

But this does lead to developer framework fatigue. On Hacker News today, for instance,

image

Having to learn new stuff all the time and trying to keep up is fatiguing and can be overwhelming.

But there’s another side of this to consider. Rapid evolution leads to better ideas, keeps your mind fresh and your job more interesting. It gives developers an opportunity to stand out with new skills and make more money.

Better ideas rise to the top

Make no mistake: web development has improved by leaps and bounds in the last decade. Rapid library and framework evolution are to thank.

(If a crotchety developer pining away about “the good ol’ days” tells you otherwise, he’s not telling the truth. Those old days were not good. They were littered with browser incompatibilities, unmaintainable apps, no concern for architecture or testing. It was truly the wild west of coding, and it sucked.)

When I started web development, front-end JavaScript meant adding a single giant JavaScript file where you’d do your DOM manipulation and event handlers. The DOM manipulation was usually using browser-specific APIs, and the handlers were hooked up in the global namespace.

This was like the Visual Basic era: put everything into a single file, maybe even a single giant function! We didn’t care about maintainability, testing, separation of concerns.

It was a simpler time, yes, but it was also the time when web development and JavaScript received ugly reputations that persisted until only recently.

Back in the bad ol’ days, web apps themselves generally sucked, too, as a reflection of the code underneath. Great web apps like Gmail, DropBox, Evernote, Pandora…these didn’t exist because building web apps was hard. (Pandora existed…but as a plugin-based Flash app back then!)

Since then, things have vastly improved. How?

Instead of a spaghetti mess of $(“#someVal”).parent().adjacent(“<div>”+ someVar + “</div>”), we’re doing nice and clean logic on variables. Through the magic of data-binding, those variables will be displayed in the UI automatically. The code is modular: simple HTML pages data-bind to controllers containing logic. Controllers wire up to services to load data from the server. The code is arranged in modules with single responsibility, well-defined application architecture, easy to test, easy to modify and extend. This results in better web apps with fewer bugs.

Intellectually stimulating

I actually like the fact that I must keep learning.

It’s true; web development can be overwhelming with all the new technologies and JavaScript frameworks released almost monthly. (If you’re overwhelmed, remember: you don’t have to learn them all.)

But at the same time, I like learning. It keeps my mind fresh. It keeps my job interesting. I like building things, and learning new and better ways to build things is intellectually stimulating.

Consider the alternative. In a space with slow evolution, such as mainframes, there’d be practically no moving technology forward; we’d be doing the same thing for decades, over and over again. See something that sucks about your dev stack? Too bad, you can’t change it.

As a developer and entrepreneur, I want to be doing new things, interesting things, moving the software craft forward. It is a little ego-stoking to think that moving software forward actually moves technology and civilization forward, even if in a small way.

I don’t believe there’s a world in which my skills go stagnant and I still enjoy building technology. It would get old, I would grow tired of my work, become demotivated. My job would become only a means to pay the bills.

I don’t want that career. I like a career that remains fresh, regularly infused with new knowledge. That new knowledge can then be used outside of the professional realm to build your own projects of value to you, personally. (I’ve done this with Chavah, MessianicChords, and EtzMitzvot.)

“Look ma, I’m worth a lot of money”

Knowledge is power. The more I consume, the more powerful (and valuable) I become. Therefore, I am actually grateful to be in a career that requires knowledge growth.

My last 2 gigs as an independent consultant were aided by my knowledge of a single JavaScript framework. And my hack-at-night job gives me money because I know another JavaScript framework. And my startup exists because of the new knowledge I’ve learned in the last two years.

Rapid evolution demands me to acquire more knowledge to remain relevant. I’d be hard-pressed to find work if I was still building web apps like it was 1998, or worse, building desktop apps like it was 1995. (Goodbye, CListViewCtrl. No tears.)

Summary

By continually learning, not only am I intellectually stimulated, I grow in professional value. Corporations pay a lot of money to the niche few who keep their skills relevant. They stand in awe and fork over the dollars by the truckload for the magical wizards who pound on the keyboard and produce revenue-generating, business-critical apps.

So yeah, Yet Another JS Framework was announced today. And I’m OK with that.

Show and tell: Beds and services for homeless youth

Check it out! Awesome success story here: my last project, ysnmn.org, is live! It’s a place for homeless youth to find beds and services in Minnesota. Also, we made it to the front page of Hacker News!

image

I’m a big fan of helping people, and I think this project really does that.

Imagine you’re a homeless 16 year old. You need food, you need a bed to sleep in. Somewhere to shower. Cheap clothing. Maybe you need medical care or counseling. Or help with parenting your child. Maybe you’re looking for a way to finish your education and get back on your feet and off the streets.

Enter ysnmn.org. You pull out your phone, go to the website. Instantly you can see shelters nearest you:

image

What buses to take from your current location to get to a drop-in center:

image

What food shelves are opened and what they serve:image

What phone numbers to call, where to get medical help:image

Where to find a warm bed for the night:
image

Or maybe you just need to be notified when a bed for a female 16 year old becomes available:
image

image

It was such a pleasure building this thing! Working with people who are actually doing good in the world through helping homeless youth is a big win. It’s so satisfying building something for goodness, rather than just for business. The most memorable moment on the project was testing the app with youth ambassadors; I remember one of the kids turning to me and saying, “I wish I had this when I was homeless.” Big smile right there on yours truly.

Technologically, I had a blast building the app as the lone developer! Smile This was a nice change from my previous gig. I had quit my consulting day job to take this project – working as independent consultant in the process – and being the lone developer, I was able to architect the app as I saw fit. We were required to use the Microsoft stack, seeing as how they funded development. But I was cool with that – C# is a great language, Azure is increasingly a solid platform, and TypeScript on the client was a pure joy. I would have preferred to use RavenDB over MS SQL, but aside from that, the freedom of architecture was quite liberating from previous projects working as a cog in a bigger corporate team.

It was my first stab at using AngularJS. I had built Knockout, Durandal, and classic MVC apps in the past, but this was my first stab with Angular. I liked it enough that I’ve adopted it for most of my side projects.

Many shout outs to Microsoft and their Azure group, and particularly Adam Grocholski who not only facilitated this, but also helped us secure some additional credits and put out a few fires during development, and his wife Ann Marie Grocholski who heads up one of the local youth shelters. You guys rock!

A big thank you to the Target Foundation, who helped fund this project with a generous grant.

A big shout out to DevJam in Minneapolis. They approached me to build this project, paid me a nice sum, provide great working environment, and are generally very cool people. I had a great time working with them. Special thanks to Matt Bjornson who worked with me and the shelters the whole time, came up with some great mockups for the UI, and saw this thing to completion. Special thanks to Susan Greve for recruiting me onto this project. So glad I took it!

A shout out to Twilio, particularly Kevin Whinnery, who helped us get started with Twilio and gave us a bunch of free credits. We are putting them to good use, using them to text homeless youth of available beds. Winning!

A shout out to SendGrid and their social network people. I showed them an early prototype of this app, told them I intended to use SendGrid for emailing homeless youth when beds became available. Their reaction? A big account with lots of free credits, plus a t-shirt for yours truly!

This was a great project. I hope to work on more like it.