Working with View Models in RavenDB

Summary: RavenDB opens up some new possibilities for working with view models: objects that contain pieces of data from other objects. With Raven, we can use .Include to simulate relational joins. But we can go beyond that for superior performance by exploiting Raven’s transformers, custom indexes, and even complex object storage.

Modern app developers often work with conglomerations of data cobbled together to display a UI page. For example, you’ve got a web app that displays tasty recipes, and on that page you also want to display the author of the recipe, a list of ingredients and how much of each ingredient. Maybe comments from users on that recipe, and more. We want pieces of data from different objects, and all that on a single UI page.

For tasks like this, we turn to view models. A view model is a single object that contains pieces of data from other objects. It contains just enough information to display a particular UI page.

In relational databases, the common way to create view models is to utilize multiple JOIN statements to piece together disparate data to compose a view model.

But with RavenDB, we’re given new tools which enable us to work with view models more efficiently. For example, since we’re able to store complex objects, even full object graphs, there’s less need to piece together data from different objects. This opens up some options for efficiently creating and even storing view models. Raven also gives us some new tools like Transformers that make working with view models a joy.

In this article, we’ll look at a different ways to work with view models in RavenDB. I’ll also give some practical advice on when to use each approach.

The UI

We’re building a web app that displays tasty recipes to hungry end users. In this article, we’ll be building a view model for a UI page that looks like this:

Figure 1

At first glance, we see several pieces of data from different objects making up this UI.

  • Name and image from a Recipe object
  • List of Ingredient objects
  • Name and email of a Chef object, the author of the recipe
  • List of Comment objects
  • List of categories (plain strings) to help users find the recipe

A naive implementation might query for each piece of data independently: a query for the Recipe object, a query for the Ingredients, and so on.

This has the downside of multiple trips to the database and implies performance overhead. If done from the browser, we’re looking at multiple trips to the web server, and multiple trips from the web server to the database.

A better implementation makes a single call to the database to load all the data needed to display the page. The view model is the container object for all these pieces of data. It might look something like this:

How do we populate such a view model from pieces of data from other objects?

How we’ve done it in the past

In relational databases, we tackle this problem using JOINs to piece together a view model on the fly:

It’s not particularly beautiful, but it works. This pseudo code could run against some object-relational mapper, such as Entity Framework, and gives us our results back.

However, there are some downsides to this approach.

  • Performance: JOINs and subqueries often have a non-trivial impact on query times. While JOIN performance varies per database vendor, per the type of column being joined on, and whether there are indexes on the appropriate columns, there is nonetheless a cost associated with JOINs and subqueries. Queries with multiple JOINs and subqueries only add to the cost. So when your user wants the data, we’re making him wait while we perform the join.
  • DRY modeling: JOINs often require us to violate the DRY (Don’t Repeat Yourself) principle. For example, if we want to display Recipe details in a different context, such as a list of recipe details, we’d likely need to repeat our piece-together-the-view-model JOIN code for each UI page that needs our view model.

Can we do better with RavenDB?

Using .Include

Perhaps the easiest and most familiar way to piece together a view model is to use RavenDB’s .Include.

In the above code, we make a single remote call to the database and load the Recipe and its related objects.

Then, after the Recipe returns, we can call session.Load to fetch the already-loaded related objects from memory.

This is conceptually similar to a JOIN in relational databases. Many devs new to RavenDB default to this pattern out of familiarity.

Better modeling options, fewer things to .Include

One beneficial difference between relational JOINs and Raven’s .Include is that we can reduce the number of .Include calls due to better modeling capabilities. RavenDB stores our objects as JSON, rather than as table rows, and this enables us to store complex objects beyond what is possible in relational table rows. Objects can contain encapsulated objects, lists, and other complex data, eliminating the need to .Include related objects.

For example, logically speaking, .Ingredients should be encapsulated in a Recipe, but relational databases don’t support encapsulation. That is to say, we can’t easily store a list of ingredients per recipe inside a Recipe table. Relational databases would require us to split a Recipe’s .Ingredients into an Ingredient table, with a foreign key back to the Recipe it belongs to. Then, when we query for a recipe details, we need to JOIN them together.

But with Raven, we can skip this step and gain performance. Since .Ingredients should logically be encapsulated inside a Recipe, we can store them as part of the Recipe object itself, and thus we don’t have to .Include them. Raven allows us to store and load Recipe that encapsulate an .Ingredients list. We gain a more logical model, we gain performance since we can skip the .Include (JOIN in the relational world) step, and our app benefits.

Likewise with the Recipe’s .Categories. In our Tasty Recipes app, we want each Recipe to contain a list of categories. A recipe might contain categories like [“italian”, “cheesy”, “pasta”]. Relational databases struggle with such a model: we’d have to store the strings as a single delimited string, or as an XML data type or some other non-ideal solution. Each has their downsides. Or, we might even create a new Categories table to hold the string categories, along with a foreign key back to their recipe. That solution requires an additional JOIN at query time when querying for our RecipeViewModel.

Raven has no such constraints. JSON documents tend to be a better storage format than rows in a relational table, and our .Categories list is an example. In Raven, we can store a list of strings as part of our Recipe object; there’s no need to resort to hacks involving delimited fields, XML, or additional tables.

RavenDB’s .Include is an improvement over relational JOINs. Combined with improved modeling, we’re off to a good start.

So far, we’ve looked at Raven’s .Include pattern, which is conceptually similiar to relational JOINs. But Raven gives us additional tools that go above and beyond JOINs. We discuss these below.

Transformers

RavenDB provides a means to build reusable server-side projections. In RavenDB we call these Transformers. We can think of transformers as a C# function that converts an object into some other object. In our case, we want to take a Recipe and project it into a RecipeViewModel.

Let’s write a transformer that does just that:

In the above code, we’re accepting a Recipe and spitting out a RecipeViewModel. Inside our Transformer code, we can call .LoadDocument to load related objects, like our .Comments and .Chef. And since Transformers are server-side, we’re not making extra trips to the database.

Once we’ve defined our Transformer, we can easily query any Recipe and turn it into a RecipeViewModel.

This code is a bit cleaner than calling .Include as in the previous section; there are no more .Load calls to fetch the related objects.

Additionally, using Transformers enables us to keep DRY. If we need to query a list of RecipeViewModels, there’s no repeated piece-together-the-view-model code:

Storing view models

Developers accustomed to relational databases may be slow to consider this possibility, but with RavenDB we can actually store view models as-is.

It’s certainly a different way of thinking. Rather than storing only our domain roots (Recipes, Comments, Chefs, etc.), we can also store objects that contain pieces of them. Instead of only storing models, we can also store view models.

This technique has benefits, but also trade-offs:

  • Query times are faster. We don’t need to load other documents to display our Recipe details UI page. A single call to the database with zero joins – it’s a beautiful thing!
  • Data duplication. We’re now storing Recipes and RecipeViewModels. If an author changes his recipe, we may need to also update the RecipeViewModel. This shifts the cost from query times to write times, which may be preferrable in a read-heavy system.

The data duplication is the biggest downside. We’ve effectively denormalized our data at the expense of adding redundant data. Can we fix this?

Storing view models + syncing via RavenDB’s Changes API

Having to remember to update RecipeViewModels whenever a Recipe changes is error prone. Responsibility for syncing the data is now in the hands of you and the other developers on your team. Human error is almost certain to creep in — someone will write new code to update Recipes and forget to also update the RecipeViewModels — we’ve created a pit of failure that your team will eventually fall into.

We can improve on this situation by using RavenDB’s Changes API. With Raven’s Changes API, we can subscribe to changes to documents in the database. In our app, we’ll listen for changes to Recipes and update RecipeViewModels accordingly. We write this code once, and future self and other developers won’t need to update the RecipeViewModels; it’s already happening ambiently through the Changes API subscription.

The Changes API utilizes Reactive Extensions for a beautiful, fluent and easy-to-understand way to listen for changes to documents in Raven. Our Changes subscription ends up looking like this:

Easy enough. Now whenever a Recipe is added, updated, or deleted, we’ll get notified and can update the stored view model accordingly.

Indexes for view models: let Raven do the hard work

One final, more advanced technique is to let Raven do the heavy lifting in mapping Recipes to RecipeViewModels.

A quick refresher on RavenDB indexes: in RavenDB, all queries are satisfied by an index. For example, if we query for Recipes by .Name, Raven will automatically cretate an index for Recipes-by-name, so that all future queries will return results near instantly. Raven then intelligently manages the indexes it’s created, throwing server resources behind the most-used indexes. This is one of the secrets to RavenDB’s blazing fast query response times.

RavenDB indexes are powerful and customizable. We can piggy-back on RavenDB’s indexing capabilities to generate RecipeViewModels for us, essentially making Raven do the work for us behind the scenes.

First, let’s create a custom RavenDB index:

In RavenDB, we use LINQ to create indexes. The above index tells RavenDB that for every Recipe, we want to spit out a RecipeViewModel.

This index definition is similiar to our transformer definition. A key difference, however, is that the transformer is applied at query time, whereas the index is applied asynchronously in the background as soon as a change is made to a Recipe. Queries run against the index will be faster than queries run against the transformer: the index is giving us pre-computed RecipeViewModels, whereas our transformer would create the RecipeViewModels on demand.

Once the index is deployed to our Raven server, Raven will store a RecipeViewModel for each Recipe.

Querying for our view models is quite simple and we’ll get results back almost instantaneously, as the heavy lifting of piecing together the view model has already been done.

Now whenever a Recipe is created, Raven will asynchronously and intelligently execute our index and spit out a new RecipeViewModel. Likewise, if a Recipe, Comment, or Chef is changed or deleted, the corresponding RecipeViewModel will automatically be updated. Nifty!

Storing view models is certainly not appropriate for every situation. But some apps, especially read-heavy apps with a priority on speed, might benefit from this option. I like that Raven gives us the freedom to do this when it makes sense for our apps.

Conclusion

In this article, we looked at using view models with RavenDB. Several techniques are at our disposal:

  • .Include: loads multiple related objects in a single query.
  • Transformers: reusable server-side projections which transform Recipes to RecipeViewModels.
  • Storing view models: Essentially denormalization. We store both Recipes and RecipeViewModels. Allows faster read times at the expense of duplicated data.
  • Storing view models + .Changes API: The benefits of denormalization, but with code to automatically sync the duplicated data.
  • Indexes: utilize RavenDB’s powerful indexing to let Raven denormalize data for us automatically, and automatically keeping the duplicated data in sync. The duplicated data is stashed away as fields in an index, rather than as distinct documents.

For quick and dirty scenarios and one-offs, using .Include is fine. It’s the most common way of piecing together view models in my experience, and it’s also familiar to devs with relational database experience. And since Raven allows us to store things like nested objects and lists, there is less need for joining data; we can instead store lists and encapsulated objects right inside our parent objects where it makes sense to do so.

Transformers are the next widely used. If you find yourself converting Recipe to RecipeViewModel multiple times in your code, use a Transformer. They’re easy to write, typically small, and familiar to anyone with LINQ experience. Using them in your queries is a simple one-liner that keeps your query code clean and focused.

Storing view models is rarely used, in my experience, but it can come in handy for read-heavy apps or for UI pages that need to be blazing fast. Pairing this with the .Changes API is an appealing way to automatically keep Recipes and RecipeViewModels in sync.

Finally, we can piggy-back off Raven’s powerful indexing feature to have Raven automatically create, store, and synchronize both RecipeViewModels for us. This has a touch of magical feel to it, and is an attractive way to get great performance without having to worry about keeping denormalized data in sync.

Using these techniques, RavenDB opens up some powerful capabilities for the simple view model. App performance and code clarity benefit as a result.

RavenDB MVC starter kit

Looking to get started building ASP.NET MVC apps with RavenDB? Check out RavenDB.ModernMvcStarterKit. It’s an MVC template that lets users register on your site, verify their registration via email, and allows them to opt-in to two-factor authentication via email or SMS.

Modern websites need a robust identity system, where users can login, confirm their registration via email, and optionally enable two-factor authentication.

Doing this with the .NET + RavenDB stack has had some friction:

  • The default MVC template is wired to work with Entity Framework, not RavenDB.
  • Making the default template work with RavenDB is non-trivial, and requires implementing a RavenDB-specific identity provider.
  • There is no publicly-available Identity provider for RavenDB. Or rather, there are 2 available, but they work against old versions of Identity Framework: tugberkugurlu’s AspNet.Identity.RavenDB and ILM’s RavenDB.AspNet.Identity.
  • The basic MVC template from Visual Studio doesn’t implement email service or SMS service, nor UI for this or Two-Factor Authentication, leaving it up to you to do all the heavy lifting.

To remove this friction, I created RavenDB.ModernMvcStarterKit.

In a nutshell, it’s an MVC sample project that uses RavenDB as the backend, supports user registration and confirmation via email (SendGrid), and optionally supports Two-Factor Authentication via email (SendGrid) or SMS (Twilio).

This project provides:

  • A RavenDB-backed identity provider, updated to work with the latest MS Identity Framework (2.1 at the time of this writing). This is a fork of ILM’s RavenDB.AspNet.Identity provider, updated to work with email and SMS authentication and the latest Identity bits. I’ve submitted a pull request with my changes to merge back into that repo.
     
     
  • A registration confirmation system: when a user registers, he is required to confirm his registration via email. The email is sent via your developer SendGrid account. Upon registering, the user is presented with the following screen:
    The user will receive an email with a confirmation link. Following the link, he’ll be taken to this page:
     
     
  • Optional two-factor authentication: Users can optionally go to their profile page and add a phone number and enable Two-Factor Authentication:
    When the user enters his phone number, we send an SMS verification code via your developer Twilio account:
    With Two-Factor Authentication enabled, when the user goes to sign-in next time, he will first login as usual, and then be prompted to enter the 2nd form of identification, either email or SMS:
    The user will receive a verfication code via SMS or email and be redirected to the verification page:
    Upon entering the verification code, they’ll be able to sign in.

Bottom line: this is a nice way to build a modern MVC app with RavenDB. It’s all about removing friction. Smile Check out the project on GitHub.

Enjoy!

Video: Dev Life Made Better with RavenDB

My talk out at RavenDB Days in Malmo, Sweden. A fun, lighthearted talk on why Raven is an excellent choice for modern apps.

image
(Slides here)

Had a blast giving this talk! Some great interactions with the audience, some great feedback afterwards. I think the audience enjoyed.

p.s. If you’re in Minnesota, Twin Cities Code Camp takes place this weekend, I’ll be giving a talk – also My Little Pony-infused Smile — at Code Camp, so stop by and check it out, I think you’ll be entertained and might learn a few things along the way.

Lap Around HTML5 RAvenDB Studio

Last month, I was honored to give a talk at the very first (!) RavenDB conference, the first of many, as I believe Raven is poised for greatness as it gains more exposure and spreads its wings outside the .NET niche.

My talk was on the new Raven Studio – built from the ground-up using modern web technologies, HTML5, LESS, TypeScript – oh yes!

[youtube https://www.youtube.com/watch?v=wlbAj-qEuV4]

I hope you guys enjoy the new RavenDB Studio! It’s now available as unstable 3.0 build. Browse the code (or heck, send me pull requests) over at the GitHub repo.

RavenDB Studio 3.0, and why we moved from Silverlight to HTML5

Summary: A big step for RavenDB: a new HTML5 Studio. Plus, some thoughts on the move from Silverlight to HTML5 and our experience in the transition.

Yesterday, I pulled the covers off something I’ve been working on for a few months, something I’m very proud of.

RavenDB, the most popular NoSQL database in the .NET world, announced a brand new RavenDB Management Studio, Raven Studio 3.0, built from the ground up using HTML5 and modern web technologies. Yes, we’re moving away from Silverlight and onto HTML5.

Ayende and myself demoed the new Raven Studio just yesterday in a live webinar:

[youtube=http://www.youtube.com/watch?v=FNDKuiftKcc&w=448&h=252&hd=1]

This has been my pet project for the last few months, and it’s something I’m quite proud of! I believe this is a huge step forward for RavenDB (more on that in a minute), and the reception from the Raven community has been awesome, ego-stoking, and totally energizing.

The old Raven Studio was built in Silverlight. Some Silverlight fans have asked, why did we move to HTML5? Are we making a big mistake moving away from Silverlight and to HTML5?

No, on the contrary, we believe HTML5 is a damn good option.

  • The RavenDB community wants an HTML5 Studio. This has probably been the most-requested item from the RavenDB community. It came up multiple times in the Raven 3.0 Wishlist, it’s come up whenever Oren talks about the Studio, it comes up when we speak to the external developer community, heck, when I was in Israel for RavenDB training last year, one of the students brought it up right there in the Hibernating Rhinos office. Silverlight has served us well over the years, but Silverlight is a dying technology that our community doesn’t want to be tied to any longer. 
     
  • Silverlight tooling is a perceived barrier to Raven adoption. I speak at Code Camps and user groups, and when I speak on RavenDB, the love flows and the excitement grows…until I show Silverlight tooling. I get the raised eyebrow. “Silverlight? Oh. I see.” Others in the Raven community have reported this as well. For some, Silverlight is a stumbling stone. 
     
  • HTML5 is a step towards cross-platform Raven. RavenDB is the best NoSQL database for .NET. But, in time, we want Raven to spread her wings and be not just the best NoSQL solution for .NET, but the best NoSQL database, period. Moving to an HTML5 toolset is a step towards this goal. 
     
  • The software industry is moving away from plugins. Plugins like Silverlight added abilities you couldn’t do on the native web, such as audio, video, gaming, 2d drawing, documents, voice, and more. Plugins filled these gaps, but with HTML5, these gaps are disappearing. We don’t need Adobe Acrobat plugins anymore to view that high fidelity document. We don’t need Java applets anymore to run that simulation. We don’t need websites built entirely with Flash. And we don’t need Silverlight for Raven Studio. There is little reason today to build something in JavaFX, Flash, or Silverlight: the native web has supplanted them. Just as it’s rare – and often undesired – to see a Java applet out in the wild, so too it will be with Silverlight in the coming years.
     
  • The native web platform is a solid foundation for the future. Microsoft products come and go. 3 years ago, Microsoft was pushing Silverlight as the platform for line of business apps and islands of richness on the web. Today, Silverlight is prevented from running in the default browser of their newest operating systems.

    HTML, on the other hand, has been a stable, ever-evolving technology for decades, and because it is the very fabric of the web, things built in HTML live indefinitely. There’s a reason you can still visit and use the 17-year old Space Jam Website. Smile But your MS Silverlight app from last year? It won’t run even on the latest MS operating system’s default browser.

As a Silverlight developer who has built professional apps (e.g. 3M Visual Attention Service) and spoken at Code Camps and user groups on Silverlight, truth be told, Silverlight is a great developer platform. C# is an excellent language, Visual Studio probably the best development environment.

But, in the words of Miguel de Icaza, creator of Moonlight (open source, cross-platform Silverlight),

“I felt that Silverlight had a bright future, and that it could turn to fill an important void, not only for web development, but for desktop development in general.  And this was largely one of my motivators. I am very sad that Microsoft strategy cut the air supply to Silverlight.”

This, coupled with the mobile computing explosion and the software industry’s shift away from plugins, results in a sickly future for Silverlight and Silverlight apps.

RavenDB rocks, and we want the tooling to rock as well. Having our tooling tied to this technology was not an attractive proposition, and it was time for us to move on.

A new technology stack for Raven Studio 3.0

After much deliberation and considering all the options available to us, we moved off of Silverlight.

Instead of Silverlight, HTML5.

Instead of C#, TypeScript.

TypeScript is awesome. TypeScript is new language, a superset of JavaScript designed for building apps on the web. Silverlight fans will be happy to know it’s built by none other than Anders Heijlsberg, the much-respected language designer and author of C#.

In TypeScript, all JavaScript is valid TypeScript code, so it’s familiar to any web developer, but it gives us nice things like an optional, flexible type system, classes, modules, and enums, and features proposed for future versions of JavaScript, but compiles to plain old JavaScript that runs in every browser.

TypeScript tooling is Visual Studio, with all the nice debugging and refactoring that brings, but it can also be written in any text editor and debugged in any browser.

For infrastructure, because we wanted the look & feel of a web application, rather than a set of web pages, we opted to build a single page application (SPA). Durandal.js gives us exactly that: a nice means to load pages on demand and compose them into a cohesive web application.

For UI, Durandal uses Bootstrap for a consistent, pleasing aesthetic, and KnockoutJS for data binding and MVVM.

Using data binding, MVVM, and Durandal makes a great developer experience, one not too foreign to the MVVM stuff in Silverlight. (Indeed, the author of Durandal.js is the same author of the popular Silverlight MVVM framework Caliburn Micro.) Look at the code and judge for yourself; you’ll see classes separated out into small, logical view models, and a clean separation between view and logic.

What has been our experience moving to HTML5?

One immediate, measurable gain was performance:

  • Memory usage dropped from 140MB to 20MB.
  • Cold starts dropped from ~7s to ~2s.
  • Warm starts dropped from ~3s to ~1s.
  • General snappiness: XAML is rather heavyweight, and you’ll notice just moving around the application, loading your documents, collections, or editing – it’s all faster in HTML5. Snappy and responsive.
  • This doesn’t happen:
    image
  • This doesn’t happen either:
    image

A lot of the above we get for free simply by Doing Less Stuff™. No .xap files to download, no dlls to load, no CLR runtime to start, no plugin host process for the browser, no browser-to-plugin communication, no managed code to start executing.

This translates into faster start times and less memory usage.

Another free item we get is JavaScript and the blazing-hot modern JS browser runtimes. The major browsers – IE, Firefox, Chrome, Safari – are in a cut-throat competition to get the fastest JavaScript runtime, to squeeze every possible ounce of performance out of JavaScript. You’ll regularly see these browser vendors advertising their JS benchmarks as proof of performance improvement. This is a free win for the new HTML5 Raven Studio: as browser vendors continue to improve their engines in this cut-throat, cross-company competition of speed, Raven Studio will reap the performance improvements.

Moving to the native web platform fixes some plugin-induced workflow hiccups. For example, keyboard shortcuts: Silverlight and other plugins eat the keyboard. So say you’re got Raven Studio opened, and you want to open a new browser tab, so you hit CTRL+T. Surprise, nothing happens. Why? Silverlight ate your keyboard shortcuts, your browser never received them, and your workflow was just interrupted.

If you’ve ever used one of those old all-Flash websites, or full-page Java applets, you’ve probably noticed some things just don’t feel right. So it was with the old Silverlight Studio. Moving to HTML5 fixes these issues.

Conclusion

Transitioning out of the plugin ghetto and moving to HTML5 has been a delight, but more importantly, it’s good for RavenDB users as we move to a faster, more lightweight tool. It’s good for the future of RavenDB to have our tooling built on the solid rock of the native web.

I understand the Silverlight fans who are sad to see the old Silverlight Studio go. I’m a Silverlight fan myself, I understand their concerns. The most I can ask of you guys is to give us the opportunity to earn your trust. It will take time, but with a faster, more lightweight, stable tool that does what you need and gets out of your way, I believe that trust will be earned.

The new HTML5 Raven Studio is on GitHub and we’d love for you to give it a spin or even contribute to the code. I’m pleased to say we already have had a few contributions since it was released just yesterday. I’m proud of this work, and I really hope you guys enjoy it!