Streaming XML in Rails 3.2 using to_xml

Recent versions of Rails (or more honestly, Rack) introduced the ability to stream your responses to HTTP requests. This has a couple of great benefits, it starts the page rendering faster for the end user, and it prevents the creation of arbitrarily large buffers on your server (which can be a performance killer).  This is provided “for free” for a regular, templated HTML page.  If you’re generating very large XML or JSON files where you would like to do something similar, you’ve got to do it manually.

In order to stream data out of your app, you simply need to pass an object that responds to “each” (and doesn’t respond to “to_s”) to the render call.  Every time the callback passed in is invoked, Rack writes the string passed in to the outgoing stream.  In older versions of Rails and Rack we had an object that we could directly write to, but that’s no longer the case.

Under the hood, the ActiveSupport #to_xml commands use a Builder::XmlMarkup object in order to do their magic, and you can pass your own instance in using the optional parameter :builder.  Builder::XmlMarkup#new takes an optional parameter :target that is an object that needs to respond to :<<.  It will then call :<< on the target instead of just writing to an internal string.

So that leaves us with a way to stream that requires us to respond to a callback, and a way to write a stream that requires us to respond to a callback.  It’d be nice if one of them would give us an object we could pass to the other, but no such luck.  We’ll have to proxy the request across from the Builder to Rack.  To do this, I made a class that takes the object to be serialized and responds to each. When each is invoked, the object is serialized using a builder with a custom target. That custom target responds to << by invoking the callback passed in to it on initialization.  The code is easier to follow than words, so here’s an example:

class XmlStreamer
  def initialize(obj)
    @obj=obj
  end

  def each &callback
    @obj.to_xml(builder: Builder::XmlMarkup.new(target: XmlStreamerCallback.new(callback)))
  end
end

class XmlStreamerCallback
  def initialize(callback)
    @callback = callback
  end

  def << xml
    @callback.call(xml)
  end
end

# In our controller...
respond_with data do |f|
  f.xml  { render xml: XmlStreamer.new(data) }
end

This example is NOT a generic implementation of this pattern.  A full implementation should also manage the parameters that can be passed to to_xml, and may even be implemented as a method on the original object.  However, this should give you a starting point for streaming your own XML building.

I plan have a JSON version of this up soon.

Fair Dice in Google+ Hangouts

I’ve been using Google+ Hangouts a lot, to keep up with friends and do business, so I decided to try my hand at creating a game for their API.  Because the act of rolling dice seems like a common thing to want to do, I thought I would share my approach and a library that implements it.

In the game I’m working on, I require people to be able to make dice rolls.  A naive implementation would have the users’ clients perform the random number generation, and then update the shared state object with the result.  However, since the game runs fully in the browser and is Javascript, a simple random number generator for each player would be very easy to “cheat” by simply using the browser’s developer tools to override the function to always return, say, a 20.

My implementation works a differently in order to prevent tampering. Every time an update is made to the shared state, the key that was changed is timestamped with a millisecond resolution by the server. This is recorded in the metadata for the update.  Because this has millisecond resolution, it is sufficiently indeterminate enough to act as a random seed.  To actually roll the dice, I send a submitDelta request to the hangouts API with a common key (in this case “HangoutDice.roll”) and a value containing the information on the number and type of dice requested. This will then call the onStateChanged handler with notification of the key’s change. All of the clients can then take the timestamp from the metadata of the key, use it to seed a pure JS RNG, and determine what the roll was.  The seeded RNG allows all clients to independently determine the roll with the same results.  This results in a very cheat-resistant, distributed dice roller.

The javascript library can be found here: https://github.com/jaswope/hangout-dice

You can start a hangout using the library here: http://dice.finalmeasure.com/dice-game/

Please note that at this time there is no good way to share a Hangouts app with other people. The only way for anyone to install it is to create a hangout with the link above, and then click on the app’s button in the hangout. There is no way to share it with other people in a hangout, even if you start it with the above link and then invite others to it.  The other participants will have to use the link at least once before it is available.

Chrome Campfire Notifications

I ended up getting one of those nifty Cr-48 laptops that Google has been giving out. I haven’t written anything about it because, well, it’s like using Chrome. If you’ve used the browser (and especially if you’re a power user) you won’t find much new. There’s very little that I do outside of the context of a browser, and if I consider the Chrome OS laptop my portable machine (to be used in conjunction with a desktop), there’s really nothing that can’t be done online.

The one thing I do miss from the desktop world is desktop notification systems (like Growl, or the native Windows/Linux ones) that I could use with my native Campfire clients. Luckily Chrome (and by extension Chrome OS) supports desktop notifications natively as well, so I created an extension that will take care of this in the browser-only world of Chrome.

Chrome Campfire Notifications is very bare bones at the moment, but I plan to release updates to allow for some amount of configuration. It does work as advertised, though, giving you notifications when Campfire messages come through.

Rails 3 Engines/Plugins and Static Assets

THIS INFORMATION IS OUT OF DATE!  Since Rails 3.2, you can easily include static assets in your engine using the “app/assets” folder.  This post is only applicable for older versions of Rails, and still not suggested for production systems.

Engine-ish

One of the big changes in Rails 3 is the move to “everything is an engine”.  You end up reading a lot about this in relation to plugins, mentioning how this architecture makes it so you can easily embed entire rails applications into others since they’re all just derivatives of Railties.  While this holds true for very simple cases, you’ll quickly find simply creating a rails app, and sticking it in vendor/plugins doesn’t work the way you’d expect from reading about it.  You really do need to create your plugin in a certain way to actually make it work.  Over here is a good writeup on creating a Rails Engine based plugin installed as a gem.

Serving Static Assets

Now, serving static assets from your plugin is a bit on the unintuitive side, and likely for a good reason.  You generally do not want to do this for any kind of production system. However, like most rules, there are times when breaking them makes pragmatic sense.  In this use case we have an internal tool that we want to be able to easily extend.  The maximum number of concurrent users will likely be in low teens, and serving static assets through the rails app is a non-issue.  Extending the app includes providing views, controllers, models, routes, and images.  The first three are extremely straightforward and intuitive, routes use a slightly different enclosing syntax in an Engine, but are otherwise identical.  Images, however are a bit tricky.

There are two ways to create an Engine based plugin, and they are a bit incompatible with each other.  You can either install your plugin as a gem, or as a plugin in the app.  In both cases, you are going to use the ActionDispatch::Static middleware to serve your content. I am also assuming that you are placing your content in the “public” directory in the root of your plugin’s file structure.

Gem Based Plugin

A gem based plugin is initialized via the definition of itself in your plugin’s lib directory.  You can see an example of this below.  In order to set it up correctly you should put your middleware line in an initializer in your declaration:

module MyEngine
  class Engine &lt; Rails::Engine
    initializer "static assets" do |app|
      app.middleware.use ::ActionDispatch::Static, "#{root}/public"
    end
  end
end

In App Plugin

Unfortunately, if the above file exists in a plugin installed to vendor/plugins your app will fail to load.  This is because rails automatically assumes all plugins are Engines, and initializes them as instances of the Plugin class which inherits from Engine.  It will autoload everything in lib, see the declaration, and then fail out with the incredible error message “[Your Plugin] is a Railtie/Engine and cannot be installed as plugin”. Which is somewhat misleading since a plugin is an Engine, just not explicitly.

To get around this issue, you just have to move where you do your initialization and setup. Inside your plugin’s init.rb (which should be in the root of your plugin’s file structure) is where you can do this kind of stuff.  Rails boots your Plugin, and then loads this file with the variable “config” set for you to do your work.  So it’s simply:

config.middleware.use ::ActionDispatch::Static, "#{root}/public"

Load Order

I strongly recommend using middleware.use as it will place your middleware after the parent app, and therefore will cause your plugins assets to be of lower priority than everything else in the stack. This means that files in the parent app’s public directory will load instead of the plugin’s in the case of a conflicting name.  You still have options though.  To give your plugin’s static files priority over everything except the parent app’s static files, use:

middleware.insert_after ::ActionDispatch::Static, ::ActionDispatch::Static, "#{root}/public"

To give your plugin top priority in static asset serving use:

middleware.insert_before ::ActionDispatch::Static, ::ActionDispatch::Static, "#{root}/public"

Cucumber, Service Based Applications, Middleware, and You

Over at Primedia, we’ve been trying to get a little bit better at our BDD, and have resolved to really use Cucumber wherever possible. This is an easy thing to say for most web apps, however ours have a number of critical parts that are external to the application itself. This means that full integration testing of the applications that use these services as well as testing the services themselves can be rather tricky. In order for us to get started there were two major problems that we had to tackle.

The first was how to test the service based apps outside of any consuming application. The specific one in question did not present any HTML pages at all, and only returned Javascript that was executed. A few ideas were floated around like dev/test only controllers or basic one-off apps that were specifically for testing the service. Eventually we cam across a solution that worked and seemed much cleaner. We combined the one-off app idea with rack middleware to create an application within our app that was only loaded in certain environments. We used Sinatra to make a simple application that would respond with test pages on routes prefixed with ‘/test’, and then added it with config.middleware.use. Combining this with Celerity and Capybara gave us a very clean testing solution for an application that only served Javascript.

There is one gotcha to this, and it only involves using Sinatra as middleware in Rails. Rails attempts to determine if the middleware you gave it is a proc, and in doing so prevents you from being able to use any middleware that can respond to Rack requests as a singleton (like Sinatra). Our quick fix was to monkey patch Sinatra to remove self.call, but a more permanent solution is fixing the way Rails 2.3 does its middleware. I hope to submit some patches for that soon.

module Sinatra
  class Base
    class << self
      remove_method :call
    end
  end
end

Our second hurdle was determining how to test our consuming apps using Cucumber. The main problem we had is that we use reverse proxying to make the services appear to be coming from the same domains as the rest of the apps. This was easily taken care of in manual tests, where we could simulate our load balancers using Apache or nginx, however with automated tests, this becomes a problem. The automated test suite doesn't run behind a proxying layer or a web server, it just hits the app directly. So, tackling that problem, I am releasing a gem I developed called rack-reverse-proxy. What this gem does is allow you to put reverse proxy rules directly into your middleware stack so they are independent of your web server. This is probably not a good idea for production systems, load balancers and web servers are much better at this sort of thing, and due to the way rack works I have to wait until the entire request is received before forwarding it. However it's great for development and testing, allowing you to test your app directly without the need of an intermediate web server layer.

  require 'rack/reverse_proxy'

  use Rack::ReverseProxy do
    # Forward the path /test* to http://example.com/test*
    reverse_proxy '/test', 'http://example.com/'

    # Forward the path /foo/* to http://example.com/bar/*
    reverse_proxy /^\/foo(\/.*)$/, 'http://example.com/bar$1'
  end

Updating Multiple Git Repos

Adam Lowe has a great writeup on setting up Vim for Ruby/Rails work.  It’s a pretty great solution, using pathogen.vim and the git repos for the bundles, but he didn’t mention how to update your bundles in an easy manner…

$ find . -name ".git"  -type d -prune -execdir git pull \;

This is a little command line snippet that will search for git repositories in all of the subdirectories of wherever you run it and update them.  Really handy in this case, because you can update all of your bundles at once, but it will work in any case where you have cloned a bunch of repos and want them to all stay up to date.

Passenger can’t find Rails

If you are running in to an issue where you are trying to get a Rails app up and running, but are receiving the error “Missing the Rails 2.x.x gem…” when you attempt to access it despite Rails being installed and available to your system, do not panic.  Rails is just giving you an extremely unhelpful error message.

Rails catches *any* problem loading the Rails gem, and then responds with that error.

 rescue Gem::LoadError => load_error
      $stderr.puts %(Missing the Rails #{version} gem. Please `gem install -v=#{version} rails`, update your RAILS_GEM_VERSION setting in config/environment.rb for the Rails version you do have installed, or comment out RAILS_GEM_VERSION to use the latest version installed.)
      exit 1

If you inspect load_error from the line above, you should see the actual error message. In my case it was that ActiveSupport required Rack v1.0.0, but Passenger had already instantiated Rack v1.1.0. This led to a failure when loading the Rails gem, which was of course incorrectly reported. My quick solution was to uninstall the new version of rack, and force the install of the older version.

As a side note: I have the world’s worst code highlighter.

Getting Ruby Going on App Engine

I gave a talk last night at the Atlanta Ruby Users Group meetup about using JRuby on Google App Engine.  It was more of an overview about what App Engine is, and how to get started with a Sinatra app.  The Github repository for the application is located here, and you can see the application in action here.

The application demonstrates a few app engine concepts.  In models.rb you’ll see a sharded counter implementation using the direct Datastore API, and a traditional model using Datamapper as your ORM (ODM?) to Datastore.  The counter also demonstrates usage of the Memcache API.  Inside app.rb you can see some simple usage of the Users API, including how to check to see if a user is logged in, present a user with a login screen, and interacting with the object.  Lastly, you will find an example of how to determine your app’s name and version in config.ru.

Following the presentation I received a number of requests for my slides, so they are included below.  Slide 13 illustrates pushing your own app to App Engine.  As I mentioned in the talk, I strongly suggest setting aside one of your app ids for testing new ideas, since you only get 10 ids total. Slides removed due to discontinuation of the slide tool.

Arduino In True Color

As part of a larger project, I needed to have a way to allow a computer to “see” the color of an object.  In this case, size and shape of the object were unimportant, however the position of the objects that need to be sorted means that the detector must be able to fit into a very small space.  This ruled out using cameras and the like.

In order to accomplish this, I turned to my Arduino, and started brainstorming.  My initial thought was to use a very bright light and three photoresistors with RGB screens over the top.  This has the benefit of being *very* fast, but it still requires a good amount of space as well as finding the right material to use for gels.  I’ve tried to find good colors like that in hobbyist quantities before, and it was always a pain.

The solution I decided on was to using a single tri-color LED paired with a single phototransistor.  By cycling through the colors and measuring the resistance at certain times, I should be able to retrieve something very close to the correct color.  I placed the phototransistor in one of the analog read pins.  I hooked my common anode LED up by placing the anode to +v, and the cathodes to PWM leads set to “OUTPUT”.  They work a bit opposite of what you may be used to, requiring you to bring the pin low (via an analog write of 0) to turn them on.

It is very easy calibrate by measuring a black object for your minimums, and then a white object for your maximum values.   After that quick setup you have a working color detector that can fit into the area of a single LED and a photoresistor.  You can see it in action below.