Inline media uploading and processing in ruby

Today, we released a new gem to make integrating off-site processing easy. With this gem and utilizing Amazon’s EC2 cloud and S3 storage backend, you can easily process media files in a non-blocking, threaded way and do it all inline with the rest of your application.

First download the gem and all it’s dependencies by typing:

  sudo gem install processor_pool -y

This gem is built on top of Sinatra. So, if you are familiar with the Sinatra syntax, then you’ll be right at home. If not, I highly suggest you check it out.

First,

  # In a file called media_processor.rb
  require "rubygems"
  require "processor_pool"

  # Set the access_key_id and the secret_access_key provided by amazon
  access_key_id, secret_access_key = 'accesskey', 'somethinglongandfuzzyhere'

  # And start the processors!
  ProcessorPool.start(access_key_id, secret_access_key)

Add to the end of the file

  get '/' do
    "Show me the money"
  end

That’s it. Start your server up with

 ruby media_processor.rb

And now you are ready to go to allow a user to hit the EC2 cloud (more on this later) and be responded with the “Show me the money” text.

Replace the above content with the following

  class PFile
    attr_accessor :filename, :temp_file
    @@temp_directory_base = "tmp"
    def initialize(f)
      self.filename = f[:filename]
      self.temp_file = f[:tempfile]
      @media_file = nil
      @temp_dir
    end
    def delete!
      #File.unlink(self.temp_file)
      self.temp_file.delete
    end
    def store_in_temp_directory
      @temp_dir = File.join(@@temp_directory_base, @upload.member_id, @upload.media_file_id)
      FileUtils.mkdir_p(@temp_dir)
      filename = File.basename(self.filename)
      FileUtils.mv(self.temp_file.path, File.join(@temp_dir, filename))
      @media_file = MediaFile.new(File.join(@temp_dir, filename))
    end
    def process!
      ImageConverter.new(@media_file).convert    
      @media_file.delete!
      "Processed images for #{@media_file.original_file_name}"
    end
    def save_to_storage
      # DO THE S3 saving here
      MediaDirectory.new(@temp_dir).copy_to_s3(STORAGE_BUCKET, upload_prefix, true)
    end
    def upload_prefix
      "#{@upload.member_id}/#{@upload.media_file_id}"
    end
  end
  class ImageConverter 
    def resize_to(name, image = nil)
      img = image || Magick::Image.read(self.file.file_name).first
      named_size = self.class.named_sizes[name]
      if named_size =~ /c$/
        dimensions = named_size.gsub(/c$/, '').split('x')
        img.crop_resized!(dimensions[0].to_i, dimensions[1].to_i)
      else
        img.change_geometry!(named_size) do |cols, rows, img|
          img = img.resize(cols, rows)
        end
      end
      img.write(path_to(name))
    end
  end
  post '/new' do
    @file = PFile.new(params[:file])
    @file.store_in_temp_directory
    @file.save_to_storage
    redirect params[:success_url] if params[:success_url]
  end

If you look closely, we are just using some simple classes to handle our conversion. But ignoring that, we are sending a post to the server with file parameters. We can send the process a file and the rest is history. Notice that this all happens on a remote server. None of this is handled locally.

That’s it.

Of course, you can imagine how you could extend this to include movie types and other types of processing, but for the time being, let’s focus the images. (Although I will be showing how to do this in rails, you can use this with any back-end, PHP, Java, etc.)

Let’s add a convenience method in the application.rb 
  def url_for_upload_server
    if RAILS_ENV == 'production'
      # determine the upload server to try to use
      p = Processors.get_random_processor(::SERVER_POOL_BUCKET)
      "http://#{p.hostname}:4567/new"
    else
      p = Processors.get_random_processor(::SERVER_POOL_BUCKET)
      "http://localhost:4567/new"
    end
  end

In your upload controller add

  @upload_url = url_for_upload_server

Now, in your upload view, add

  <% form_for @media, :id => "fileform", :url => upload_url, :multipart => true do %>
  ...
  <% end %>

Lets handle this inline and non-blocking

Add a button field to the end of your form like so:

  <button class="button" id="buttonUpload"  onclick="return beginFileUpload();">Upload</button>

and this to your javascript file (note, this is jQuery)

  function beginFileUpload()
  {
      // todo: some kind of feedback that upload has started...
      $("#uploading").show();
      $("#buttonUpload").hide();

      $("#validation_code").load('<%= url_for(:controller=>"media_file", :action=>"upload") %>', 
            {    title: $("#title").val(),
                 description: $("#description").val()
            },
            function(responseText, textStatus, XMLHttpRequest) {
                $("#validation_code").val(responseText)
                if (textStatus == 'success')
                    completeFileUpload();
                else
                    alert('Error uploading file.');
                });
      return false;
  }

  function completeFileUpload()
  {
      $("#fileform").submit();
      return false;
  }

With that, you have inline file uploading with inline image processing off-site.

For more information, check the docs at http://rubyforge.org/projects/processorpool/