Effortlessly farm work to an EC2 instance without batting an eye

ari on February 20th, 2008 2 Comments

Inline media uploading and processing in ruby

Today, we released a new gem to make integrating off-site processing easy. With this gem and utilizing Amazon’s EC2 cloud and S3 storage back-end, you can easily process media files in a non-blocking, threaded way and do it all inline with the rest of your application.

First download the gem and all it’s dependencies by typing:

sudo gem install processor_pool -y

This gem is built on top of Sinatra. So, if you are familiar with the Sinatra syntax, then you’ll be right at home. If not, I highly suggest you check it out.

First,

# In a file called media_processor.rb
require "rubygems"
require "processor_pool"
 
# Set the access_key_id and the secret_access_key provided by amazon
access_key_id, secret_access_key = 'accesskey', 'somethinglongandfuzzyhere'
 
# And start the processors!
ProcessorPool.start(access_key_id, secret_access_key)

Add to the end of the file

get '/' do
  "Show me the money"
end

That’s it. Start your server up with

ruby media_processor.rb

And now you are ready to go to allow a user to hit the EC2 cloud (more on this later) and be responded with the “Show me the money” text.

Replace the above content with the following

class PFile
  attr_accessor :filename, :temp_file
  @@temp_directory_base = "tmp"
  def initialize(f)
    self.filename = f[:filename]
    self.temp_file = f[:tempfile]
    @media_file = nil
    @temp_dir
  end
  def delete!
    #File.unlink(self.temp_file)
    self.temp_file.delete
  end
  def store_in_temp_directory
    @temp_dir = File.join(@@temp_directory_base, @upload.member_id, @upload.media_file_id)
    FileUtils.mkdir_p(@temp_dir)
    filename = File.basename(self.filename)
    FileUtils.mv(self.temp_file.path, File.join(@temp_dir, filename))
    @media_file = MediaFile.new(File.join(@temp_dir, filename))
  end
  def process!
    ImageConverter.new(@media_file).convert
    @media_file.delete!
    "Processed images for #{@media_file.original_file_name}"
  end
  def save_to_storage
    # DO THE S3 saving here
    MediaDirectory.new(@temp_dir).copy_to_s3(STORAGE_BUCKET, upload_prefix, true)
  end
  def upload_prefix
    "#{@upload.member_id}/#{@upload.media_file_id}"
  end
end
class ImageConverter
  def resize_to(name, image = nil)
    img = image || Magick::Image.read(self.file.file_name).first
    named_size = self.class.named_sizes[name]
    if named_size =~ /c$/
      dimensions = named_size.gsub(/c$/, '').split('x')
      img.crop_resized!(dimensions[0].to_i, dimensions[1].to_i)
    else
      img.change_geometry!(named_size) do |cols, rows, img|
        img = img.resize(cols, rows)
      end
    end
    img.write(path_to(name))
  end
end
post '/new' do
  @file = PFile.new(params[:file])
  @file.store_in_temp_directory
  @file.save_to_storage
  redirect params[:success_url] if params[:success_url]
end

If you look closely, we are just using some simple classes to handle our conversion. But ignoring that, we are sending a post to the server with file parameters. We can send the process a file and the rest is history. Notice that this all happens on a remote server. None of this is handled locally.

That’s it.

Of course, you can imagine how you could extend this to include movie types and other types of processing, but for the time being, let’s focus the images. (Although I will be showing how to do this in rails, you can use this with any back-end, PHP, Java, etc.)

Let’s add a convenience method in the application.rb

def url_for_upload_server
  if RAILS_ENV == 'production'
    # determine the upload server to try to use
    p = Processors.get_random_processor(::SERVER_POOL_BUCKET)
    "http://#{p.hostname}:4567/new"
  else
    p = Processors.get_random_processor(::SERVER_POOL_BUCKET)
    "http://localhost:4567/new"
  end
end

In your upload controller add

@upload_url = url_for_upload_server

Now, in your upload view, add

<% form_for @media, :id => "fileform", :url => upload_url, :multipart => true do %>
...
<% end %>

Lets handle this inline and non-blocking

Add a button field to the end of your form like so:

<button class="button" id="buttonUpload"  onclick="return beginFileUpload();">Upload</button>

and this to your javascript file (note, this is jQuery)

function beginFileUpload()
{
    // todo: some kind of feedback that upload has started...
    $("#uploading").show();
    $("#buttonUpload").hide();
 
    $("#validation_code").load('<%= url_for(:controller=>"media_file", :action=>"upload") %>',
          {    title: $("#title").val(),
               description: $("#description").val()
          },
          function(responseText, textStatus, XMLHttpRequest) {
              $("#validation_code").val(responseText)
              if (textStatus == 'success')
                  completeFileUpload();
              else
                  alert('Error uploading file.');
              });
    return false;
}
 
function completeFileUpload()
{
    $("#fileform").submit();
    return false;
}

With that, you have inline file uploading with inline image processing off-site.

For more information, check the docs at http://rubyforge.org/projects/processorpool/

Tags: , , , , , ,

This entry was posted on Wednesday, February 20th, 2008 at 5:58 pm. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

« What Tools do You Use? || What’s that? RSpec Haml Scaffolds? Yes, it is! »

2 Responses to “Effortlessly farm work to an EC2 instance without batting an eye”

  1. Daniel Fischer Says: February 20th, 2008 at 6:12 pm

    Awesome dude, keep up the good work. This is going to help out a lot of people.

  2. Nicolás Sanguinetti Says: February 21st, 2008 at 6:12 pm

    Woah, cool. I know I would’ve loved this about 4 months ago.

  3. Nicholas Barthelemy Says: March 3rd, 2008 at 6:13 pm

    Wow. This is beautiful! Thank you for sharing.

  4. Juliebs Says: March 19th, 2008 at 6:13 pm

    thank you, bro

  5. tommy Says: April 14th, 2008 at 6:13 pm

    btw, i think there’s a typo i had to do gem install ProcessorPool, not processor_pool

Leave a Reply