Handling APIs with Ruby XML Parsing
Have you ever wanted needed to write a Ruby wrapper for an XML based API?
If you have a burning desire to seamlessly exchange data over the web or you just want to use the latest Interweb 2.0 service – you’re most likely contemplating writing your API client in Ruby… yes that’s why you’re here isn’t it?
Why Use Ruby to Parse XML?
It’s a fact – Ruby kicks ass at parsing XML. You can find tons of examples of XML API clients written in Ruby.
ActiveResource parses XML and handles RESTful HTTP
The new ActiveResource Rails gem found in Rails2 makes pretty light work of handling XML APIs via REST. Unfortunately not everyone is rushing to support REST just yet. If you support a big Rails 1.2 app you can’t just run out an add ActiveResource to your project (which is my case). This post is not about REST or ActiveResource so if you’re looking for that, click the link you just clicked skipped over.
Enough with the useful useless Ruby XML facts…
Show me how to write a Ruby XML API wrapper
I’ve been working with an API recently for a third party registration system on a project we’re rolling out soon. The third party provides their API using domain scoped query URLs and HTTP GET params and returns XML documents.
Huh?
http://example.com/aaflinn/lookup?username=billlumberg&password=swingline
XML Messages
When you get a matching user/pass combination you get something like this.
<auth>
<user>
<username><![CDATA[billlumberg]]></username>
<fullname><![CDATA[Bill Lumberg]]></fullname>
<zipcode><![CDATA92131]></zipcode>
<email><![CDATA[bill.lumberg@initech.com]]></email>
</user>
</auth>
When you put the wrong password you get an error like so.
<auth>
<error><![CDATA[Invalid username/password combination]]></error>
</auth>
If you don’t enter a username at all you’ll get an error thusly.
<auth>
<error><![CDATA[No username given.]]></error>
</auth>
API Wrapper Concepts
Here are some concepts that I felt were important when I started writing my wrapper.
- Simple – it should be easy to code (I hate writing stupid code)
- DRY – if it’s worth writing the wrapper make sure it’s reusable
- Self Documenting – it should be as self documenting as possible (rdoc)
- “Exceptional Code” – it should raise errors on exceptions and handle errors from the libraries it makes use of
- Don’t Spam – Don’t abuse APIs (grrr!)
Requirements
The wrapper should use a GET query string to perform a query to the service provider passing an md5 hashed password and username combination. If the user exists parse the XML result document and return an instantiated user object based on the XML. If no user exists raise some type of rescuable error (RecordNotFound).
Additionally the wrapper should be able to create a new user. This particular service provider uses an HTTP GET query string but in some cases you might find a plain old POST like you’d see in a form or you’ll need to build XML and pass that back to the service provider (which I am not doing here). The wrapper should be able to be able to interpret error messages and determine the status of our create request in a graceful way.
Ruby API Wrapper by Example
The names have been changed to protect the innocent. I’ve edited the wrapper a bit to reduce some complexity and renamed it to hide the actual API provider. Read on in the comments of the code, I’ve done everything on that bullet list and you can read the code as you read the comments setting up just about every line of code written.
#
# Usage
#
# === setup ===
# require ‘ApiExample’
# ApiExample::account = ‘test’
# ApiExample::logger = Logger.new(‘example.log’)
#
# ===Find A User===
# user = ApiExample::User.find(‘bill.lumberg’, :password => ‘swingline’)
#
# ===Create A User===
# user = ApiExample::User.create(:username => ‘bill.lumberg’, :password => ‘swingline’, :email => ‘bill.lumberg@initech.com’)
#
require ‘base64’
require ‘digest/md5’
require ‘net/http’
require ‘rexml/document’
require ‘cgi’
require ‘logger’
module ApiExample
# Example Database Host
HOST = ‘www.example.com’
# These params are required to register a new user
REGISTRATION_PARAMS = [ :username, :password, :email ]
# ApiExample account (brand account not user account)
@@account = nil
@@success_url = nil
@@fail_url = nil
@@logger = Logger.new(STDERR)
mattr_accessor :account, :logger, :success_url, :fail_url
# Exception handling
class ApiExampleError < StandardError; end
class UnexpectedError < ApiExampleError; end
class RegistrationError < ApiExampleError; end
class RecordNotFound < ApiExampleError; end
def md5password(password)
Base64.encode64(Digest::MD5.digest(password)).strip
end
# Using Struct here allows us to make our object act similar to an
# ActiveRecord object. In this example it’s not so obvious of a pain
# in the ass it is but the real class has about 20 or so attributes
# Using means I don’t have to create attribute read and writes
# attribution – I got this idea from the Ben Vinegar’s
# <a href="http://rubyforge.org/projects/freshbooks/">freshbooks gem</a>
User = Struct.new(:username, :email, :password, :fullname, :zipcode)
# Extend the Struct attributes by adding the class and instance methods we want
class User
attr_accessor :attributes, :errors, :new_record
# class method for create a new user
# Usage:
# user = ApiExample::User.create(:username => ‘bill.lumberg’, :password => ‘swingline’, :email => ‘bill.lumberg@initech.com’)
#
def self.create(attributes = {})
object = new(attributes)
object.create
object
end
# class method for finding an existing user
# Usage:
# ApiExample::User.find(‘bill.lumberg’, ‘swingline’) # plain text will be sent md5 hashed rather than clear text
#
def self.find(username, password)
query_params = { :username => username, :password => ApiExample::md5password(password_option[:password]) }
query = query_params.collect{ |k, v| [k, v].map{ |kv| CGI::escape(kv.to_s) }.join(’=’) }.join(’&’)
# @@account doesn’t need to be encoded because we are setting internally, the other stuff is vulnerable
uri = URI::HTTP.build(:host => ApiExample::HOST, :path => "/#{ApiExample::account}/lookup", :query => query)
# I’ve got a bunch of these loggers throughout which are used to debug, remove as you see fit.
ApiExample::logger.debug("URI: #{uri}")
# Make the actual HTTP Request
result = Net::HTTP.get_response(uri)
# Parse the XML Result of the HTTP Request
response = REXML::Document.new(result.body)
ApiExample::logger.debug("RESPONSE: #{response}")
# Check to see if there is a user node, if there’s not raise an error similar to ActiveRecord
unless response.elements[’//user’].nil?
attributes = {}
# Members is the Struct method for the attributes we setup…they correspond to what
# we expect the XML to return, so lets only handle what we know
members.each do |field_name|
node = response.elements[’//user’].elements[field_name.to_s]
next if node.nil?
attributes[field_name.to_sym] = node.text # you can do casting here if you need, I did
end
object = allocate
object.attributes = attributes
object
else
# Raise an error, setting the message to what the API sets as the error in XML
# if there is no ‘error’ node in the XML set an ‘unknown error’ message
raise RecordNotFound, (response.elements[’//error’].text rescue "Couldn’t find ApiExample::User with Username: #{username} because of an unknown error!")
end
end
# instance method to setup new object ala ActiveRecord
# Usage:
# user = ApiExample::User.new(:username => ‘bill.lumberg’)
#
def initialize(attributes = {})
@new_record = true
self.attributes = attributes
end
# instance method ala ActiveRecord
#
def errors
@errors = [] if @errors.blank?
@errors
end
# instance method to get an attributes hash ala ActiveRecord
#
def attributes
butes = {}
members.each{ |member| butes[member.to_sym] = self.send(member) } # you can cast here, I did
butes
end
# instance method to set attributes via a hash ala ActiveRecord
#
def attributes=(new_attributes = {})
members.each do |member|
unless new_attributes.has_value?(member.to_sym)
# #{member}= because you might possibly write an attribute writer to handle the input to =
self.send(member+’=’, new_attributes[member.to_sym]) # you can cast here, I did
ApiExample::logger.debug("#{member}: #{self.send(member).inspect}") # for snooping
end
end
end
# instance method to create the new instantited object ala ActiveRecord
# note the difference in create and self.create are the same as in ActiveRecord
# Usage:
# user = ApiExample::User.new(:username => ‘bill.lumberg’)
# user.password = ‘swingline’
# user.email = ‘bill.lumberg@initech.com’
# user.create
#
def create
# we pass our success and fail URLs even though we aren’t using them for their intended purpose
query_params = { :success => ApiExample::success_url, :fail => ApiExample::fail_url }.merge(attributes)
ApiExample::logger.debug(query_params.inspect) # for snooping
# check to make sure all required params are included
if ApiExample::REGISTRATION_PARAMS.all?{ |param| query_params.include?(param) }
# URL Encode everything
query = query_params.collect{ |k, v| [k, v].map{ |kv| CGI::escape(kv.to_s) }.join(’=’) unless v.blank? }.compact.join(’&’)
# @@account doesn’t need to be encoded because we are setting internally, the other stuff is vulnerable
uri = URI::HTTP.build(:host => ApiExample::HOST, :path => "/#{ApiExample::account}/register", :query => query)
ApiExample::logger.debug("URI: #{uri}") # for snooping
# Make the actual HTTP Request
response = Net::HTTP.get_response(uri)
ApiExample::logger.debug("RESPONSE: #{response}") # for snooping
# Check for failure and errors
if response[‘location’].include?(ApiExample::fail_url)
# Raise an error for each message
CGI::parse(URI::parse(response[‘location’]).query)[‘error’].each do |error_message|
raise RegistrationError, error_message
end
elsif response[‘location’].include?(ApiExample::success_url)
# Change the status to reflect the fact that we’ve saved the object…this could get much more
# in-depth in terms of ensuring our data was correctly saved… you wouldn’t expect to do
# that kind of thing in an RDBMS so why here?
self.new_record = false
else
# it wasn’t the fail url, it wasn’t the success url so what the hell was it?
raise UnexpectedError, "Unknown response URL!"
end
else
# For each missing required param add an error message
ApiExample::REGISTRATION_PARAMS.each do |missing_param|
self.errors << "#{missing_param} can’t be blank." unless query_params.include?(missing_param)
end
end
end
end
end
Rails Deployments Tips
Very often when projects go into production there are a few tweaks needed to make sure the app works correctly in its production environment and stays working. Below are some tips I’ve come up with from my experience rolling rails apps. Most of them are no brainers and some probably apply only to the Rails development we’re doing at work.
Rotate Rails Log Files
Log files get big really quick on high traffic sites. Some server apps rely on the operating system or a log rotating script to archive or delete old logs. Rails has a built in feature thanks to Ruby’s Logger class. The following code will keep 50 archived logs each 1mb in size and automatically rotate the log out once it hits 1mb.
In conf/environments.rb
Rails::Initializer.run do |config|
# ...
config.logger = Logger.new(File.join(RAILS_ROOT, 'log', "#{RAILS_ENV}.log"), 50, 1.megabyte)
Ignore Sensitive Parameters in Logs
Sick of seeing your bank password showing up in your Rails based shiny new web 2.0 app? Yeah I know don’t use your bank password but hey if you’ve done it users of your shiny new web 2.0 app probably will too. Keep them safe by keeping their passwords out of your logs. The following hides the password and password_confirmation params from your logs.
In conf/environments.rb
# Include your application configuration below
ActionController::Base.filter_parameter_logging :password, :password_confirmation
Robots.txt
If you have a development site that isn’t password protected to the public for whatever reason you’ll want to make sure the development site isn’t getting indexed by search engines and cluttering up results for your production site.
Create a file called robots-development.txt and on your development site use mod_rewrite to point to the file.
User-agent: *
Disallow: /
Password Protect Mod Proxy
So this goes along with the last topic. If you’re using password protection along with mod proxy you’ll notice that your assets are password protected but page requests still make it to mongrel. Painful on your search results if msn, yahoo or googlebot come to visit.
<Proxy balancer://mongrel_cluster>
AuthName "Keep Out"
AuthType Basic
AuthUserFile htpasswd.users
Require valid-user
BalancerMember http://192.168.1.2:8000
#...
Turn on /etc/init.d scripts!
This seems like a no-brainer but if you installed apache from source and/or you are using mongrel cluster you’re doing this manually.
Copy the script from the mongrel_cluster resource directory to /etc/init.d
# cp /usr/local/lib/ruby/gems/1.8/gems/mongrel_cluster-1.0.5/resources/mongrel_cluster /etc/init.d/mongrel_cluster
# chmod +x /etc/init.d/mongrel_cluster
On Redhat
# chkconfig --level 345 httpd on
# chkconfig --level 345 mongrel_cluster on
# update-rc.d httpd defaults
# update-rc.d mongrel_cluster defaults
More to come…
And We're Back!!!!
Some friends recently reminded me that my blog has been down for almost 2 months. So I’ve managed to spend some time to bring the site back up in Typo 5. It kind of goes without saying that deploying rails apps on shared hosts sucks and I can stand behind that. But that’s OK – right tool for the right job.
In the last two months there’s been a lot written about Ruby. There’s been some bad press (zed and friends) and a lot of good press (Rails Machine, Rails 2, Ruby 1.9). Some crying fowl about Rails not being enterprise ready, the community sucking, it’s hard to deploy or whatever else. Who cares! Most of the complaints and accusations I’ve read are from people that need to point the finger in the mirror.
While my site was down a lot changed. Anyone keeping track of me (yeah lots of you I’m sure) knows my startup was a non-starter and I spent 2 long dreadful PHP filled months at a search marketing company and have since moved on to more reputable endeavors with a Connecticut firm building Rails based entertainment apps. I’m working with a team of experienced producers and engineers and I couldn’t be happier. So you can expect some new Ruby and Rails tidbits here again soon.
One last thing to mention before I sign-off…if you read my post mentioning PHP’s lack of late static binding PHP and ActiveRecord you can rejoice or cringe depending on your perspective because the team is adding the static:: runtime resolution that will get PHP that life extending kidney implant that it so desperately needs. This new feature will allow Zend and friends to build a pragmatic Active Record in PHP. I’ve bolded this because a friend of mine said they are fixing the problem but it’s not a fix – it’s no doubt a feature request from Zend that was originally slated for PHP6. Given the slow adoption rate of PHP5 I’m sure the team decided to add almost all the new goodies originally slated for PHP6 to PHP5.3 in an effort to keep up. With all the new (untested) features I’m sure sysadmins will be rushing to deploy this new version.
