Your web-browser is very outdated, and as such, this website may not display properly. Please consider upgrading to a modern, faster and more secure browser. Click here to do so.


Sep 16 '13

How To Bundle Ruby Gems & JRuby into a Standalone jar file for Wukong Map/Reduce Streaming Hadoop Jobs

The title says it all. I needed a way to deploy wukong map/reduce jobs alone with my own rubygems without asking for updates every time I need a new gem.

1. Grab a copy of my skeleton working directory…

git clone

2. Edit the job.rb with your own wukong map/reduce job

3. Any libraries, etc. can go in ./lib

4. Run rake

rake deploy

5. Copy the job to hdfs

hdfs dfs -put job.jar working_directory/job.jar

6. Run your streaming job…

$HADOOP_CMD jar $HADOOP_STREAMING -archives 'hdfs://host:port/working_directory/job.jar' -mapper 'jruby job.jar/job.rb --map' -reduce 'jruby job.jar/job.rb --reduce' -input words -output wordcount

Tags: hadoop ruby jruby nosql

Mar 12 '13

Command Line Tool for Oracle Coherence

I’ve been working with Coherence for about 2 years and am still surprised there is no decent command line tool for basic debugging of Oracle Coherence distributed cache (and no I don’t count the out of the box tool as decent).

I’m working on a command line utility for Oracle Coherence which is basically a thin wrapper over Oracle’s C++ client library.

So far I’ve it’s working on Mac OS X and Windows but I’m certain it will work on Linux and Solaris as well.

Basic usage:

./ <cache-name> get <key1> [key2] ...
./ <cache-name> mget
./ <cache-name> put <key> <value>
./ <cache-name> mput <key1> <value1> [<key2> <value2>] ...
./ <cache-name> delete <key> [key2] ...
./ <cache-name> size
./ <cache-name> keys
./ <cache-name> values
./ <cache-name> key_exists <key>
./ <cache-name> value_exists <value>
./ <cache-name> clear

It’s free and open source (MIT license). If you find it useful, drop me a line.

Feb 26 '12

Snippy update to Rails 3.x

Snippy has been updated to the latest (at the time) Rails 3.x

In the future I’ll likely move the code highlighting away from Ultraviolet/TextPow to a Javascript based code highlighter like Rainbow -

I’d also like to move away from Sphinx for full text search because I think it makes the setup more complex than needed.

Aug 3 '10

Scala Stuff I’m excited about

For around the last 5 years I’ve been big into Ruby (Rails and friends).  Lately though I’ve been working on a few projects in Scala.  Scala’s been around for almost a decade but only recently it seems to have developed a larger community and better tools.  As I’ve been working with Scala in my day to day I’m getting really enthusiastic about some of the tools that I think make Scala really great to work with.

So here’s a quick list of what I’m working with and why I like it.

1. Scalate - Scala Template Engine: like JSP without the crap but with added Scala coolness.

As it states, Scalate is a template engine.  It’s somewhat generic though and supports a few different dialects.  I haven’t worked with it yet but it supports a Scala dialect of HAML called Scaml which looks very promising.  I’m a long time user of Haml so this gets me excited.

It also supports Mustache templates which is what brought me to use Scalate in the first place.  I love working with Mustache because it’s a simple and logic-less approach (the right approach) to templates.

2. Circumflex ORM - lightweight Web framework and ORM for Scala

Circumflex is a larger project at it’s core to provide a web framework (inspired by Ruby’s Sinatra micro-framework), Markdown and Freemaker template processing and ORM.  I’ve been working with the ORM which is relatively new so I’ve submitted a few patches where I felt like it was needed.  The ORM package is interesting in that it features a SQL-like Scala DSL.  Here’s an example:

class Country extends Record[Country] {
  val code = "code" VARCHAR(2) DEFAULT("'ch'") NOT_NULL
  val name = "name" TEXT

object Country extends Table[Country] {
  def findByCode(code: String): Option[Country] = {
    val co = this AS "co"
    val q = SELECT (co.*) FROM co WHERE (co.code LIKE code)
    return q.unique

Obviously a very interesting way to interface with relational databases.  It also supports generic `get` by id and `all` methods Rails style validations, query caching and some other goodies.  It’s an up and coming project so obviously it’s not perfect but given the choice between programming with Circumflex ORM vs JPA I think Circumflex offers a much better programming model </flamebate>.

3. Scalatra (formerly Step) - Tiny Scala web framework, inspired by Sinatra

Scalatra is a micro-framework inspired by Ruby’s Sinatra.  It’s very simple to create named paths via HTTP verbs (GET, POST, PUT, DELETE, etc.) not unlike JAX-RS but with out all the annotations and other malarkey.  Scalatra also supports Scalate so you get Haml and Mustache style templates with only a few lines of code!

Here’s a quick example:

class ScalatraExample extends ScalatraServlet {
  // send a text/html content type back each time
  before {
    contentType = "text/html"

  // respond to '/' with a greeting
  get("/") {
    <h1>Hello world!</h1>

  // send redirect headers
  get("/see_ya") {

  // parse matching requests, saving things prefixed with ':' as params
  get("/date/:year/:month/:day") {
      <li>Year: {params("year")}</li>
      <li>Month: {params("month")}</li>
      <li>Day: {params("day")}</li>

4. Cascal - a high-level scala based cassandra library

Cascal is as it’s name and description implies a Scala interface for the NoSQL database Apache Cassandra.  I’ve been testing Cassandra as a high throughput buffer for a write intensive API and so far it live up to the hype.  Cascal seems to offer the most native Scala interface.  Of course there are several others out there, Cliff Moon’s Scromium looks promising.

And there’s plenty more to come.  Coming from the Ruby world I’m finding Scala to be a great middle ground between Ruby and Java.  It brings with it a vast array of high quality libraries that can sometimes be lacking in the Ruby world.  Getting into Scala has introduced me to these great tools and plenty more existing Java libraries.. HawtDB, Play!, Hazelcast, etc.

7 notes Tags: scala scalate scalatra circumflex mustache orm cassandra

Apr 1 '10

Kyoto Cabinet patch fix compatibility on Mac OSX Snow Leopard (Kyoto Cabinet 0.9.9)

This patch will allow you to compile Kyoto Cabinet 0.9.9 on Mac OS X 10.6 (Snow Leopard).

10 notes Tags: kyoto cabinet tokyo cabinet nosql patch

Mar 29 '10

Tokyo Tyrant patch fix unix sockets on Mac OSX Snow Leopard

On Mac OSX 10.6 (Snow Leopard) I receive this warning during make:

ttutil.c: In function ‘ttopenservsockunix’: 
ttutil.c:190: warning: call to __builtin___snprintf_chk will always overflow destination buffer 
ttutil.c: In function ‘ttopensockunix’: 
ttutil.c:130: warning: call to __builtin___snprintf_chk will always overflow destination buffer 

I then receive this error when attempting to start ttserver using a socket: 

$ ttserver -host /tmp/ttserver.sock -port 0 worker.tct 
2010-03-13T00:46:42-05:00       SYSTEM  ————- logging started [18312] ———— 
2010-03-13T00:46:42-05:00       SYSTEM  server configuration: host=/tmp/ttserver.sock port=0 
2010-03-13T00:46:42-05:00       SYSTEM  maximum connection: 8191 
2010-03-13T00:46:42-05:00       SYSTEM  opening the database: worker.tct 
Abort trap 

The following patch applies to Tokyo Tyrant 1.1.40 and fixes the issues and passes tests.

I’ve also forwarded this to Mikio Hirabayashi. 


29 notes Tags: tokyo cabinet tokyo tyrant patch nosql

Mar 29 '10

Tokyo Cabinet patch to add B+ Tree traversal commands

The following is a patch that adds support for two commands iterrec and iterout, for dealing with B+ Tree databases.  The commands are not currently available via misc and are not available via Tokyo Tyrant’s Lua extension.  The purpose of this addition is to expose list management functions to Tokyo Tyrant via TCADB.  Adding these commands would allow you to write tcrdbmisc based functions for remote management of B+ Tree lists.

The patch is meant for Tokyo Cabinet 1.4.43


14 notes Tags: tokyo cabinet tokyo tyrant patch nosql